from the United Nations: “UNMISS and Partners Conduct Human Rights Community Awareness Programmme”

Title: Understanding the Contributions of the Humanities to Human Development: A Methodological White Paper

Author(s): Danielle Allen, Chris Dean, Maggie Schein, Sheena Kang, Melanie Webb, Annie Walton Doyle

Publisher: Harvard University

Year: 2016

URL: http://www.pz.harvard.edu/sites/default/files/HULAWhitepaper.pdf

Topics: humanities, education, impact evaluations, assessment tools, evaluation as assessment

Methods: assuming/defining theoretical concepts about education, coding the learning pathways of the humanities, and then correlating the “logic” of the learning pathways with comparable logical constructs from the study of psychology

What it says: In this white paper, the Humanities and Liberal Arts Assessment (HULA) group at Harvard University proposes a theoretical and methodological framework for understanding and assessing the contributions of the humanities to human development, based on preliminary analysis of qualitative materials from partner organizations and other researchers.* The theoretical framework builds upon two different concepts about education: 1) education as a system of institutions, which is maintained by the state to serve utilitarian purposes (such as cultivating civic service or civic responsibility); and 2) education as individual acts of instruction, which relates to personal development. The practice of humanities is then likened to the practice of “crafts” that help advance education and contribute to human development. The methodological framework treats the humanities as “crafts” that follow certain “craft logics” (pathways by which the craft is practiced, towards the achievement of the goals of the craft). The main idea is that if each humanities practice could be broken down according to categorical logics of its practice, then each tool used and each step of progress achieved in undertaking the practice could be coded in a standardized way to help researchers assess its utility or value.

What I think about it: As a layman, I found the presentation of the language and construction of the HULA methodology too abstract and too academic, which could harm its mass adoption. The definitions and applications of the methodology need to be greatly simplified for the value of the concept to shine through. I am not entirely convinced that it is necessary to make so many parallel comparisons (humanities as “crafts,” each craft as an “artifact,” manner and purpose of humanities practices according to “craft logics,” each logic pathway translated from some comparable construct in psychology) as it could be more effective to simply make a strong case that every practice of the humanities could be coded according to certain logics, and define these logics in an easy-to-understand code book of sorts. I proposed a simplified summary of the main idea above, which could be a good place to start unpacking some of these concepts in a way that even non-experts like me can better understand and then adopt.

What it all means: HULA argues that gathering, coding and analyzing humanities as “crafts” that follow “craft logic” can help us break down the elements that make up a craft, order the elements in a logical developmental pathway, and ultimately understand how humanities practices lead to the achievement of particular educational or human development outcomes. Assumptions are made about each humanities practice in terms of the elements it comprises, how it works, what it is trying to achieve, and which skills it develops – and the effect or effectiveness of each of these components are then coded categorically. Applying the HULA methodology according to the proposed definitions and categorizations requires that the user is familiar with or can easily understand concepts that are rather academic and often abstract, which may ultimately limit its adoption by the wider public.

* It is unclear from this white paper how many partner organizations have been consulted, although the paper did explicitly note that the study sample included at least a “30-year archive of successful grant applications to the Illinois Humanities Council.”

In a previous Createquity Research Spotlight, we wrote about the “What Works” initiative in the UK, which borrows a policy evaluation methodology from the medical community and applies it to (among other things) the arts. In this case, HULA is developing metrics to evaluate arts and humanities using the discipline’s own tools.

HULA does this by repositioning the humanities as “an assemblage of craft practices,” whereby each craft embodies distinctive goals, logical methods,and results that are passed from master to apprentice over thousands of years. For example, in the craft of pottery-making, we could attribute a set of tools, techniques, and sequencing – a general sense of purposeful and procedural logic – that all contribute toward creating a beautiful or useful product. By organizing humanities as individual practices and crafts, we can start to identify different steps, logical patterns, and tools that each activity utilizes to produce an outcome.

In this research context, human development itself is the desired outcome. The HULA white paper thus explores three key research questions:

1. How do the humanities contribute to human development?
2. To what extent are the humanities effective in achieving this goal?
3. How can we measure the effects of the humanities anyway?

HULA defines “human development” in terms of education, subdividing that definition in two ways: 1) education as a system of institutions, maintained by the state to serve utilitarian purposes (such as cultivating civic service or civic responsibility); and 2) education as an individual act of instruction, which relates to personal development. In short, the teaching of humanities is likened to the practice of “crafts,” which also help advance education and contribute to human development.

To explore how different humanities practices contribute to these educational outcomes, HULA identifies various elements – or building blocks – that make up different humanities practices. All of these elements are then sorted and coded according to HULA’s proposed methodological framework; this allows us to see which elements are different or common to various humanities practices, and thus track how they lead to similar or dissimilar learning pathways toward the goal of human development.

In Figure 1 below, HULA defines four sequential stages of a learning pathway and attributes possible elements of a given craft to certain learning processes or outcomes: input, processing, and short-term and long-term results. In this example, the red arrows represent the pathway of a political philosophy instructor who engages students in close readings (verbal input) and logical debate (cognitive-analytical analysis), with the aim to encourage understanding of political concepts (the short-term goal), which in turn may enable students to become more civic-minded citizens (the long-term goal).

Figure 1. Example of a HULA Learning Pathway. Adapted from “Humanities and Liberal Arts Assessment White Paper”, by The HULA Research Team. 2015, p.15.

HULA’s application of coding schemes to different humanities activities requires a set of assumptions about each humanities practice or craft, including the elements it comprises, how it works, what it is trying to achieve, and which skills it develops. These assumptions draw on the implicit logic associated with the practice or craft. Once these elements are broken down, the study assigns the process and outcome advanced by each element. The elements are thus coded categorically along a structured framework of possible learning pathways, which allows us to track the progress and outcome of a given humanities practice or craft. To see how this coding process works in detail, the HULA study on 30 Years of Illinois Humanities Council Grant-Making features appendices of the “code structure” and “code sets” used to track the different methods and mechanisms followed by various humanities activities.

Does this sound complicated? Indeed, applying this methodology, with its myriad definitions and categorizations, requires users to absorb concepts that can be rather academic and abstract. This barrier could limit the adoption of HULA’s methodology beyond the realm of the academy.

Still, the research logic of the study is worth a deeper dive. The core concept of its design makes sense, especially when the terms of its application are simplified. Ultimately, by coding and analyzing humanities as a series of crafts – each of which has its own elemental purpose and logic –we have a new way of unpacking what each practice is really about, what elements it comprises, how it works, and towards which learning outcomes it steers.

If we can manage to make the language of the HULA model a bit more accessible, we just might have a promising methodology for assessing the value of the humanities – using evaluation tools drawn from the discipline itself, as opposed to metrics from other disciplines, which are often an imperfect fit.

Cover image: “Pottery” courtesy of Lennart Tange. via Flickr Creative Commons license.

The UK government has been asking just this question. Drawing from the experience of the medical community’s National Institute of Health and Care Excellence (NICE) model, which systematically assesses and synthesizes the cost-effectiveness of medical interventions to help the UK National Health Service (NHS) prioritize its public spending, the UK Cabinet office has envisioned its What Works initiative as a “NICE for social policy.”

Launched in March 2013, the What Works Network set out to evaluate the performance of public policies and programs using evidence collected throughout their implementation. For every policy or program, What Works tracks which social benefits that program has achieved and how much money those benefits cost per participant. What Works evaluations aim to help policymakers and practitioners improve their decision-making process by providing evidence and advising on which interventions offered the best value for money.

Generally speaking, the What Works methodology rates a policy or program on its effectiveness according to indicators that are specific to the sector in question. For example, the success of an education policy might be rated according to how many additional months of progress a student makes in a classroom. This rating becomes even more useful to practitioners when complemented by an estimate of how much money it costs per student to implement the policy.

Now in the Network’s third year of operation, seven “What Works Centres” and two affiliate members – What Works Scotland and the Public Policy Institute for Wales – have been established thus far across the UK, each focused on a particular area of policy. They monitor and evaluate interventions according to a standardized methodology in seven categories: educational achievement, local economic growth, crime reduction, health and social care, wellbeing, improved quality of life for older people, and early intervention for at-risk children. And yes, the What Works initiative is evaluating arts interventions within the broader context of these public policy areas.

For example, the What Works Centre for Well-Being analyzes the impacts of culture and sports on wellbeing according to the same four dimensions – satisfaction with life, happiness, worthwhileness, and anxiety – used by the National Statistics of the UK to assess personal wellbeing and subjective wellbeing. Similarly, the Sutton Trust and Education Endowment Foundation’s (EEF) Teaching and Learning Toolkit provides an easy-to-read ranking on the cost-effectiveness of arts participation for improving educational outcomes for students aged 5-16, relative to other interventions.

So what does What Works say about how the arts work? A report issued by the UK government in 2014 presented a selection of early findings from the six What Works Centres that had been active up to that date. Two projects relating to the arts were included. One project from the Centre for Local Economic Growth examined 36 evaluations covering the impact of major sport and culture projects on the local economy and found that the overall measurable impacts were rare, and small if they existed at all. Built facilities, however – with sporting facilities comprising the vast bulk of the evidence – might increase the value of properties in their immediate vicinity. The Sutton Trust-EEF Teaching and Learning Toolkit found that arts participation had a low positive impact on student education attainment (defined as “additional months progress you might expect pupils to make as a result of an approach being used in school, taking average pupil progress over a year as a benchmark”), but for much lower cost compared to some other learning interventions with similar impact, such as attending summer school or using teaching assistants. The Teaching & Learning Toolkit collects impact evidence from EEF projects covering 34 education topics, and impact results are regularly updated and summarized as they are collected.

What Works’s venture into evidence-based policy was still in its infancy at the time of the report’s publication. Since then, several new arts-related projects have been commissioned and evaluated. An “Act, Sing, Play” project sought to answer whether exposure to high quality music education was more cost-effective than drama participation for improving students’ literacy and math scores, and another “SHINE on Manchester” project assessed to what extent Saturday music education improved students’ literacy and math scores. While assessments of these two projects did not yield convincing evidence that participation in the arts helped achieve the designated outcomes of improved literacy and math scores, they also did not discount the possibility that arts participation might yield other positive outcomes.

Is evidence-based evaluation of public policy the wave of the future? In the United States, a loose alliance of several organizations would like to make it so. Results for America aims to spearhead smart policy changes at all government levels by encouraging the use of best available data, evidence and evaluation about what’s effective. The Laura and John Arnold Foundation (LJAF) is likewise active in this area, having recently absorbed the Coalition for Evidence-Based Policy, a Washington think tank that had success in advocating for government reforms, into its grantmaking. At this very moment, LJAF is holding a $15 million competition to encourage government and nonprofit organizations to implement highly effective programs, and Results for America just launched a global initiative “identifying the policies, programs and systems that governments are using to support the production and use of data and evidence” called Results for All.

In theory, this approach of collecting, synthesizing, and ranking evidence from a diverse range of policy and program evaluations will help make that evidence accessible to a wide audience – and that is undoubtedly a good thing. At the same time, paring down the impacts of policies and programs to cost-effectiveness might be challenging when goals are less readily quantifiable, or where effectiveness needs to be assessed according to more innovative or perhaps even abstract criteria. In such cases, less relevant targets might become more appealing to policymakers because they are cheaper or easier to tag with numbers, resulting in an oversimplified framework for measuring impact that displaces a true understanding of effectiveness. Arts and cultural policies arguably are particularly vulnerable to this risk, particularly given that we are only beginning to understand the true nature of their value to individuals and society. At Createquity, we don’t think it is impossible for the benefits of the arts to be assessed under a What-Works-style evaluation framework, but we do have to be careful that we are attempting to measure the right things – the things that arts are actually good for.

The relatively small and weak body of information and data on the impacts of arts/culture policies and programs shows that there are significant gaps and limitations – but also much room to grow – for What Works’s assessments of arts interventions going forward. In the meantime, we can do our part to contribute evidence to What Works inquiries by submitting tips, research and assessments of public policies to the relevant Centres. As of publication, EEF is seeking new education interventions to fund and evaluate, and welcomes applications from “projects that show promising evidence of having a measurable impact on attainment or a directly related outcome” until December 9.

Cover image: “Mathematica” courtesy of the Ivan T. via Flickr Creative Commons license.