The ubiquitous multiple choice answer sheet. Photo by COCOEN.

Pretty much no one likes standardized tests. The concept is nothing new, of course – the New York State Regent Exam dates back to Civil War times. A century and a half later, the implementation of No Child Left Behind (NCLB) hinged all of the federal government’s reward and punishment on a school’s “Adequate Yearly Progress” (AYP), a now-infamous composite measure of school performance primarily based on test scores. In the decade since, test-bashing has become something akin to a national pastime, and folks are acting out.

Earlier this year teachers in Seattle flat out refused to administer mandated state exams, claiming that the tests were a misuse of precious school resources, unfairly used as part of teacher evaluations, and an inaccurate indication of student learning. And Seattle isn’t alone. The organization United Opt Out National has assembled a state-by-state guide for opting your kids out of testing, claiming, “high-stakes testing is destructive to ALL children, educators, communities…and the democratic principles which underlie the purposes of public education.”

Let’s say they’re right and standardized tests have got to go. What would be a scalable alternative? One possible solution percolating amongst education reformers may surprise you: portfolios. The practice of assessing learning with portfolios has deep roots in the arts world, visual arts and creative writing especially. Could portfolios save our public school students from a life of drill-and-kill?

The Mechanics

A portfolio is a collection of individual work samples (some of which may have been graded previously), assessed as a whole. It’s a way of combining disparate items into one aggregate assessment demonstrating the application of skills and concepts learned in a classroom setting. Portfolios can be either summative or formative in structure. A summative portfolio focuses on the product or end result of the student’s learning such as, for example, a digital recording of a final performance, a scientific lab report, or a final series of photographs. A formative assessment takes into account the student’s process of learning and can include works-in-progress or evidence of the effort leading to the final product. This type of portfolio might include an actor’s annotated script, the shape and light charcoal studies for a still life painting, or math problem demonstrating the steps in between question and answer.

There are three key elements of assessing learning with portfolios:

• Clearly defined skills and/or knowledge to be assessed,
• Work samples, determined either by the student or the teacher or both, and
• A rubric with which to score collection of work with clear descriptors for each level of success, usually using a point system, and ideally made available to students before portfolios are submitted. (Examples can be found here.)

Sometimes the items included are accompanied by written reflection on the process of creating the final product or the intention behind the work. To demonstrate content knowledge, more traditional academic writing may be included as well.

Part of a Larger Movement

Portfolios are one assessment tool under the larger umbrella of an emerging mode of student evaluation called performance assessment. According to Beyond Basic Skills: The Role of Performance Assessment in Achieving 21st Century Standards of Learning by Linda Darling-Hammond and Frank Adamson:

For many people, performance assessment is most easily defined by what it is not: specifically, it is not multiple choice testing. In a performance assessment, rather than choosing among pre-determined options, students must construct an answer, produce a product, or perform an activity.

Using performance assessment, a student might be asked to write a letter to the editor about a historical event from a specific point of view, draw a series of electrical circuits explaining how changes in configurations would affect the flow of electricity, or demonstrate the ability to use a map by actually navigating.

Performance assessment is rising in popularity as the Common Core State Standards inch closer to full implementation. These new learning standards, adopted by 45 states, four U.S. territories, and the District of Columbia, require “a greater focus on critical thinking, synthesis and analysis, problem solving, communication, media and technology.” States including Maine, New Hampshire, New York, and Ohio are starting to adopt performance assessment systems—which reveal students’ ability to apply information, not just to remember and regurgitate it—to meet the new standards.

Not all performance assessment techniques include portfolios. But portfolios (whether arts-specific or not) are an important piece of the performance assessment puzzle, and learning from the arts’ experience with portfolios could be useful as performance assessment reform initiatives move forward in schools and districts across the country.

Portfolio Assessment in Practice

Some schools and districts are already making use of portfolio assessment. The Beacon School, a public high school in New York City, has been celebrated as a national model of portfolio assessment since it opened in 1993. From the beginning, Beacon leaders wanted to assess their students using methods similar to those employed in graduate schools. Students assembled a portfolio of long-term projects and representative samples from all of their classes – science, history, English, foreign languages – and defended their work to a faculty panel.

By the end of its first decade, Beacon’s plans for portfolio assessment had been somewhat derailed. In the late 1990s New York State began to require students to pass the state’s Regents Exam to graduate. This new requirement meant time was diverted from labs and projects to test preparation. In his 2004 reporting on the Beacon School, writer Jay Matthews explained, “even the most ardent advocates [of portfolios] have acknowledged that samples of student work cannot compete with the ability of standardized testing to quickly and cheaply determine the overall performance of a school district.”

The school still maintains as much of its original assessment strategy as possible. Its website stresses the dual priorities of the school and the state: “Beacon offers a dynamic, inquiry-based curriculum for all students that exceeds standards set by the New York State Regents. Technology and arts are infused throughout the college preparatory curriculum. Each year students must present performance-based projects to panels of teachers, and pass New State Regents tests and community service to graduate.”

School districts in Tennessee are using portfolio assessment for a different purpose that harkens back to its arts-based roots and combines student evaluation with teacher evaluation. In Tennessee’s teacher evaluation system, 35% of a teacher’s employment review is based on test scores. This is a problem for the 70% of teachers in most schools who do not teach a state-tested subject, but portfolios may offer the solution. As reported back in 2012 in The Commercial Appeal newspaper, “[The head of Memphis City Schools arts education, Dru Davison,] and 40 Memphis art teachers wrote a four-page rubric for what peer reviewers should see in the student work for each of 40 art disciplines, from marching band to jazz band.” Each teacher chooses samples of his or her students’ work to form the teacher’s review portfolio, which is then assessed by a blind peer group based on the pre-determined rubrics. The Tennessee Fine Arts Growth Measures System, as it’s called, is in the pilot phase in Memphis City Schools and is starting to garner some national attention. Participation is voluntary for Tennessee school districts. More than ten are participating this school year, up from three last year.

Laura D. Goe, a research scientist at the Educational Testing Service, told EdWeek about the initiative, “Tennessee has the right idea in promoting this effort to achieve some rigor and comparability in a set of content that is difficult to measure…To me, it is a model for where we want to ultimately go [with teacher evaluation], and where I think we will go in most subjects.”

Challenges

The case studies in Tennessee and the Beacon School are intriguing and reflect well on portfolios as a school-wide, district-wide, and maybe even statewide option for evaluating students and even teachers. Could portfolio assessment become a core mechanism for measuring student learning, on the scale of standardized tests?

Statewide and national testing systems depend on reliable, valid results. Reporting on The Beacon School, Jay Matthews wrote, “the argument between advocates of standardized tests and advocates of portfolios usually ends with each side saying it cannot trust the results produced by the other.” For portfolio assessment, it is often the problem of subjectivity that causes concern among test supporters.

Standardized tests, with so-called selected response questions such as multiple choice or true/false, don’t need to be graded by humans. Such questions can arguably be biased, and there is the possibility of human error in the setting of the machines and the handling of the scoring sheets, but the grading is never subjective thanks to grading machines, which also make the process comparatively faster and less expensive. By contrast, portfolios must be graded by humans and grading between raters or even the same raters at different times can be inconsistent. A good rubric and rigorous training can eliminate some personal bias, but not all (see page 22 of Beyond Basic Skills for more information). This problem is a big one if portfolios are to be adopted on a large scale. The ability to reliably compare standardized tests makes it possible to identify outliers among schools, districts, and states, to learn from overachievers, and support underachievers. If the assessment itself and the grading method are not the same for all students, the evaluation won’t be useful for these purposes.

Fortunately, there are large-scale testing systems that already deal with this problem. The Advanced Placement exams taken by high school students in advanced classes in subjects like history and English, for example, are composed of a series of essay questions of various lengths. Students take the exams and then their teachers ship their answer booklets off to be graded by trained “readers.” The writing portion of the SAT is another example. The volume of exams to be graded by each trained reader means that for a handful of SAT essays, factual errors may be overlooked as readers are rewarded for speed. In such cases, students end up demonstrating their knowledge of the grading system rather than their writing ability. While there may be issues with the quality of grading vs. quantity of exams, the fact that the SAT has been accepted for years as a satisfactory (if far from perfect) method of judging students’ college readiness should mean that the subjectivity challenge of grading portfolios is nothing new or prohibitive.

In education, it all comes down to implementation. If a student’s portfolio is filled with work samples that aren’t authentic demonstrations of knowledge and skills learned, it is no better an indicator of learning than a long chain of multiple-choice questions with memorized answers. For portfolio assessment to succeed as a nationwide option for student evaluation, appropriate learning goals must be set with rigorous and specific rubrics. Teachers must also be well trained in administering and scoring assessments, and students well prepared. It’s a novel concept to some, but if portfolios continue to spread as a viable, scalable assessment method, we might emerge from the era of crushing accountability into a new age – one in which testing has a positive effect on learning. 

The lobby of the Willard Hotel is rumored to be the birthplace of the term “lobbying.”
Photo by Ellen Meiselman

In March of 2012, Grantmakers in the Arts (GIA) launched the Arts Education Funders Coalition. The goal of the Coalition is “to research and identify federal policy opportunities that promote equitable access to arts education in all public schools.” It consists of about 135 individuals from 115 organizations within GIA’s membership and is led by a small advisory committee of prominent voices in arts advocacy, education, and philanthropy.

This is new territory for GIA. The organization’s president and CEO, Janet Brown, acknowledged in a recent conversation that public policy can be uncomfortable, risky, and “very difficult to get funders to invest in.” Successes are few, far between, and at the mercy of our volatile political process. But when a critical mass of arts education funders felt funding nonprofit programs was no longer a sufficient strategy to achieve their aspirations to further arts education in public schools, they decided to attempt to affect policy directly, and GIA took on the challenge.

Setting the stage

When attempting to influence public policy, an organization must first decide where and how to target its efforts. Will the focus be on the federal, state, or local level? Should a law be enacted, tweaked, or repealed? With the Arts Education Funders Coalition, GIA decided to focus on the federal level and work toward adding pro-arts language to existing education legislation.

In announcing the formation of the Coalition, Janet Brown explained:

Why is GIA involving itself in federal policy, you might ask. It’s because that’s where decisions are made in education in America. Although we’d love to believe that education policies are determined locally, the reality is federal policy drives the actions made by state departments of education and local superintendents and school boards. Our obsession with testing to determine learning is evidence of this. Equity issues are best dealt with at the federal level where the governmental “carrot” is meant to level the playing field.

Despite its national aspirations, the Funders Coalition hasn’t garnered much attention to date from the arts sector. According to Janet Brown, “[The Arts Education Funders Coalition is] not a very visible project because it’s a different kind of advocacy than American for the Arts (AFTA).” AFTA, the lead advocacy organization for our sector, mobilizes email campaigns and organizes Arts Advocacy Day, which brings hundreds of people to Washington each year in an overt attempt to draw the attention of policymakers to issues concerning the arts community, among other advocacy efforts. (The organizations are talking to each other: AFTA’s Vice President of Government Affairs and Arts Education, Narric Rome, is on the advisory committee of the GIA Arts Education Funders Coalition.) GIA is taking a quieter approach, banking in part on the assumption that a pro-arts education agenda would have more clout coming from a group of people who have skin in the public education game:

As a group of funders who have contributed millions of dollars to the public education system or to the nonprofit arts sector to compensate for lack of arts education in public schools, Coalition members and other funders have a stake in developing effective policy that will secure the place of arts education in twenty-first century education.

To lead the effort, GIA hired a Washington, DC firm specializing in education policy, the Penn Hill Group, to help develop an agenda and do the on-the-ground lobbying. Executive vice president Alex Nock has been presenting the Funders Coalition’s progress as part of GIA’s web conference series.

The Funders Coalition’s agenda takes on many aspects of federal education policy, including juvenile justice, research, Head Start, teacher evaluation, and the cornerstone of federal education legislation, the Elementary and Secondary Education Act (ESEA, formerly known as No Child Left Behind). It describes the arts-positive change the Funders Coalition would like to see in each area, including:

• “that any school improvement structure adopted in ESEA reauthorization… include arts education as a strategy in the overall plan to turn around a low-performing school,”
• “that arts education be integrated into the Head Start standards and partnerships be encouraged between Head Start providers and community arts organizations,” and
• “that the [Investing in Innovation] program adopt an absolute priority for arts education that requires the Department of Education to fund quality applications with an arts education focus.”

The agenda is multifaceted, but GIA’s focus is on language. According to Janet Brown, the Coalition faced a choice at its outset. It could propose new, stand-alone legislation to advance arts education, or lobby to change existing laws. After considering the contentious political climate, relatively low priority of education policy on the congressional to-do list, and extreme amount of time and effort brand new legislation would require, GIA and the Penn Hill Group deemed the latter option more realistic.

The story so far

We’re just weeks away from the Coalition’s second birthday. What progress has it made?

On November 18,2013, GIA sent an email to members of the Funders Coalition with good news: “Through work with Members of Congress and their staff, [we were] able to ensure that arts education would play a prominent role in the preschool programs funded under [a proposed] bill should it pass Congress over the next year.” Time will tell how “prominent” that role actually is, but the announcement suggests an encouraging victory for what is a relatively new effort.

Pre-K may have been low-hanging fruit for the Funders Coalition. Without standardized testing and other competitors for the class time that older students face, Pre-K curricula naturally have more room for the arts. But according to Brown, the inclusion of arts-friendly language in the bill was not inevitable. “If we had not been there the language would not have been included,” Brown said. “Bills are written based on the knowledge of the staff who are writing them.” If that’s the case, it’s a good thing the Penn Hill Group and GIA are there to educate them.

The Coalition has also succeeded in getting more specific language included in Senator Tom Harkin’s bill to reauthorize ESEA, which will hopefully be brought to the Senate floor for consideration in March or June. ESEA has been waiting for reauthorization since 2007, and Alyson Klein for EdWeek accurately called its chances back in June: “Everyone knows that the reauthorization of the Elementary and Secondary Education Act is ultimately headed absolutely nowhere [in 2013], thanks to partisan divisions.” The language included in Senator Harkin’s bill is a win for the Funders Coalition, but the success of the bill is difficult to forecast given Congress’s recent track record of inaction when it comes to ESEA.

The next act

The Funders Coalition’s effort raises familiar but tricky questions about arts education advocacy:

• Is legislative language without designated funding enough to make real change? New York City’s arts education legislation is a great example of the folly of language that isn’t backed by specific, dedicated funding. In 2007, $67.5 million previously earmarked for arts education was released to the discretion of school principals. Accountability measures were put in place to theoretically ensure that students received the arts education prescribed by law, but many arts advocates would argue that the state of arts education in New York City schools has declined as a result of the change. The conventional wisdom is that if decision makers don’t have to spend money on a non-tested subject, they likely won’t, focusing resources instead on subjects in which the school is held accountable for its performance. To ensure the delivery of arts education in schools, policy needs to mandate what is to be provided to students, allocate dedicated funding, and establish a mechanism to ensure compliance. How much will be budgeted, for example, for the arts if they are to play a prominent role in preschool programs across the country? Will the appropriated funding be enough for full-time arts teachers in every school or simply materials with which general classroom teachers can incorporate arts projects? Language will help, but it’s not everything. Funding is a very important piece of the puzzle. Which begs the question…

• Does it make more sense to work at the federal or state/local level? Federal policies set priorities which states are encouraged to adopt via competitive funding programs and other means of reward and punishment. All states receiving Race to the Top grants, for example, must develop comprehensive teacher evaluation systems as a condition of funding. Nevertheless, education is constitutionally assigned as a state concern and the bulk of education funding – over 90% – comes from state and local sources. Therefore, state and local officials have the most control over how education funds are spent. If you grant the premise that money is a key, if not the key, ingredient in successful reform, the states could be the best place to advocate for arts education.

For now, it’s too early to tell what answers we might glean from GIA’s experience, and whether its quieter strategy will pay off. It could be that potential victories for the Funders Coalition really do influence state priorities and lead to expanded arts education opportunities in schools. At the very least, as the Funders Coalition continues its work we should know more about the potential for funders to be advocates. There are many valuable lessons to be learned as this effort continues.

With “How Art Works: The National Endowment for the Arts’ Five-Year Research Agenda,” the National Endowment for the Arts is getting proactive. Acknowledging that the NEA’s research efforts have been mostly descriptive in the past, “How Art Works” is intended to usher in a new era of strategic inquiry for the agency and the sector alike.

The practical goal of “How Art Works” is actually broader than this: beyond a research agenda for the NEA itself, it “proposes a way for the nation’s cultural researchers, arts practitioners, policy-makers, and the general public to view, analyze, and discuss the arts as a dynamic, complex system.” The strategy involves stating “feasible, testable” hypotheses about all manner of arts-related impacts on individuals and society in the form of a system map. The map in turn is intended to provide a theory of change to guide arts research and to facilitate field-wide investigation and discussion.

The map’s starting point – the Big Bang for the entire arts ecosystem as we know it – is the “human impulse to create and express.” This impulse is the motivation to experience or participate in artistic creation. The rest of the nodes on the map spring from this drive and fit into four broad categories: Inputs, Art itself, Quality-of-Life Outcomes, and Broader Societal Impacts. In order to clearly define each node on the map, “How Art Works” includes a graphic representation of what the authors call a “Multi-Level Measurement Structure” for each component or variable within the main nodes, which will inform the measurement model for future research involving each part of the system.

In the final section of “How Art Works,” the authors assess to what extent the NEA’s current research priorities square with the hypotheses represented in the map. These pages feature a list of priority projects initiated or planned by the NEA’s Office of Research and Analysis over five years beginning in 2012, with each project keyed to an element of the map.

“How Art Works” does a good job of defining a reasonable and comprehensive model of the arts’ impact against which to consider the NEA’s own research efforts. However, the report is a bit like an impressionist painting: from far away it looks complete, but when you get close, individual features are hard to make out. There are a few contributing factors to this problem:

• There are times when the system map is lacking specificity and detail in previously well-researched areas of the arts sector, arts education being one example.
• The report fails to take full advantage of prior attempts to map the impacts of the arts. As a result, the report isn’t entirely duplicative, but it isn’t 100% additive either.
• Negative ramifications of the arts are arguably understated. If the system map presented is meant to be realistic – a picture of how art really does work, and not a romantic representation of how we would like it to work – the possible negative effects of self-expression should be acknowledged more explicitly.

By the time this research agenda was released, the NEA had already made an inaugural round of fourteen research grants. The research is compelling, and the best examples will hopefully lead us forward as a field. Happily, the quality of that work will not be diminished by a grand vision that is, arguably, still under construction.

(For a brief summary of this article, check out “How Art Works: the I’m-late-for-work version.”)

With “How Art Works: The National Endowment for the Arts’ Five-Year Research Agenda,” the National Endowment for the Arts is getting proactive. Acknowledging that the NEA’s research efforts have been mostly descriptive in the past, “How Art Works” is intended to usher in a new era of strategic inquiry for the agency and the sector alike.

Released in September 2012, “How Art Works” “stems from a collaborative research inquiry.” Over the course of ten months prior, the NEA’s Office of Research and Analysis and the global consulting firm Monitor Institute conducted interviews and solicited input from a host of people inside and outside the arts sector. (A full list of thought partners is included in Appendix B.) The resulting report lays out a conceptual framework that is meant to aid in the planning and assessment of the NEA’s research priorities from 2012 to 2016, and to facilitate reporting to the White House Office of Management & Budget, Congress, and the public on the Endowment’s progress.

Summary 

The practical goal of “How Art Works” is actually broader than this: beyond a research agenda for the NEA itself, it “proposes a way for the nation’s cultural researchers, arts practitioners, policy-makers, and the general public to view, analyze, and discuss the arts as a dynamic, complex system.” The strategy involves stating “feasible, testable” hypotheses about all manner of arts-related impacts on individuals and society in the form of a system map. The map in turn is intended to provide a theory of change to guide arts research and to facilitate field-wide investigation and discussion.

Mapping the Impact of the Arts

The fundamental hypothesis of “How Art Works” is as follows:

[E]ngagement in arts contributes to quality of life. Quality of life contributes to society’s capacity to invent, create, and express itself. This capacity contributes back to art, both directly and indirectly.

The effects of engagement in the arts are cyclical and reciprocal, inherent and instrumental, and can be seen in artists, participants, and society at large.

To specify and organize the range of possible interactions, reactions, and transactions, the major feature of “How Art Works” is a system map, a visual representation of the parties and forces at play. According to Innovation Network, “this approach involves first visually mapping the system of interest and then identifying which parts and relationships are expected to change, and how.” A system map facilitates discussion with the goal of getting everyone on the same page.

In the “How Art Works” system map, the major components of the arts ecosystem are represented as “nodes” connected by arrows representing the relationships between those nodes.

The map’s starting point – the Big Bang for the entire arts ecosystem as we know it – is the “human impulse to create and express.” This impulse is the motivation to experience or participate in artistic creation. The rest of the nodes on the map spring out of this drive and fit into four broad categories: Inputs, Art itself, Quality-of-Life Outcomes, and Broader Societal Impacts. In order to clearly define each node on the map, “How Art Works” includes a graphic representation of what the authors call a “Multi-Level Measurement Structure” for each component or variable within the main nodes, which will inform the measurement model for future research involving each part of the system. For example, here is the Measurement Structure for one of the two system Inputs, education and training:

Each Measurement Structure includes a list of “definitional questions and methodological challenges” (not pictured here), which are designed to help readers imagine how one might effectively gather data on all of these various and in some cases slippery concepts. Challenges include things like “insufficient data available,” difficult-to-isolate factors of change, or the limitations of prior research.

To summarize the map, we’ll walk through each of the nodes within the four main categories and briefly touch on the methodological challenges and observations on potential future research.

Inputs. According to “How Art Works,” the translation of the universal impulse to express into actual artistic activity depends on two Inputs, education and training and arts infrastructure.

Education and training represents all manner of arts learning opportunities, “from YouTube and street jam sessions, to K-12 and adult arts education, to apprenticeships and conservatory training.” It is education, both informal and formal, that allows for skillful expression of ideas for artists and develops a stronger sense of personal taste for arts consumers.

Arts infrastructure is described as the “institutions, places, spaces, and formal and informal social support systems that facilitate the creation and consumption of art.” These support systems include venues, organizations, schools, networks, unions, and associations, as well as less concrete elements like financial and volunteer support and public policy. Of course, the quality of available infrastructure matters as much as the quantity.

Art. Given the initial spark to create and the infrastructure and training to fan the flame, the next category is the Art itself. The artistic product, represented on the map by the yellow circle, has two sub-categories: creation and participation. Arts creation is the production of an artistic work within an established or emerging set of artistic principles with “the intention of communicating richly to others.” Arts participation includes creation but extends more broadly; it is defined as “the act of producing, interpreting, curating, and experiencing arts… and the consumption of these outputs.”

Finally, we move on to outcomes of the art, of which there are two types included as part of the map: Quality-of-Life Outcomes (so-called first-order outcomes) and Broader Societal Impacts (second-order outcomes).

Quality-of-Life Outcomes. Quality-of-life outcomes are the direct effects of interaction with the arts on individuals and communities. The report acknowledges that these outcomes could be positive or negative, though the word “benefit” is consistently used to describe them. The two nodes in this category are:

• The benefit of art to individuals, and
• The benefit to society and communities.

The outcome called the benefit of art to individuals encompasses the “cognitive, emotional, behavioral, and physiological effects” on participating individuals over time.

The second outcome, benefit to society and community, is farther-reaching than the first. This node encompasses “the role the art plays as an agent of cultural vitality, a contributor to sense of place and sense of belonging, a vehicle for transfer of values and ideal, and a promoter of political dialogue.”

The report points out a few definitional questions, methodological challenges, and issues for further exploration to keep in mind with these Quality-of-Life Outcomes:

• In current research many of these effects are intertwined and overlapping. Determining what can be isolated and tested will be an important step in illuminating these outcomes.
• One complication when measuring the benefit of art to individuals is the uneven distribution of participation in arts activities. Presumably some people who like the arts like them a lot and choose to engage deeply and consistently, whereas others may never feel compelled. This is a methodological challenge because it makes data about impact difficult to generalize across the entire population.
• It is unclear whether the impact of the arts on children versus adults is different enough to demand separate evaluation strategies.

Broader Societal Impacts. The final node of the system map represents the second-order outcomes on society at large. A more detailed version of the map (immediately below) shows three new elements within this category:

• societal capacities to innovate and express ideas,
• new forms of self-expression, and
• outlets for creative expression.

These outcomes are included as “works in progress” in the report, which declines to offer a measurement structure for any of them. The latter two in particular are moving targets, as new art forms, techniques, and platforms continue to emerge; the authors cite platforms such as YouTube and Facebook and forms of expressions like data visualization as examples.

System Multipliers. Beyond the nodes themselves, the authors posit that certain forces act on all of the various components of the system to varying degrees, all at once, to affect the system’s environment. These so-called System Multipliers are like societal weather. They don’t necessarily change what you’re going to do in a day, but they might change how you do it, or how long it takes.

There are five multipliers proposed: markets and subsidies, politics, technology, demographics and cultural traditions, and space and time. It’s easy to imagine how these forces could shift the arts sector in big and small ways. Technology, for example, has completely changed the way we share and consume art. It has also changed the ways artists get compensated for their work, for better or worse, in ways many did not see coming even five years ago.

Obviously, the reach of these multipliers is vast. The authors explain at a high level how each multiplier could affect the arts sector, but do not provide insights on potential research applications beyond suggesting they be periodically tracked using evidence from individual research projects.

Shaping the Research to Come

In the final section of “How Art Works,” the authors assess to what extent the NEA’s current research priorities square with the hypotheses represented in the map. These pages feature a list of priority projects initiated or planned by the NEA’s Office of Research and Analysis over five years beginning in 2012, with each project keyed to an element of the map.

After reviewing the opportunities and potential areas of emphasis, the authors call attention to “a research gap associated with the nodes and relationships on the left side of the map: Societal Capacities to Innovate and to Express Ideas, and, in the expanded version of the map, New Forms of Self-Expression and Outlets for Creative Expression.” These nodes “mark a vast unsettled terrain” that “ultimately may hold the most promise and profit for those seeking to measure arts-related impacts.” Despite this promise, however, “How Art Works” notes that “most of the NEA’s research agenda for the next five years will continue to focus on arts infrastructure, education/training, arts participation and creation, and individual and community-level benefits,” areas where the NEA already has significant investment. The reasoning is that there are gains to be made in the short term by building on recent findings about Inputs and the benefit of art to individuals.

The report encourages the NEA to decide whether the rapidly-evolving “work-in-progress” elements outlets for creative expression and new forms of self-expression, as well as the catalyst of the system, the human impulse to express and create, are areas the Office of Research and Analysis can address in the next five years, or whether others will need to take the lead. “A reasonable approach might be to lodge these concepts in the broader dialectic of the arts research community, so that new hypotheses, research questions, populations, data sources, and methods might be proposed by groups outside the NEA.”

Finally, the authors point to the need to strengthen structural support for this research, for example through the NEA’s ongoing efforts to partner with other federal agencies for both program delivery and data collection. These priorities include “a clear need to build national time-series (preferably longitudinal) data collections including arts variables.” Additionally, over the next few years the NEA’s Office of Research and Analysis plans to consolidate arts-related data and make it publicly available to other cultural researchers.

Analysis

“How Art Works” does a good job of defining a reasonable and comprehensive model of the arts’ impact against which to consider the NEA’s own research efforts. However, the report is a bit like an impressionist painting: from far away it looks complete, but when you get close, individual features are hard to make out. It is a step forward to be sure, but its broad-brush approach and failure to build explicitly on past attempts, combined with a few untended loose ends, represent a missed opportunity to be a watershed for cultural researchers.

The Necessary Specificity

”How Art Works” is certainly not lacking in grand designs. A map of any sector from public agencies to individual consumers that fits within a report as short as this one is inevitably going to work at a high level. But there are times when the map’s lack of specificity and detail seems to call into question its value entirely. Take, for example, Illustration 5 (the second image above), the Measurement Structure for education and training. This diagram divides all of arts education and training into two sub-categories – arts subjects and non-arts subjects – and then further divides those into art itself and evaluation of art, then further into modes of instruction, and finally into discipline “types” being studied.

This framework is frustratingly generic. Arts education is an area we actually know quite a bit about. There are professional associations representing every corner of the field, from in-school arts teachers to community-based organizations to art therapists. There’s even the National Center for Creative Aging, which has a bird’s-eye view of current trends in arts education for older adults. The NEA’s own prior research is especially strong in this area, though this isn’t reflected in the text or the diagram. At some point in the process of making this and other models widely applicable, specificity – and with it utility – have been somewhat lost. If the goal is to direct further research, the system map should take full advantage of the knowledge we have already.

Doing so might have pointed out areas where the frameworks don’t quite make sense, even at the very high level on which they operate. For example, the generic language of “art as a subject” and “art in non-arts courses” brings to mind child-age learners in school settings. This seems a strange choice, since education and training according to the authors is meant to include arts learning in informal settings for learners of all ages. It may be that the map needs to be generic for universality, but it certainly shouldn’t implicitly narrow the field with imprecise language.

The audience for How Art Works is described briefly as “researchers, practitioners, policy-makers in the arts and in other sectors.” But this lack of specificity serves neither experts in the field, who are offered scant new insights into their focus areas, nor those less familiar with the arts, who will glean only the most general outlines of the work involved, even for well-established fields of practice like arts education. For those completely oblivious to the arts sector and all of its components – those who have never considered the connection between a community dance studio, what goes on inside the walls of the Kennedy Center, and, say, the economy – this report could be educational, but only to a point. Some nodes are more instructive than others. Benefit of art to society and communities for example (pictured above), could enlighten and spur new thinking for people who don’t spend all day dissecting the impact and values of the arts. But education and training wouldn’t provide much information for people who don’t already know what arts education looks like in all of its various forms.

Building on Prior Work

As a framework for understanding the effects of the arts, “How Art Works” misses important chances to incorporate and build on prior work like Gifts of the Muse. Published in 2005, Gifts is an extensive literature review that attempts to compile all of the various claimed benefits of the arts to individuals and society and evaluate and connect them in one text. It details the intrinsic benefits (captivation, pleasure, expanded capacity for empathy, cognitive growth, creation of social bonds, expression of communal meaning) and instrumental benefits (attitudinal, behavioral, health, community, economic) of the arts, assesses the available evidence, and makes recommendations on how to fill some of the gaps. Like “How Art Works,” Gifts constructed a working model of the benefits of the arts, the summary version of which is pictured here:

A few years before Gifts, the Cultural Dynamics Project (a collaboration between The Bolz Center for Arts Administration at the University of Wisconsin-Madison, the consulting group National Arts Strategies, and the California non-profit Cultural Initiatives Silicon Valley) likewise created a system map of the arts and cultural sector. The collaborators’ hope was that such a map would “inform our arts policy, expand our understanding as arts professionals, guide and contextualize existing and future research on the field, and drive more thoughtful funding initiatives among those seeking to support and evolve this complex system.” That sounds familiar. It’s worth noting that the individuals involved with the Cultural Dynamics Project didn’t let 8.5×11-inch paper shrink their vision or cause them to over-generalize. The map comes complete with instructions on how to attach four sheets together to reveal the full diagram. Another attempt to conceptualize the benefits of the arts was Alan Brown’s 2006 essay “An Architecture of Value.” Brown’s model explicitly builds on Gifts of the Muse and focuses on the language we use to explain “the value and benefits of arts experiences.”

While “How Art Works” has the additional mandate of providing research guidance for a particular (and important) player in the field, it’s odd that so much of the report – and presumably the effort that went into making it – was spent on ground well-covered by these and other previous works – especially since several of the individuals involved in those previous initiatives were consulted during the discovery process. “How Art Works” isn’t entirely duplicative, but it isn’t 100% additive either. With the ultimate goal of getting everyone on the same page so that arts research can proceed in a more coordinated manner, it’s ironic that these prior efforts to map the system were scarcely mentioned in even the footnotes or appendices of the report.

Neglecting the Negative

“How Art Works” sets out to “[provide] a conceptual frame for planning and assessing research priorities so that the NEA can improve its ability to meet a core goal: To Promote Knowledge and Understanding about the Contributions of the Arts.” One would expect such a report to accentuate the positive when it comes to the impact of the arts. But if the system map presented is meant to be realistic – a picture of how art really does work, and not a romantic representation of how we would like it to work – the possible negative effects of self-expression should be acknowledged more explicitly. Take, for example, this sentence from an early paragraph describing the forces at play in the system map in the simplest terms:

The impact [of the arts on a person] also flows back to the artist, directly in some instances (e.g., the artist sells a work of art) and indirectly through education, infrastructure, and society’s general embrace of creativity and freedom of expression.

In fact, creativity channeled through the arts does not always lead to an expansion of freedom of expression. Sometimes it leads to a backlash and a crackdown. We have seen widespread objections to certain works of art: the agency has been repeatedly accused of funding “pornography” (see reason number five of these Ten Good Reasons to Eliminate Funding for the National Endowment of the Arts). In 1998, after decades of periodic controversy over NEA grants, the Supreme Court ruled that the agency could make “decency” a consideration in funding decisions. This kind of stipulation doesn’t stop the production of controversial work, but it does send a message about the value of certain works of art, and art more generally to society, and it has real implications for artists’ ability to find financial support for their work. The impact that circles back to the artist is sometimes a structural restriction of creativity and expression.

Art itself can also be intrinsically harmful. The authors acknowledge in a footnote that, “[b]ecause communities do not all have the same values, ideals, or political inclinations, art that is seen as beneficial by one community can appear threatening to another.” But this understates the potential for art to do damage; for example, there is no shortage of racist music out there reinforcing and promoting prejudice. Finally, the arts can have unintended indirect effects, such as their ever-controversial role in neighborhood gentrification. Many observers have argued that  the influx of artists into an area can spur a kind of neighborhood change that is harmful to residents who subsequently are priced out of their homes or experience a more metaphorical sense of displacement. That kind of impact is not included in the benefits to society and communities structure, but nevertheless represents an example of a tangible outcome that could be caused by the arts.

Implications

So, what if it worked? What if we pursued this research agenda as laid out, and we really did have all of the metrics under all of the nodes defined in five years? We would know how things work now. But I can’t help wishing that this research agenda were more aspirational. If we had instead a working map of an ideal arts sector, could we be stronger at the end of five years instead of just smarter?

Such a map would show how and where we want average people to encounter the arts in their community, and what the impacts of those encounters would be. It would chart how public sector dollars can work with private and community foundations and individual donors to support a diverse and healthy arts infrastructure. It would theorize what tax structures and other policies could benefit individual artists, and those hypotheses would be tested in the research to follow, instead of simply stating that all of these elements exist.

Such a report would also draw on prior research in these specific areas. A literature review was conducted as part of the process of creating “How Art Works,” presumably for the purpose of assessing the current state of knowledge in the arts. But somehow those insights don’t seem to have made it into the final product, save for an annotated bibliography provided in Appendix A and sorted by map node. By drawing a map that got into details about what we already know and what we have yet to find out, “How Art Works” could have launched the sector into a new era of coordinated, proactive research.

Instead, we will have to be satisfied with the report’s more limited ambitions. By the time this research agenda was released, the NEA had already made an inaugural round of fourteen research grants. Fordham University studied the impact of arts programming on the social skills and mental health outcomes of at-risk youth, exploring the benefit of arts to individuals, society, and communities. Harvard looked into factors contributing to “birth” and “death” rates of arts and cultural institutions, which pertains to arts infrastructure. The University of Dayton studied the relationship between arts engagement and quality of life, investigating arts participation and arts creation. And the University of Georgia performed qualitative research to generate a hypothesis about community-built practice such as playgrounds, mosaic sculptures, murals, and community gardens, looking into the benefits of arts for societies and communities. This research is compelling, and the best examples will hopefully lead us forward as a field. Happily, the quality of that work will not be diminished by a grand vision that is, arguably, still under construction.

Further reading:

• Barry Hessenius, Research and Data Blogathon Day #4 (Barry’s Blog)
• Ian David Moss, Arts Policy Library: Gifts of the Muse (Createquity)
• Alan Brown, An Architecture of Value (WolfBrown)
• The Cultural Dynamics Working Group, The Cultural Dynamics Map: Exploring the Arts Ecosystem in the United States
• Ian David Moss, Live-blogging the “How Art Works” convening (Createquity)
• Keith Sawyer, How Art Works (Creativity & Innovation)
• Alexis Clements, A Grand Unified Theory of Art? (Hyperallergic)
• Mark Robinson, How Art Works? (Thinking Practice)

Kenyan shilling. Photo by Claudia Daggett.

A few years ago four grad students from Harvard and M.I.T. decided they wanted to use their brains and dollars to improve the lives of some of the poorest people in the world. They researched different strategies of philanthropy, looked at the data available, and based on the evidence they chose a novel approach. No microlending, school-building, or vaccination campaigns for them: they would just give away cash, no strings attached. They called their new charity, simply enough, GiveDirectly.

The Concept

This is how it works: money is transferred from the organization to pre-identified families in Kenya via cell phones. GiveDirectly’s selection of recipients is based solely on need as signaled by mud or thatch roofs, as opposed to more durable materials. The standard amount is $1,000 over one to two years, about as much as a poor Kenyan family might spend in a single year. There are no restrictions on what a family can buy with money from GiveDirectly. Unlike with more traditional philanthropic efforts, there are no mandatory health check-ups or vaccinations, no obligatory training programs, and no mandates of any kind. The cash is completely unfettered. It is a “UCT”: unconditional cash transfer.

One common use of GiveDirectly cash transfers is the purchase of a metal roof. Mud and thatch don’t hold up well to weather elements and must be repaired or replaced often. There are major savings to be had with the purchase of a metal roof, which typically lasts at least a decade.

That’s not the only way in which recipients spend the money, however. If you own a motorbike in Kenya that can withstand the terrain, won’t break down on long journeys, and can carry a passenger, you can be a taxi driver. With an influx of cash, you can literally buy yourself a livelihood.

These sorts of purchases improve quality of life and increase earning capacity, and some even have a ripple effect beyond a single family. A cow, for example, can provide milk for an entire village, an income stream for the owner, and can create more cows to continue to multiply the benefits.

The Logistics

Since a normal transfer from bank account to bank account isn’t an option, and transportation and distribution of thousands of dollars of physical cash would require security and increase liability for the charity, the transactions are carried out via cell phone. In Kenya the “mobile-money system” is called M-Pesa, and it’s one of the most successful of its kind; transfers are discreet, simple, and perfect for this atypical exchange.

One M-Pesa location in Kenya. Photo by Fiona Bradley.

Using this technology, GiveDirectly can keep track of hundreds of recipient families with a single spreadsheet and send them money each month safely and securely. The recipient gets a text message indicating that the money has arrived and goes to his or her local M-Pesa franchise to pick it up (the one described in the radio show “This American Life”’s GiveDirectly coverage is basically a VW bus turned bank with a ledger, a box of cash, and one staff person). It might sound dodgy to the Western world, but the M-Pesa system is dependable and ubiquitous in Kenya and saves people time and money previously spent traveling to traditional banks or delivering money to family in remote and inaccessible areas.

The Evidence 

As Jacob Goldstein for the New York Times writes, “At its most basic level… GiveDirectly’s work is an attempt to test one of the simplest ideas in economics — that people know what they need, and if they have money, they can buy it.” As radical as the approach may seem, it is grounded in a strong evidence base. According to Holden Karnofsky, co-founder of a charity that supports GiveDirectly, “cash transfers…happen to be the most extensively studied non-health intervention we know of.” Indeed, traditional charities often call the very act of transferring funds into the hands of low-income people success. In GiveDirectly’s model, by contrast, the impact measures are more nuanced. Money changing hands is the intervention, not the desired outcome.

GiveDirectly is committed to investigating and recording its own impact. The organization recently completed a randomized control trial in Kenya, which reveals that; “recipients are not just spending their transfers, providing a one-time boost to their consumption without affecting their overall well-being.” The trial shows that food-consumption increased 20 percent for transfer recipients and the value of recipients’ livestock increased by 50 percent. The study even showed that recipients’ stress levels improved – their actual stress hormone levels decreased. The full report and a summary are available here.

The Implications

Pondering the broader lessons of UTCs and GiveDirectly, I’m reminded of all of the nonprofit organizations out there for which an influx of funding could really change their organizational “standard of living.” What GiveDirectly is providing goes by another name in the nonprofit sector: general operating support.

General operating support is the holy grail of nonprofit fundraising, defined by the Foundation Center as “grants for the day-to-day operating costs of an existing program or organization; also called unrestricted grants.” There may be reporting requirements Kenyans don’t have to adhere to, and an application process with more vetting than GiveDirectly’s system of identifying families in need, but the concepts aren’t too far off. The similarities between general operating support to organizations and cash transfers to families, however, might not be as obvious for some in the nonprofit sector.

“We had conversations with people [in the non-profit sector] who said there was a lot of internal resistance to unconditional transfers,” Niehaus [one of GiveDirectly’s four founders] told [reporter Dana Goldstein]. “If this works, what are we all here for? Why do we have jobs? There’s an industry that exists that tries to make decisions for poor people and determine what’s best for them.”

Shouldn’t we want the same kind of aid for the poor that we in the nonprofit sector would want for ourselves?

Launched in April, CultureHive currently hosts over seven hundred resources organized into four categories: guides/toolkits, research, case studies, and articles. The AMA has a team of experts gathering resources and creating new ones on a freelance basis, and users are encouraged to submit their own examples of good practice as well.

The platform itself looks great and couldn’t be simpler. The homepage highlights the most recently added and the most popular resources, but it’s really all about the search box, which stretches the length of the page.

I tried typing “youth” into the search box, hoping to find resources about marketing to and engaging young people, and up came fifteen results, all downloadable pdfs. Narrowing the results further to case studies, there were nine choices left that covered pricing strategies, membership structures, orchestra models, family concerts, libraries, and museums, all with a special focus on young people.

Three things seem to set CultureHive apart. First, while CultureHive isn’t the first online repository of helpful resources for arts marketers, it might be the first to balance specificity about marketing for cultural organizations with relevance to diverse corners of the arts sector. In the U.S., for example, the Theater Communications Group and the National Guild for Community Arts Education have online collections of marketing resources, but they cater to their particular slices of the arts world, so they lack broad applicability – and they provide limited access for non-members. Conversely, some sources do cover multiple arts fields but go a bit too broad, neglecting arts-specific information in favor of something akin to Marketing 101. This seems to be the case for the one-size-fits-all marketing resources provided by many state arts agencies, such as those in North Carolina and Texas.

Second, simplicity and speed are paramount in CultureHive’s design – the site is extremely easy to use. Americans for the Arts’s Arts Marketing Project has an online resources section with a lot to offer, but it’s not keyword searchable and can require some digging to locate what you need.

Finding this combination of breadth, depth, and user-friendliness in a single place is rare and could make CultureHive a go-to site, but I would argue that its success is contingent on the third distinguishing factor: an emphasis on sharing. At the bottom of every single page is a link that says, “upload a resource.” The fact that this is a living collection of materials sets the site apart most of all – but only if it keeps growing. Since launching in April, the site has been getting about one submission per week. For CultureHive to really take off, this pace will probably need to pick up considerably.

We’ll have to wait and see if CultureHive lives up to its promise. I’m also curious whether arts marketers outside the UK will find the site useful. At present, the resources available are largely for and by UK arts marketers and organizations. There are some resources featuring examples from the States – including the Philadelphia Orchestra, the Museum of Modern Art in New York, the Providence Performing Arts Center in Rhode Island, and the San Jose Symphony Orchestra – and CultureHive hopes to expand its bank of U.S. information over time. (American readers, feel free to lend a hand with the internationalization: head over to CultureHive and upload away.)

For the moment, in a world where a Google search of “arts marketing to youth case studies” will get you nearly 22 million hits, it’s nice to search for exactly what you’re looking for in a catalogue of material vetted by experts. CultureHive provides a valuable service that will only improve as its pool of resources grows. In the best case scenario, CultureHive could help spur more effective marketing across the arts sector, leading more people to buy tickets, take classes, visit exhibitions – and hopefully like what they hear, see, do, learn and feel enough to keep coming back.