With “How Art Works: The National Endowment for the Arts’ Five-Year Research Agenda,” the National Endowment for the Arts is getting proactive. Acknowledging that the NEA’s research efforts have been mostly descriptive in the past, “How Art Works” is intended to usher in a new era of strategic inquiry for the agency and the sector alike.

The practical goal of “How Art Works” is actually broader than this: beyond a research agenda for the NEA itself, it “proposes a way for the nation’s cultural researchers, arts practitioners, policy-makers, and the general public to view, analyze, and discuss the arts as a dynamic, complex system.” The strategy involves stating “feasible, testable” hypotheses about all manner of arts-related impacts on individuals and society in the form of a system map. The map in turn is intended to provide a theory of change to guide arts research and to facilitate field-wide investigation and discussion.

The map’s starting point – the Big Bang for the entire arts ecosystem as we know it – is the “human impulse to create and express.” This impulse is the motivation to experience or participate in artistic creation. The rest of the nodes on the map spring from this drive and fit into four broad categories: Inputs, Art itself, Quality-of-Life Outcomes, and Broader Societal Impacts. In order to clearly define each node on the map, “How Art Works” includes a graphic representation of what the authors call a “Multi-Level Measurement Structure” for each component or variable within the main nodes, which will inform the measurement model for future research involving each part of the system.

In the final section of “How Art Works,” the authors assess to what extent the NEA’s current research priorities square with the hypotheses represented in the map. These pages feature a list of priority projects initiated or planned by the NEA’s Office of Research and Analysis over five years beginning in 2012, with each project keyed to an element of the map.

“How Art Works” does a good job of defining a reasonable and comprehensive model of the arts’ impact against which to consider the NEA’s own research efforts. However, the report is a bit like an impressionist painting: from far away it looks complete, but when you get close, individual features are hard to make out. There are a few contributing factors to this problem:

• There are times when the system map is lacking specificity and detail in previously well-researched areas of the arts sector, arts education being one example.
• The report fails to take full advantage of prior attempts to map the impacts of the arts. As a result, the report isn’t entirely duplicative, but it isn’t 100% additive either.
• Negative ramifications of the arts are arguably understated. If the system map presented is meant to be realistic – a picture of how art really does work, and not a romantic representation of how we would like it to work – the possible negative effects of self-expression should be acknowledged more explicitly.

By the time this research agenda was released, the NEA had already made an inaugural round of fourteen research grants. The research is compelling, and the best examples will hopefully lead us forward as a field. Happily, the quality of that work will not be diminished by a grand vision that is, arguably, still under construction.

(For a brief summary of this article, check out “How Art Works: the I’m-late-for-work version.”)

With “How Art Works: The National Endowment for the Arts’ Five-Year Research Agenda,” the National Endowment for the Arts is getting proactive. Acknowledging that the NEA’s research efforts have been mostly descriptive in the past, “How Art Works” is intended to usher in a new era of strategic inquiry for the agency and the sector alike.

Released in September 2012, “How Art Works” “stems from a collaborative research inquiry.” Over the course of ten months prior, the NEA’s Office of Research and Analysis and the global consulting firm Monitor Institute conducted interviews and solicited input from a host of people inside and outside the arts sector. (A full list of thought partners is included in Appendix B.) The resulting report lays out a conceptual framework that is meant to aid in the planning and assessment of the NEA’s research priorities from 2012 to 2016, and to facilitate reporting to the White House Office of Management & Budget, Congress, and the public on the Endowment’s progress.

Summary 

The practical goal of “How Art Works” is actually broader than this: beyond a research agenda for the NEA itself, it “proposes a way for the nation’s cultural researchers, arts practitioners, policy-makers, and the general public to view, analyze, and discuss the arts as a dynamic, complex system.” The strategy involves stating “feasible, testable” hypotheses about all manner of arts-related impacts on individuals and society in the form of a system map. The map in turn is intended to provide a theory of change to guide arts research and to facilitate field-wide investigation and discussion.

Mapping the Impact of the Arts

The fundamental hypothesis of “How Art Works” is as follows:

[E]ngagement in arts contributes to quality of life. Quality of life contributes to society’s capacity to invent, create, and express itself. This capacity contributes back to art, both directly and indirectly.

The effects of engagement in the arts are cyclical and reciprocal, inherent and instrumental, and can be seen in artists, participants, and society at large.

To specify and organize the range of possible interactions, reactions, and transactions, the major feature of “How Art Works” is a system map, a visual representation of the parties and forces at play. According to Innovation Network, “this approach involves first visually mapping the system of interest and then identifying which parts and relationships are expected to change, and how.” A system map facilitates discussion with the goal of getting everyone on the same page.

In the “How Art Works” system map, the major components of the arts ecosystem are represented as “nodes” connected by arrows representing the relationships between those nodes.

The map’s starting point – the Big Bang for the entire arts ecosystem as we know it – is the “human impulse to create and express.” This impulse is the motivation to experience or participate in artistic creation. The rest of the nodes on the map spring out of this drive and fit into four broad categories: Inputs, Art itself, Quality-of-Life Outcomes, and Broader Societal Impacts. In order to clearly define each node on the map, “How Art Works” includes a graphic representation of what the authors call a “Multi-Level Measurement Structure” for each component or variable within the main nodes, which will inform the measurement model for future research involving each part of the system. For example, here is the Measurement Structure for one of the two system Inputs, education and training:

Each Measurement Structure includes a list of “definitional questions and methodological challenges” (not pictured here), which are designed to help readers imagine how one might effectively gather data on all of these various and in some cases slippery concepts. Challenges include things like “insufficient data available,” difficult-to-isolate factors of change, or the limitations of prior research.

To summarize the map, we’ll walk through each of the nodes within the four main categories and briefly touch on the methodological challenges and observations on potential future research.

Inputs. According to “How Art Works,” the translation of the universal impulse to express into actual artistic activity depends on two Inputs, education and training and arts infrastructure.

Education and training represents all manner of arts learning opportunities, “from YouTube and street jam sessions, to K-12 and adult arts education, to apprenticeships and conservatory training.” It is education, both informal and formal, that allows for skillful expression of ideas for artists and develops a stronger sense of personal taste for arts consumers.

Arts infrastructure is described as the “institutions, places, spaces, and formal and informal social support systems that facilitate the creation and consumption of art.” These support systems include venues, organizations, schools, networks, unions, and associations, as well as less concrete elements like financial and volunteer support and public policy. Of course, the quality of available infrastructure matters as much as the quantity.

Art. Given the initial spark to create and the infrastructure and training to fan the flame, the next category is the Art itself. The artistic product, represented on the map by the yellow circle, has two sub-categories: creation and participation. Arts creation is the production of an artistic work within an established or emerging set of artistic principles with “the intention of communicating richly to others.” Arts participation includes creation but extends more broadly; it is defined as “the act of producing, interpreting, curating, and experiencing arts… and the consumption of these outputs.”

Finally, we move on to outcomes of the art, of which there are two types included as part of the map: Quality-of-Life Outcomes (so-called first-order outcomes) and Broader Societal Impacts (second-order outcomes).

Quality-of-Life Outcomes. Quality-of-life outcomes are the direct effects of interaction with the arts on individuals and communities. The report acknowledges that these outcomes could be positive or negative, though the word “benefit” is consistently used to describe them. The two nodes in this category are:

• The benefit of art to individuals, and
• The benefit to society and communities.

The outcome called the benefit of art to individuals encompasses the “cognitive, emotional, behavioral, and physiological effects” on participating individuals over time.

The second outcome, benefit to society and community, is farther-reaching than the first. This node encompasses “the role the art plays as an agent of cultural vitality, a contributor to sense of place and sense of belonging, a vehicle for transfer of values and ideal, and a promoter of political dialogue.”

The report points out a few definitional questions, methodological challenges, and issues for further exploration to keep in mind with these Quality-of-Life Outcomes:

• In current research many of these effects are intertwined and overlapping. Determining what can be isolated and tested will be an important step in illuminating these outcomes.
• One complication when measuring the benefit of art to individuals is the uneven distribution of participation in arts activities. Presumably some people who like the arts like them a lot and choose to engage deeply and consistently, whereas others may never feel compelled. This is a methodological challenge because it makes data about impact difficult to generalize across the entire population.
• It is unclear whether the impact of the arts on children versus adults is different enough to demand separate evaluation strategies.

Broader Societal Impacts. The final node of the system map represents the second-order outcomes on society at large. A more detailed version of the map (immediately below) shows three new elements within this category:

• societal capacities to innovate and express ideas,
• new forms of self-expression, and
• outlets for creative expression.

These outcomes are included as “works in progress” in the report, which declines to offer a measurement structure for any of them. The latter two in particular are moving targets, as new art forms, techniques, and platforms continue to emerge; the authors cite platforms such as YouTube and Facebook and forms of expressions like data visualization as examples.

System Multipliers. Beyond the nodes themselves, the authors posit that certain forces act on all of the various components of the system to varying degrees, all at once, to affect the system’s environment. These so-called System Multipliers are like societal weather. They don’t necessarily change what you’re going to do in a day, but they might change how you do it, or how long it takes.

There are five multipliers proposed: markets and subsidies, politics, technology, demographics and cultural traditions, and space and time. It’s easy to imagine how these forces could shift the arts sector in big and small ways. Technology, for example, has completely changed the way we share and consume art. It has also changed the ways artists get compensated for their work, for better or worse, in ways many did not see coming even five years ago.

Obviously, the reach of these multipliers is vast. The authors explain at a high level how each multiplier could affect the arts sector, but do not provide insights on potential research applications beyond suggesting they be periodically tracked using evidence from individual research projects.

Shaping the Research to Come

In the final section of “How Art Works,” the authors assess to what extent the NEA’s current research priorities square with the hypotheses represented in the map. These pages feature a list of priority projects initiated or planned by the NEA’s Office of Research and Analysis over five years beginning in 2012, with each project keyed to an element of the map.

After reviewing the opportunities and potential areas of emphasis, the authors call attention to “a research gap associated with the nodes and relationships on the left side of the map: Societal Capacities to Innovate and to Express Ideas, and, in the expanded version of the map, New Forms of Self-Expression and Outlets for Creative Expression.” These nodes “mark a vast unsettled terrain” that “ultimately may hold the most promise and profit for those seeking to measure arts-related impacts.” Despite this promise, however, “How Art Works” notes that “most of the NEA’s research agenda for the next five years will continue to focus on arts infrastructure, education/training, arts participation and creation, and individual and community-level benefits,” areas where the NEA already has significant investment. The reasoning is that there are gains to be made in the short term by building on recent findings about Inputs and the benefit of art to individuals.

The report encourages the NEA to decide whether the rapidly-evolving “work-in-progress” elements outlets for creative expression and new forms of self-expression, as well as the catalyst of the system, the human impulse to express and create, are areas the Office of Research and Analysis can address in the next five years, or whether others will need to take the lead. “A reasonable approach might be to lodge these concepts in the broader dialectic of the arts research community, so that new hypotheses, research questions, populations, data sources, and methods might be proposed by groups outside the NEA.”

Finally, the authors point to the need to strengthen structural support for this research, for example through the NEA’s ongoing efforts to partner with other federal agencies for both program delivery and data collection. These priorities include “a clear need to build national time-series (preferably longitudinal) data collections including arts variables.” Additionally, over the next few years the NEA’s Office of Research and Analysis plans to consolidate arts-related data and make it publicly available to other cultural researchers.

Analysis

“How Art Works” does a good job of defining a reasonable and comprehensive model of the arts’ impact against which to consider the NEA’s own research efforts. However, the report is a bit like an impressionist painting: from far away it looks complete, but when you get close, individual features are hard to make out. It is a step forward to be sure, but its broad-brush approach and failure to build explicitly on past attempts, combined with a few untended loose ends, represent a missed opportunity to be a watershed for cultural researchers.

The Necessary Specificity

”How Art Works” is certainly not lacking in grand designs. A map of any sector from public agencies to individual consumers that fits within a report as short as this one is inevitably going to work at a high level. But there are times when the map’s lack of specificity and detail seems to call into question its value entirely. Take, for example, Illustration 5 (the second image above), the Measurement Structure for education and training. This diagram divides all of arts education and training into two sub-categories – arts subjects and non-arts subjects – and then further divides those into art itself and evaluation of art, then further into modes of instruction, and finally into discipline “types” being studied.

This framework is frustratingly generic. Arts education is an area we actually know quite a bit about. There are professional associations representing every corner of the field, from in-school arts teachers to community-based organizations to art therapists. There’s even the National Center for Creative Aging, which has a bird’s-eye view of current trends in arts education for older adults. The NEA’s own prior research is especially strong in this area, though this isn’t reflected in the text or the diagram. At some point in the process of making this and other models widely applicable, specificity – and with it utility – have been somewhat lost. If the goal is to direct further research, the system map should take full advantage of the knowledge we have already.

Doing so might have pointed out areas where the frameworks don’t quite make sense, even at the very high level on which they operate. For example, the generic language of “art as a subject” and “art in non-arts courses” brings to mind child-age learners in school settings. This seems a strange choice, since education and training according to the authors is meant to include arts learning in informal settings for learners of all ages. It may be that the map needs to be generic for universality, but it certainly shouldn’t implicitly narrow the field with imprecise language.

The audience for How Art Works is described briefly as “researchers, practitioners, policy-makers in the arts and in other sectors.” But this lack of specificity serves neither experts in the field, who are offered scant new insights into their focus areas, nor those less familiar with the arts, who will glean only the most general outlines of the work involved, even for well-established fields of practice like arts education. For those completely oblivious to the arts sector and all of its components – those who have never considered the connection between a community dance studio, what goes on inside the walls of the Kennedy Center, and, say, the economy – this report could be educational, but only to a point. Some nodes are more instructive than others. Benefit of art to society and communities for example (pictured above), could enlighten and spur new thinking for people who don’t spend all day dissecting the impact and values of the arts. But education and training wouldn’t provide much information for people who don’t already know what arts education looks like in all of its various forms.

Building on Prior Work

As a framework for understanding the effects of the arts, “How Art Works” misses important chances to incorporate and build on prior work like Gifts of the Muse. Published in 2005, Gifts is an extensive literature review that attempts to compile all of the various claimed benefits of the arts to individuals and society and evaluate and connect them in one text. It details the intrinsic benefits (captivation, pleasure, expanded capacity for empathy, cognitive growth, creation of social bonds, expression of communal meaning) and instrumental benefits (attitudinal, behavioral, health, community, economic) of the arts, assesses the available evidence, and makes recommendations on how to fill some of the gaps. Like “How Art Works,” Gifts constructed a working model of the benefits of the arts, the summary version of which is pictured here:

A few years before Gifts, the Cultural Dynamics Project (a collaboration between The Bolz Center for Arts Administration at the University of Wisconsin-Madison, the consulting group National Arts Strategies, and the California non-profit Cultural Initiatives Silicon Valley) likewise created a system map of the arts and cultural sector. The collaborators’ hope was that such a map would “inform our arts policy, expand our understanding as arts professionals, guide and contextualize existing and future research on the field, and drive more thoughtful funding initiatives among those seeking to support and evolve this complex system.” That sounds familiar. It’s worth noting that the individuals involved with the Cultural Dynamics Project didn’t let 8.5×11-inch paper shrink their vision or cause them to over-generalize. The map comes complete with instructions on how to attach four sheets together to reveal the full diagram. Another attempt to conceptualize the benefits of the arts was Alan Brown’s 2006 essay “An Architecture of Value.” Brown’s model explicitly builds on Gifts of the Muse and focuses on the language we use to explain “the value and benefits of arts experiences.”

While “How Art Works” has the additional mandate of providing research guidance for a particular (and important) player in the field, it’s odd that so much of the report – and presumably the effort that went into making it – was spent on ground well-covered by these and other previous works – especially since several of the individuals involved in those previous initiatives were consulted during the discovery process. “How Art Works” isn’t entirely duplicative, but it isn’t 100% additive either. With the ultimate goal of getting everyone on the same page so that arts research can proceed in a more coordinated manner, it’s ironic that these prior efforts to map the system were scarcely mentioned in even the footnotes or appendices of the report.

Neglecting the Negative

“How Art Works” sets out to “[provide] a conceptual frame for planning and assessing research priorities so that the NEA can improve its ability to meet a core goal: To Promote Knowledge and Understanding about the Contributions of the Arts.” One would expect such a report to accentuate the positive when it comes to the impact of the arts. But if the system map presented is meant to be realistic – a picture of how art really does work, and not a romantic representation of how we would like it to work – the possible negative effects of self-expression should be acknowledged more explicitly. Take, for example, this sentence from an early paragraph describing the forces at play in the system map in the simplest terms:

The impact [of the arts on a person] also flows back to the artist, directly in some instances (e.g., the artist sells a work of art) and indirectly through education, infrastructure, and society’s general embrace of creativity and freedom of expression.

In fact, creativity channeled through the arts does not always lead to an expansion of freedom of expression. Sometimes it leads to a backlash and a crackdown. We have seen widespread objections to certain works of art: the agency has been repeatedly accused of funding “pornography” (see reason number five of these Ten Good Reasons to Eliminate Funding for the National Endowment of the Arts). In 1998, after decades of periodic controversy over NEA grants, the Supreme Court ruled that the agency could make “decency” a consideration in funding decisions. This kind of stipulation doesn’t stop the production of controversial work, but it does send a message about the value of certain works of art, and art more generally to society, and it has real implications for artists’ ability to find financial support for their work. The impact that circles back to the artist is sometimes a structural restriction of creativity and expression.

Art itself can also be intrinsically harmful. The authors acknowledge in a footnote that, “[b]ecause communities do not all have the same values, ideals, or political inclinations, art that is seen as beneficial by one community can appear threatening to another.” But this understates the potential for art to do damage; for example, there is no shortage of racist music out there reinforcing and promoting prejudice. Finally, the arts can have unintended indirect effects, such as their ever-controversial role in neighborhood gentrification. Many observers have argued that  the influx of artists into an area can spur a kind of neighborhood change that is harmful to residents who subsequently are priced out of their homes or experience a more metaphorical sense of displacement. That kind of impact is not included in the benefits to society and communities structure, but nevertheless represents an example of a tangible outcome that could be caused by the arts.

Implications

So, what if it worked? What if we pursued this research agenda as laid out, and we really did have all of the metrics under all of the nodes defined in five years? We would know how things work now. But I can’t help wishing that this research agenda were more aspirational. If we had instead a working map of an ideal arts sector, could we be stronger at the end of five years instead of just smarter?

Such a map would show how and where we want average people to encounter the arts in their community, and what the impacts of those encounters would be. It would chart how public sector dollars can work with private and community foundations and individual donors to support a diverse and healthy arts infrastructure. It would theorize what tax structures and other policies could benefit individual artists, and those hypotheses would be tested in the research to follow, instead of simply stating that all of these elements exist.

Such a report would also draw on prior research in these specific areas. A literature review was conducted as part of the process of creating “How Art Works,” presumably for the purpose of assessing the current state of knowledge in the arts. But somehow those insights don’t seem to have made it into the final product, save for an annotated bibliography provided in Appendix A and sorted by map node. By drawing a map that got into details about what we already know and what we have yet to find out, “How Art Works” could have launched the sector into a new era of coordinated, proactive research.

Instead, we will have to be satisfied with the report’s more limited ambitions. By the time this research agenda was released, the NEA had already made an inaugural round of fourteen research grants. Fordham University studied the impact of arts programming on the social skills and mental health outcomes of at-risk youth, exploring the benefit of arts to individuals, society, and communities. Harvard looked into factors contributing to “birth” and “death” rates of arts and cultural institutions, which pertains to arts infrastructure. The University of Dayton studied the relationship between arts engagement and quality of life, investigating arts participation and arts creation. And the University of Georgia performed qualitative research to generate a hypothesis about community-built practice such as playgrounds, mosaic sculptures, murals, and community gardens, looking into the benefits of arts for societies and communities. This research is compelling, and the best examples will hopefully lead us forward as a field. Happily, the quality of that work will not be diminished by a grand vision that is, arguably, still under construction.

Further reading:

• Barry Hessenius, Research and Data Blogathon Day #4 (Barry’s Blog)
• Ian David Moss, Arts Policy Library: Gifts of the Muse (Createquity)
• Alan Brown, An Architecture of Value (WolfBrown)
• The Cultural Dynamics Working Group, The Cultural Dynamics Map: Exploring the Arts Ecosystem in the United States
• Ian David Moss, Live-blogging the “How Art Works” convening (Createquity)
• Keith Sawyer, How Art Works (Creativity & Innovation)
• Alexis Clements, A Grand Unified Theory of Art? (Hyperallergic)
• Mark Robinson, How Art Works? (Thinking Practice)

Infographic from “Why Contests Improve Philanthropy,” published by the Knight Foundation May 2013

A month later, Starr shot back with a post entitled “Dump the Prizes,” in which he eviscerated the weaknesses of contest philanthropy and jokingly suggested “mandatory jail time for crowdsourcing or crowd-judging.” Wrote Starr:

For social sector organizations, money is the oxygen they need to stay alive, so leaders have to chase prizes just like they do other, more sensible sources of funding. Some in the industry justify this as a useful learning process. It’s not. Few competitions (with some notable exceptions) provide even the most rudimentary feedback. Too many of these contests and prizes seem like they are more about the givers than the getters anyway. […]

The Hilton Humanitarian prize is a single winner-take-all award of $1.5 million to one lucky organization each year. With a huge prize like that, everyone feels compelled to apply (that is, get nominated), and I can’t tell you how much time I’ve wasted on fruitless recommendations. Very smart people from the foundation spend a lot of time investigating candidates—and I don’t understand why. The list of winners over the past ten years includes a bunch of very well-known, mostly wonderful organizations…

A lot of people argue that innovation competitions, challenges, and X Prizes are a vital part of that market and that they drive important advances that wouldn’t happen otherwise. I doubt it. There’s no real evidence for it, and I suspect that they do little more than speed things up a bit. The innovators I know do so to solve problems, not to win prizes. The only in-depth analysis of social impact contests I’ve seen was a 2009 McKinsey report, which began with a contests-are-wonderful perspective and carried on for 100 pages in the same vein without even a whiff of skepticism. Like many discussions of prizes, it confused anecdote with evidence and correlation with causation. We need a real study.

Not two weeks later, Patel was back with an on-point rebuttal that basically boiled down to, “we know that contests can be done badly; we’re talking about the benefits of good contests.”

A lot of this discussion reminds me of the perennial debate over the value of measurement and strategic frameworks like logic models. Details matter. It’s much more difficult to implement research-based frameworks thoughtfully than it is to implement them at all. But thoughtful or not, such frameworks can still be time- and labor-intensive to navigate, so when they are not implemented well you just make people’s lives more complicated without really adding any value. Similarly, prize philanthropy, especially when it devolves into popularity contest philanthropy, can easily do more to create stress than to improve outcomes.

I strongly share Patel’s belief in the value of open selection processes (such as those used in Knight’s contests). Nevertheless, I don’t think he comes out the winner here on every point. One of Starr’s key critiques is that, because of the competitiveness of open contests, applicants can invest a lot of time and energy in a game that they are very likely to lose. Indeed, the Knight Arts Challenge, one of the contests Patel mentions, has had over 13,300 applications in its various incarnations and funded only 257 winners, a success rate of less than two percent. (ArtPlace, whose initial formation was heavily influenced by Knight Arts VP Dennis Scholl, operates using a similar model.) While Knight takes care to minimize the administrative burden placed on applicants in the initial round, one wonders how much potential is wasted in the 98% of projects that don’t get funded.

Fortunately, as I mentioned, the report that started things off is pretty great – partly because of its interactive presentation, but more because it actually anticipates many of these issues and offers suggestions to address them. For example, some of the ideas Patel offers to make contests better include:

• Include other funders in your reviewer pool. You can share contest knowledge with them. They may fund ideas you don‘t.

• Treat your applicants as problem identifiers not just solution providers. Even entries that don‘t offer a feasible project idea worth funding still provide you with potentially useful feedback on the issues they think need fixing.

• Make it the default option that applicants post their entries publicly. In the news challenge applicants can opt to submit their entries privately, but generally over 90% of all submissions are posted publicly. Why? Because applicants see the benefits of attracting attention to their ideas and generating support.

It’s obvious that Knight is still figuring things out to some extent, even as it champions the virtues of open contests. But overall, I share the foundation’s faith in the format, especially the open-access nature of it. Kevin Starr brings up some good points, but his foundation doesn’t even accept applications…and he’s far from the only grantmaker who takes that approach. The more who do, the faster we end up with an environment that’s inhospitable to new voices that aren’t coming through the traditional networks. That may well be a problem for innovation in the social sector generally, but I would argue it’s especially important to keep those channels open in the arts, given our field’s emphasis on diversity of expression and individual creativity.

Since I didn’t write out the speech in advance, I don’t have a transcript for it. However, below is a reconstruction of the new material from my notes, so you can get a taste for it if you don’t have time to watch the whole thing right now. (You’ll notice I make a number of generalizations in the speech about the ways in which arts practitioners interact with research. These are based on observation and personal experience, and are best understood as my working hypotheses.)

*

[starting at 26:55]

Why is this integration between data and strategy important? Because research is only valuable insofar as it influences decisions. This is why logic models are awesome – they are a visual depiction of strategy. And there is no such thing as strategy without cause and effect. Think about that for a second. Our lives can be understood as a set of circumstances and decisions. We make decisions to try to improve our circumstances, and sometimes the circumstances of those around us. Every decision you make is based on a prediction, whether explicitly articulated or not, about the results of that decision. Every decision, therefore, carries with it some degree of uncertainty. This uncertainty can be expressed another way: as an assumption about the way the world works and the context in which your decision is being made. These assumptions are distinguished from known facts.

If you can reduce the uncertainty associated with your assumptions, the chances that you will make the right decision will increase. So, how do you reduce that uncertainty? Through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile. Alas, research and practice in our field are frequently disconnected in problematic ways. Six issues are preventing us from reaching our potential.

Issue #1: Capacity

Supply and demand apply as much to research as it does to artists. There are far more studies out there than a normal arts professional can possibly fully process. I wish I could tell you how many research reports are published in the arts each year, but nobody knows! To establish a lower bound, I went back over last year’s [2011] “around the horn” posts, which report new research studies that I hear about. I counted at least 41 relevant arts-research-related publications – a tiny fraction, I’m sure, of total output. To make matters worse, research reports are long, and arts professionals are busy. For the Createquity Writing Fellowship program, participants are required to analyze a work of arts research for the Createquity Arts Policy Library. I collect data on how long it takes to do this, and consistently, it requires 30-80 hours to research, analyze and write just one piece! Multiply this by the number of new studies each year, and you can start to see the magnitude of the problem.

Issue #2: Dissemination

Which research reports is an arts practitioner likely to even know about? Certainly not all of them, because there is almost no meaningful connection between the academic research infrastructure and the professional arts ecosystem. Lots of research relevant to the arts is published in academic journals each year, but unless the faculty member was commissioned to do their work by a foundation, we never hear about it. Academic papers are typically behind a pay firewall, and most arts organizations don’t have journal subscriptions. To give an example, after I wrote about Richard Florida’s Rise of the Creative Class, Florida pointed me to a study in two parts by two Dutch researchers. It’s one of the best resources I’ve come across for creative class theory, but I’ve never heard anyone even mention either study other than him and me.

Issue #3: Interpretation

Research reports inevitably reflect the researcher’s voice and agenda. This is especially true of executive summaries and press releases, which is often all anyone “reads” of research “reports.” Probably the most common agenda, of course, is to convey that the researcher knows what he/she is talking about. Another common agenda is to ensure repeat business from, or at least a continuing relationship with, the client who commissioned the study. The reality, however, is that research varies widely in quality. There’s no certification process; anyone can call themselves a researcher. But even highly respected professionals can make mistakes, pursue questionable methods, or overlook obvious holes in their logic. And, in my experience, the reality of any given research effort is usually nuanced – some aspects of it are much more valuable than others. Unfortunately, many arts professionals lack expertise to properly evaluate research reports, not having had even basic statistics training.

Issue #4: Objectivity

Research is about uncovering the truth, but sometimes people don’t want to know the truth. Advocacy goals often precede research. How many times have you heard somebody say a version of the following: “We need research to back this up”? That statement suggests a kind of research study that we see all too often: one that is conducted to affirm decisions that have already been made. By contrast, when we create a logic model, we start with the end first: we identify what we are trying to achieve and only then determine the activities necessary to achieve it.

Here are a bunch of bad, but common reasons to do a research project:

• To prove your own value.
• To increase your organization’s prestige.
• To advance an ideological agenda.
• To provide political cover for a decision.

There is only one good reason to do research, and that is to try to find out something you didn’t know before.

Issue #5: Fragmentation

The worst part of the problem I just described is that it drives what research gets done – and what doesn’t get done. There is no common research agenda adopted by the entire field, which is a shame, because collective knowledge is pretty much the definition of a public good: if I increase my own knowledge, it’s very easy for me to increase your knowledge too. The practical consequences of this fragmentation are severe. It results in a concentration of research using readily available data sources (ignoring the fact that the creation of new data sources may be more valuable). It results in a concentration of research in geographies and communities that can afford it, because people don’t often pay for research that’s not about them. And it results in a concentration of research serving narrow interests: discipline-specific, organization-specific, methodology-specific. My biggest pet peeve is that research is almost never intentionally replicated – everybody’s reinventing the wheel, studying the same things over and over again in slightly different ways. A great example of a research study crying out for replication is the Arts Ripple Effect report, which I talked about earlier. The results of that study are now guiding the distribution of millions of dollars in annual arts funding. Are those results universal, or unique to the Greater Cincinnati region? We have no way to know.

Issue #6: Allocating resources

Everyone knows there’s been a trend in recent years towards more and more data collection at the level of the organization or artist. Organizations, especially small ones, complain all the time about being expected to do audience surveys, submit onerous paperwork, and so forth. And you know what, I agree with them! You might be surprised to hear me say that, but when you’re talking about organizations that have small budgets, no expertise to do this kind of work, and the funder who is requesting the information is not providing any assistance to get it…just take a risk! You make a small grant that goes bad, so what? You’re out a few thousand dollars. The sun will rise tomorrow.

As an example of what I’m talking about, I participated in a grant panel recently. I enjoyed the experience, and am glad I did it, but there’s one aspect of the experience that is relevant here. There were seven panelists, and we were all from out of town. Each of us spent, I’d say, roughly 40 hours reviewing applications in advance of the panel itself. Then we all got together for two full days in person to review these grants some more and talk about them and score them. We did this for 64 applications for up to $5,000 each, and in the end, 92% 94% were funded.

So consider this as a research exercise. The decision is who to give grants to, and how much. The data is the grant applications. The researchers are the review panel. What uncertainty is being reduced by this process? How much worse would the outcome have been if we’d just taken all the organizations, put them into Excel, run a random number generator, and distributed the dollars randomly up to $5,000 per organization? And I’m not saying this to make fun of this particular organization or single them out, because honestly it’s not uncommon to take this kind of approach to small-scale grantmaking. And yet if you compare it to ArtPlace’s first round of grants, theoretically they had thousands of projects to choose from, and they gave grants up to $1 million for creative placemaking projects – but there was no [open] review process; they just chose organizations to give grants to. So there’s a bit of a mismatch in the strategies we use to decide how to allocate resources.

There’s a concept called “expected value of information” described in a wonderful book called How to Measure Anything, by Douglas W. Hubbard. It’s a way of taking into account how much information matters to your decision-making process. In the book, Hubbard shares a couple of specific findings from his work as a consultant. He found that most variables have an information value of zero; in other words, we can study them all we want, but whatever the truth is is not going to change what we do, because they don’t matter enough in the grand scheme of things. And he also found that the things that matter the most, the kinds of things that really would change our decisions, often aren’t studied, because they’re perceived as too difficult to measure. So we need to ask ourselves how new information would actually change the decisions we make.

There is so much untapped potential in arts research. But it remains untapped because of all the issues described above. So what can we do about it?

First, we need a major field-building effort for arts research. Connecting researchers with each other through a virtual network/community of practice would help a lot. So would a centralized clearinghouse where all research can live, even if it’s behind a copyright firewall. The good news is that the National Endowment for the Arts has already been making some moves in this direction. The Endowment published a monograph a couple of months ago called “How Art Works,” the major focus of which was a so-called “system map” for the arts. But the document also had a pretty detailed research agenda for the NEA, not for the entire field, that lays out what the NEA’s Office of Research and Analysis is going to do over the next five years, and two of the items mentioned are exactly the two things I just talked about: a virtual research network and a centralized clearinghouse for arts research.

This new field that we’re building should be guided by a national research agenda that is collaboratively generated and directly tied to decisions of consequence. The missing piece from the research agenda in “How Art Works” is the tie to actual decisions. Instead it has categories, like cultural participation, and research projects can be sorted under those buckets. But it’s not enough for research to simply be about something – research should serve some purpose. What do we actually need to know in order to do our jobs better?

We should be asking researchers to spend less time generating new research and more time critically evaluating other people’s research. We need to generate lots more discussion about the research that is already produced. That’s the only way it’s going to enter the public consciousness. Each time we fail to do that, we are missing out on opportunities to increase knowledge. It will also raise our collective standards for research if we are engaging in a healthy debate about it. But realistically, in order for this to happen, field incentives are going to have to change – analyzing existing research will need to be seen as equally prestigious and worthy of funding as creating a new study. Of course, I would prefer if people are not evaluating the work of their direct competitors – but I’ll take what I can get at this point!

Every research effort should take into account the expected value of the information it will produce. Consider the risk involved in various types of grants made. What are you trying to achieve by giving out lots of small grants, if that’s what you’re doing? Maybe measure the effectiveness of the overall strategy instead of the success or failure of each grant. This is getting into hypothesis territory, but based on what I’ve seen so far I would guess that research on grant strategy is woefully underfunded, while research on the effectiveness or potential of specific grants is probably overfunded. We probably worry more than we need to about individual grants, but we don’t worry as much as we should about whether the ways in which we’re making decisions about which grants to support are the right ways to do that.

Finally, we should be open-sourcing research and working as a team. I’m talking about sharing not just finished products and final reports, but plans, data, methodologies as well. I’m talking about seeking multiple uses and potential partners at every point for the work we’re doing. This would make our work more effective by allowing us to leverage each other’s strengths – we’re not all experts at everything, after all! And it would cut down on duplicated effort and free up expensive people’s time to do work that moves the field forward.

I thank everyone for their time, and I’d love to take any questions or comments on these thoughts about the state of our research field.

• The Los Angeles Times, via music critic Mark Swed, revives the Secretary of Culture talk, this time nominating Peter Sellars and Leon Botstein for the job. It’s an earnest appeal for an idea worthy of consideration, but if it was a political nonstarter four years ago, it’s hard to see how it has more legs now.
• Video game makers are finding themselves in the regulatory crosshairs after the Newtown shooting massacre, despite limited evidence that games played a role in motivating recent shootings. (Certainly a lesser role than…guns.)
• Margy Waller continues the charitable deduction discussion in a guest post over at New Beans.
• The NEA has relaunched the “Your Town” program as the Citizens’ Institute on Rural Design, which invites proposals for community-led workshops to improve quality of life in rural areas with the help of creative placemaking experts.

MUSICAL CHAIRS

• Well, that didn’t take long: Richard Dare, who just left the Brooklyn Philharmonic to take the top job at the New Jersey Symphony Orchestra, abruptly stepped down after a New York Times investigation into his past.
• Clayton Lord is leaving Theatre Bay Area, where he pioneered a number of research initiatives, to take the position of Vice President of Local Arts Advancement at Americans for the Arts. Lord replaces Mitch Menchaca, who left to become COO of Chorus America.

IN THE FIELD

• Joe Patti makes some astute observations about convergences between the arts and religion, including ways in which churches are getting into the placemaking game and the tidbit that newly minted pastors are more interested in starting their own flock than in joining established institutions. Fascinating stuff!

ALL ABOUT THE BENJAMINS

• The Kickstarter juggernaut continues on, raising an astonishing $274 million for projects in 2012. Yancey Strickler’s prediction that the company would outpace the budget of the NEA proved more than prescient. The Economist has more.
• Artspace, the nonprofit artist housing developer, is the recipient of a new $2.7 million grant from the Ford Foundation. Createquity previously reported on a $3 million program-related investment Artspace received from Ford in 2011.
• “Household incomes in Manhattan are about as evenly distributed as they are in Bolivia or Sierra Leone” – the New York Times on what it means to be “middle class” on the country’s most expensive island.
• Rob Dietz and Dan O’Neill wonder why we haven’t yet given up on GDP.
• Need lots of money for a wacky, wildly ambitious and possibly ill-advised art project? Britain’s Artangel might just be your ticket.

BIG IDEAS

• I think Diane Ragsdale has become the arts sector’s “questioner-in-chief.” She’s back after a monthlong hiatus with a doozy of a question.
• Michael Hickey ruminates on scale, efficiency, and what size grants would encourage the most art created per dollar. (But is it about the most, or the best?)
• A year in the life of Collective Impact.
• Phil Buchanan, writing at the Center for Effective Philanthropy blog: “I think this whole aversion to dependency may be yet another example of where analogies to the for-profit world have created confusion in our sector. We’re so enamored with market analogies that we can’t get our heads around the fact that certain work simply requires ongoing philanthropic support. Other than large-scale government support, there is no ‘exit’ event on the horizon for nonprofits, no analog to the IPOs that allow early private sector investors to cash in and get rich.”
• I wholeheartedly endorse this pledge on Andrew Taylor’s part not to discuss “generic topics” in 2013. Taylor identifies three in particular that provoke discussions with “lots of heat, but not much light”: business models, advocacy, and value.

RESEARCH CORNER

• A new research study from Germany appears to show a causal relationship between music training and “verbal memory” (and by extension, speech and language processing). The study seems well-designed although the sample is on the small side.
• The New York Public Library system appears to be more vital and in demand than ever despite budget  cuts over the past decade, according to a new report from the Center for an Urban Future.
• A woman named Amy Webb found her life partner online…data scientist style. She conducted a detailed study of popular women’s behavior on JDate by creating 10 fake male profiles and interacting with 96 women over the course of a month, taking notes all along. It’s all ever so slightly unethical, but it does make for some, ah, engaging reading.
• This is a great example of how a theory of change can illuminate assumptions needing testing. Do more entrepreneurs and more companies = more innovation?

ETC.

• In 2011, Createquity shared the story of Aaron Swartz, who had been indicted by federal authorities for “liberating” some 4.8 million academic journal articles from the online database JSTOR in an act of protest against restrictive copyright policies. In a sad coda, the 26-year-old Swartz, who suffered from depression, committed suicide this month in the face of a potential 35-year prison sentence (although the prosecutor in the case claims that she was only seeking six months). Among the other tributes making their way around the web, it turns out that GiveWell’s Holden Karnofsky (of course) had developed a personal friendship with the kid supergenius.

• As you know, there was an election last week, and Barack Obama won it. Thankfully this means that Barry Hessenius’s worst fears about the NEA likely won’t be realized, but Barry does have some useful advocacy advice that is worth a read regardless of the outcome. Ted Johnson has a helpful pre-election analysis of issues relevant to Hollywood in the election. Americans for the Arts has been active too: Jay Dick offers a post-election advocacy to-do list, and the Arts Action Fund offers a thorough roundup of the election results and their implications. Among the lesser-known developments include the fact that many moderate Republican legislators in Kansas who stood up for arts funding in that state lost their primaries to more conservative challengers; similarly, several pro-arts Republicans in Congress have either retired or lost their seats, further polarizing the parties in their orientation toward arts funding. On the plus side, two cities – Portland, OR and Austin, TX – passed pro-arts ballot measures.
• The final version of the Chicago Cultural Plan has been released – with a new arts education plan for Chicago Public Schools to boot.

ALL ABOUT THE BENJAMINS

• RIP Walter Carsen, one of Canada’s most prominent arts philanthropists.
• This is a truly thorough overview of the arts funding ecosystem in metropolitan Atlanta, both past and present. A must-read if you have any plans to work in the arts there.
• The Knight Foundation is stepping up its commitment in Detroit with a $20 million round of grantmaking.
• If you work for an arts funder and you’re reading this, can you do me and the entire world a giant favor and make sure your organization is giving the Foundation Center your grants data? They are making it easier and easier to participate, and it ultimately helps researchers like me make sense of the arts funding landscape. You can help them refine their grants taxonomy as well.

IN THE FIELD

• Orchestral musician labor disputes are in the news again, and nowhere is the hotbed hotter than in freezing Minnesota, where both the Minnesota and St. Paul Chamber Orchestras face work stoppages. Eric Nilsson says neither side is fully accepting reality, and even the Minneapolis City Council is getting involved. Both groups have canceled performances through the end of 2012, and musicians are starting to look for jobs elsewhere. Meanwhile, the Spokane (WA) Symphony is on strike and canceling performances.
• I’m intrigued by this announcement of SphinxCon, a new diversity summit organized by Sphinx, a Detroit-based organization dedicated to cultivating more musicians of color in classical music. Aaron Dworkin and company have managed to pull together a pretty incredible speaker list pairing (mostly white) arts service organization leaders with a largely non-white group of artists, academics, and other perspectives. Who knows if it’ll lead to anything, but it seems like the ingredients for a real conversation are there.

BIG IDEAS

• I can’t emphasize enough how important this 282-word blog post from Adam Thurman is. Adam has a gift for concision, and his three-part distinction between making art, making money doing art, and making a living from art is essential for artists and policymakers alike. And speaking of Adam’s genius, this post on arts marketing (featuring the memorable quotes, “[Y]ou are probably ok with whatever you did last night.  Maybe you watched TV, maybe you read a book, maybe you got drunk and did lines of cocaine.  Whatever you did, you were ok with it.” and “The reality is that if these [new] audiences never come your way they will be fine.  You, on the other hand, will be in serious trouble.”) is well worth a read too.
• Stephanie N. Stallings thinks jazz could use some binders full of women and speculates that hip-hop has overtaken it as America’s greatest cultural diplomacy tool.
• Over at Next American City, Neeraj Mehta considers the “who” of creative placemaking (as in, “who benefits?”).
• So Google’s getting into the virtual museum business now?
• Online higher education banned in Minnesota, then reinstated.
• Chad Bauman writes eloquently on the symbiosis between an arts community and its local newspaper – and what it means that so many of those newspapers seem to be hanging on by a thread.
• Eric Booth submits a lengthy dispatch from the first international Teaching Artist Conference in Oslo.

RESEARCH CORNER

• A new report from Emerson College’s Center for the Theater Commons, authored by Diane Ragsdale, examines the relationship between nonprofit and commercial theater.
• Chorus America has released its Choral Operations Survey Report for 2012.
• I’m looking forward to seeing the results of what looks like a very strong study being undertaken by the LA Philharmonic, USC, and Heart of Los Angeles to investigate the impact of early childhood music training. Meanwhile, a just-released report from Carnegie Hall and WolfBrown examines the potential for music to make a difference in the juvenile justice system.
• If you’ve ever doubted me that logic models matter, check out this analysis of the difficulties faced by One Laptop Per Child, a hugely ambitious, billion-dollar initiative to develop and distribute low-cost laptops to schoolchildren in developing countries. The passage below is an eloquent depiction of how failing to think through the details of a strategy can mean its doom:
  

  Doing an end-run around lousy infrastructure and poorly-trained teachers might actually work with the right support to guide the child’s learning. Unfortunately, Negroponte has also stated that you actually can give a kid a laptop and walk away.

  According to Jeff Patzer, a former OLPC intern, that’s precisely what they did in Peru. Hardware degraded faster than expected, and OLPC allowed Peru to build its own branch of the system software that was incompatible with patches. Interns were not prepared to educate teachers, and teachers were not prepared to use the XO to teach students.

  “The only thing that happens is the laptops get opened, turned on, kids and teachers get frustrated by hardware and software bugs, don’t understand what to do, and promptly box them up to put back in the corner.” Patzer explained.

ETC.

• Joe Queenan on having read more than 6000 books. My favorite part of this column is the fact that, because it’s in the Wall Street Journal, his offhand mention of Williams Sonoma is accompanied by its latest stock quote.

[7:29] Oops! I ran out of juice (the electrical, not the metaphysical kind) just as things were wrapping up. Anyway, I thought Andrew Taylor did a great job of moderating that last panel and pointing out some of the more interesting features of the model as it’s been developed. At the end of the day, as he pointed out, this was developed to satisfy a federal OMB requirement for all departments to have logic models describing what they do. And since the NEA now has that, it’s mission accomplished in a sense. But I can’t help but feel that this was a missed opportunity to dig deeper than we’ve yet seen into the detailed dynamics of how arts work. Tony Siesfeld at one point put up a slide of an insanely complex drawing by a RISD student to illustrate what the model could have looked like had they tried to map everything, implying that this was something to be avoided. But honestly, I think we need that complexity – or at least, that’s where the true value of these efforts lies, since no one seems to be eager to take it on. I don’t have any gigantic issues with the model that was shown today other than that it seems, well, kind of obvious. It’s the other aspects of the presentation – particularly how the model fits into the NEA’s research agenda, and how the NEA has used it as a framework for gap analysis – that intrigue me more.

[5:08] Kushner: same mapping process will work for other countries, but values attached to different nodes may be different. (But that would require values to be attached to the nodes in the first place!)

[5:00] This panel is really elevating the conversation. Making a valiant attempt to connect the map to everything else.

[4:49] Roland Kushner: who are the audiences for this research? Are we the wrong people? Also struck by relative absence of dollars in the system. If these benefits are flowing one way, what’s flowing the other way? Also wishes that it said more about aesthetics.

[4:47] Andrew Taylor loves the fact that “Human Impulse” needs to “jump” over the system in order to get into it.

[4:45] Roland Kushner: what happens when we start to fill the holes? What do we do with it next?

[4:44] Lots of praise for the NEA’s research work on this panel, which consists of David Fraher, Roland J. Kushner, and and Kathy Dwyer Southern. They’re particularly responding to the fact that the agency is reaching out to other arms of government and trying to do work in context.

[4:36] some chatter about the “Education and training” bucket. Does it refer to education and training of artists or of the general public? Also, how does this translate between cultures?

[4:30] Questioner making the point that the map placing the arts in the center is perhaps too comfortable for artists. Anne L’Ecuyer says it’s “mapping the artistic ego.”

[4:10] Ximena Varela questioning subsuming of economic impact into the larger “benefit to society and communities” category in the main map.

[4:03] John Borstel making useful analogy of the map to anatomy – understanding how all the parts work together.

[3:37] Creating a taxonomy or “menu” of arts benefits also seems like a solved problem. Gifts of the Muse explored this territory seven years ago, and I see it is heavily cited in the report. I also like Alan Brown’s “Architecture of Value.”

[3:35] My thoughts: I’ll be interested to dig into the report further, but from this presentation I’m surprised that it doesn’t get into more detail. The result is, as the first questioner pointed out, a very highly abstracted framework that seems like an anticlimactic result for a year’s work.

[3:30] OK, break time. Next panel: Impacts on Individuals, moderated by Anne L’Ecuyer, with panelists John Borstel, Shahin Shikhaliyev, Ximena Varela.

[3:24] Regarding the virtual research network, sometimes NEA feels like they’re a step behind the field – they’re especially wanting to tap into international dialogue about arts research.

[3:20] Joan Jeffri, commenting from the audience, commends everybody on job well done, makes plea for continuing to involve graduate students.

[3:17] Sunil ending with a couplet for Joan Shigekawa via Frost: “We dance around in a ring and suppose. But the Secret sits in the middle and knows.”

[3:16] Sunil: The “dark horse” of the report is the appendix listing resources and data sources, available online here.

[3:13] Cross-cutting projects to support capacity building include research grants, an online data repository, and a virtual research network.

[3:11] Sample NEA research projects for second-order outcomes of the arts (societal capacities to innovate and express ideas; outlets for creative expression; new forms of self-expression): The Arts, New Growth Theory, and Economic Development; Analysis of Arts Variables in the Rural Establishment Innovation Survey; Study of Design Patents and Product Innovation. Shortage of high quality research in this domain.

[3:09] Sample NEA research projects for direct and indirect economic benefits, benefit or art to society & communities, benefit of art to individuals: NEA-NIH-NAS Public Workshop and Paper Series on the Arts and Aging (completed); National Children’s Study Arts/Music Supplement; Arts and Livability Indicators; Arts and Cultural Production Satellite Account

[3:07] Sample NEA research projects for arts participation: An Average Day in the Arts (completed); SPPA 2012 First Look, Summary Report, and Monograph Series; GSS Arts Supplement Report and Monograph(s)

[3:05] Sample NEA research projects for arts infrastructure and education and training (input variables): Artists and Arts Workers in the United States (completed); How the United States Funds the Arts (3^rd edition); Understanding Arts Education Access by School and School District Characteristics

[3:04] NEA says that they’ve taken the map and painted upon it the NEA’s research projects over the next five years. “Arts participation” has five projects, “arts infrastructure” has four, “benefit of art to individuals” has 10.

[3:01] NEA has a five year strategic plan whose research goal is to promote knowledge and understanding about the contributions of the arts. Emphasizing “contribution” again. Also distinguishing between evidence of the arts’ value vs. impact. Value is primarily internal, impact refers to effects on arenas outside the arts.

[2:59] Over to Sunil Iyengar, Director of the NEA’s Office of Research and Analysis, who will talk about how this work fits in to the NEA’s research agenda.

[2:57] Audience member/choreographer expresses gratitude for having art at the center. Alludes to shift in funders’ perspective.

[2:55] AFL-CIO’s David Cohen delivers a provocative question: what happens if you take out the word “art” and replace it with “science”? Points out the very high level of abstraction. Siesfeld does say that they tried it with “sport.”

[2:53] Obligatory “more research is necessary” message delivered: Siesfeld calling the map “a beginning, not an end.”

[2:48] Key findings of the project, as reported by Siesfeld:

1. System map reflects several key “truths” reaffirmed throughout this collaborative research project:
   • Nice to see that art is at the center.
   • The human impulse to create is a key driver/input.
   • Benefits are not always equally distributed and not always positive.
   • Art makes important contributions to quality of life. Not a cause per se, but can enhance or detract.
2. System map provides “negative capability” – can imagine the system without having to resolve apparently contradictory aspects. E.g., is it high art or low art? In the map, it doesn’t matter. You could look at symphonies or mashed-up music, can test the same things.
3. The system map does provide an integrative & holistic framework for organizing research and measurement for arts impact.

[2:46] Whoa, Siesfeld just spoke of gentrification and artist colonization in Oakland as a “net positive.”

[2:43] Discussing system “multipliers” – factors that hit the system and influence it: markets/subsidies, politics, technology, demographics/cultural traditions, and space & time.

[2:38] Siesfeld taking the position that this is a model of “contribution, not attribution.” There are many factors that can improve quality of life, art is one of them.

[2:38] Siesfeld notes that lots of attention is being paid right now to the economic benefits of the arts.

[2:37] Here’s a link to the report, which includes the map in question.

[2:33] OK, now we’re getting to content. Map has arts participation at the center, with arts creation as a subset of participation. It divides first-order benefits into society/communities (economic & civic) and individuals (cognitive & emotional).

[2:28] The process took nearly a year, and the map went through a number of revisions based on feedback from practitioners. Initial version was very simple, looking almost like a heart, but gradually got more complex as more elements were added.

[2:22] Siesfeld quoting Hewlett Foundation ex-president Paul Brest on theories of change. Initially they thought about doing a traditional theory of change, but quickly decided that what was being described was far too complex for TOCs’ left-to-right structure. So they wanted to incorporate non-linear elements, and thus went with systems mapping.

[2:18] Tony Siesfeld from Monitor Institute is now presenting the map (which, to be clear, is a conceptual map rather than a geographic map). The goals of the project are to “establish a measurement framework for assessing arts’ impact over time” and advance our understanding of relationship between the arts and key domains such as creativity, sustainable communities, health and wellness, economic prosperity. It’s also designed to flow into a research agenda by identifying which areas of the map have lots of research available, and which have less.

[2:15] The systems map so far has been variously referred to as “both too complex and too simple” and “productively inaccurate.” Reminds me of the maxim that “all models are wrong, but some are useful.” In any case, interesting to see the crew downplaying things a bit.

[2:03] Starting off with introductions from Sherburne Laughlin, Phyllis Peres, Peter Starr, and Rocco Landesman. Two simple questions: 1. What is art? 2. What exactly happens when art happens?

I’ve gotten away from event blogging for the most part, but I thought I would emerge from semi-retirement for this one since it’s the most public forum to date at which the National Endowment for the Arts has shared its plans for research. If you want to follow along, you can do so at the Twitter hashtag #HowArtWorks or watch the livestream here. Keep refreshing for updates, which will appear in reverse chronological order.

Now that the talk has subsided a bit, it’s a good opportunity to clarify and elaborate upon some of the subtler points that I was trying to make in that piece, which definitely left a bit of room for people to read in their own interpretations.  So, just to be clear: despite the provocative title, I wasn’t trying to slam the practice of creative placemaking itself, nor call it into question as a focus area for policy and philanthropy in general. As I wrote in response to one of the comments on the original post, I believe strongly in the power of the arts to have a role in revitalizing communities, and I view the desire to direct resources toward bringing such efforts to life as a very positive impulse on the part of funders and policymakers. Furthermore, although I agree with the point made by John Shibley and others that the arts may not be the best way to foment economic development, no one said that cities and regions can only use one strategy. Economic development is a complex beast, and intuition and common sense would hold that there are most likely some specific situations in which the arts can have a real, irreplaceable, and catalytic impact.

My critique is really about how we don’t have much information about what those situations are – nor about how infusions of philanthropic capital can make a difference in those situations. What’s more, I am not confident that the tools we’re currently developing, as useful as they may be for other purposes, will get us there on their own. My contention is that logic models and their conceptual cousins, theories of change, can be useful tools in filling this gap – by forcing us to articulate our assumptions about the way the world works, and by providing a framework that we can use to test those assumptions. The problem is that most of the logic models that I see aren’t worked out to the level of detail that I believe is necessary to gain really useful information about the dynamics of these complex processes. In my post, I provided a couple of examples of theories that, while surely far from perfect, at least attempt to recognize some of the numerous and interlocking assumptions embedded in grantmaking of the kind engaged in by today’s funders supporting creative placemaking.

It’s clear from some of the responses, however, that not everybody shares my optimism in the utility of logic models. Laura Zabel writes that she “hates” them for being too reductive. Diane Ragsdale, taking a cynical view, is worried that they may be misused by funders in order to make themselves seem smarter than they really are or blame grantees for failed strategies. ArtPlace’s response suggests that logic models raise the bar for research too high, and that because proving a causal connection between these investments and the change they produce (or don’t) is so difficult, we’re better off not trying. While I can sympathize with each of these critiques, I also think that they give logic models a bad rap. I feel that logic models are a tool of tremendous power whose potential is only beginning to be unlocked. It’s true that, just like philanthropy and policy, logic models can be done very badly. But that doesn’t mean there’s no gain for us in trying to do them well.

Before we get into all that, however, I’m guessing that some of you probably could use a refresher course on logic models and the terminology associated with them, which can be quite confusing. So let’s start with a little background on what this is all about.

What the Hell Is a Logic Model, Anyway?

Simply put, a logic model is a method of describing and visualizing a strategy. Logic models have their conceptual origin in the “logical framework approach” originally developed for USAID in 1969 by Leon J. Rosenberg of Fry Consultants. Their use was largely concentrated in the international aid arena until 1996, when United Way of America published a manual on program outcome measurement and encouraged its hundreds of local agencies and thousands of grantees to adopt logic models as a matter of course. Since then, large private funders such as the Kellogg and Hewlett Foundations have integrated logic models into their program design and execution, and the concept is commonly taught in graduate programs in public policy, urban planning, and beyond.

Even though logic models have achieved greater adoption over the past several decades, there is little standardization in the content, format, and level of ambition seen in professionally produced logic models for institutions large and small. Worse, everyone seems to want to come up with their own terms to describe features of the logic model, and as a result, you’ll notice a lot of variation in language as well. Below, I’ll do my best to isolate the elements that most of these efforts have in common.

Nearly all logic models contain the following fundamental elements. In combination, they describe a linear, causal pathway between programs or policy interventions and an aspirational end-state.

• Activities are actions or strategies undertaken by the organization that is the subject of the logic model. These activities usually take place in the context of ongoing programs, although they can also be one-time projects, special initiatives, or policies such as legislation or regulation.
• Outputs refer to observable indications that the above activities are being implemented correctly and as designed.
• Outcomes are the desired short-, medium-, or long-term results of successful program or policy implementation.
• Impacts (or Goals) represent the highest purpose of the program, policy, or agency that is the subject of the logic model. Sometimes you’ll find these lumped in with Outcomes.

Logic model for a bicycle helmet public information campaign, courtesy RUSH Project

Many realizations of logic models combine these essential elements with additional information that provides contextual background for this causal pathway. Several of these supplemental concepts are listed below in approximate order from most common to most obscure.

• Measures (or Indicators) for outputs, outcomes, and impacts are concrete, usually quantitative data points that shed light on the degree to which each result has been achieved.
• Inputs are resources available to the program or organization in accomplishing its goals.
• Assumptions are preconditions upon which the model rests. If one or more assumptions proves unsound, the integrity of the model may be threatened.
• Benchmarks extend the concept of measures to incorporate specific target goals (so, not just “# of petitions delivered to Congress” but “50,000 petitions delivered to Congress”).
• Target Population refers to the audience(s) for the activities listed in the logic model.
• Influential Factors are variables or circumstances that exist in the broader environment and could affect the performance of the strategy as designed (e.g., an upcoming election cycle whose outcome might change the underlying landscape in which the program operates).

What About Theories of Change?

A frequently-employed alternative logic model approach strips out this latter set of contextual elements and instead aims to visualize the linear causal chain at a finer grain of detail. This version of a logic model is typically referred to as a theory of change (or, sometimes, a program theory). A well-executed theory of change diagram “unpacks” the processes and factors that lead to successful outcomes, exposing relationships between isolated variables that can then become the subject of research or evaluation.

Partial theory of change from Project SuperWomen case study (ActKnowledge/Aspen Institute Roundtable on Community Change)

Sometimes logic models and theories of change are presented as distinct concepts, while other times they really refer to the same thing. This is because logic models and theories of change evolved out of distinct communities of practice, but the philanthropic field has not always respected the distinction in the terminology it’s adopted to describe these tools. In my own practice I prefer to use theories of change, but for the sake of simplicity and readability, in the rest of this article I’m going to use the term “logic model” inclusively to refer to any diagram that clearly shows some combination of activities and outcomes, regardless of what other elements it may include or the visual approach it takes.

*

OK, now that we have our definitions in order, we can start talking about what makes logic models so awesome.

Awesome #1: Logic Models Describe What’s Already Going On in Your Head

So, here’s the thing: the core questions involved in creating any logic model—What am I trying to do? Why am I trying to do it? How will I know if I’ve succeeded or not?—represent the very essence of strategy. As a rabbi might say, “the rest is commentary.” If you have a strong sense of what the answers to these questions are, then you have a logic model in your head whether you realize it or not. All the diagram does is make it explicit.

To illustrate this, we can look at a simple example. Let’s say I decide I’m done with this whole “arts” thing and I want to go to law school instead. I know, though, that in order to get into a good law school I need to get a good score on the LSAT. So, how can I make sure I get a good score? Intuitively, I decide that taking a test prep class is the way to go.

Why do I think taking a test prep class is a way to increase my score on the LSAT? Well, if my score isn’t as high as it could be, it’s probably due to some combination of two factors. First, I may not know the material well enough. So, if the class helps me to learn how to answer the test questions better, I’ll likely perform better on the test. Second, there may be a psychological factor as well. If I’m someone who gets nervous on tests, then my performance on them may suffer. The practice exams and deep engagement with the material that comes with a class could help me to get more comfortable with the idea of the LSAT and make it seem less intimidating, thus improving my performance.

Seems logical enough, right? And voila, it lends itself quite easily to a logic model:

The truth is that any decision you make, if it has any element of intentionality at all, can be diagrammed as a logic model. You might hate logic models with every fiber of your being and think they’re the stupidest thing ever created, but I’m telling you right now: if you believe in strategy, then you believe in logic models.

Awesome #2: Logic Models Are Incredibly Flexible

Now, there’s a difference between having a logic model in your head and having a good logic model in your head. The example above is simple, but it’s limited by that simplicity. It doesn’t explain why I might have decided to go to law school, or explore other ways that I could get into the school I want besides increasing my test score. In short, it’s pretty much just a straight-up mapping of a decision already made.

The best logic models don’t do that. Instead, they proceed with the end in mind (what is the goal we want to achieve?) and then methodically work backwards to understand what activities or strategies would be most appropriate to achieve that end. The ultimate outcome of this exercise may be a very different set of strategies than the ones you were originally contemplating or the programs you already have in place! Because of that, the logic model creation process can be great for opening up new ways of thinking about old problems or longstanding dreams, as well as clarifying what’s really important to you and/or your organization.

I mentioned earlier that not everyone is a fan of logic models. Here’s what Laura Zabel had to say about them in her post responding to mine:

I hate logic models. For me they are, somehow, simultaneously too reductive and too complex. Too simple, too linear for how I think the world works and too dry, too chart-y for how beautiful the world is. They make me irrationally grumpy.

Arlene Goldbard, in a 2010 essay, is similarly grumpy about logic models:

[R]equiring one of these charts as part of a grant proposal bears about as much real relationship to community organizations’ work as would asking each to weave a placemat…[T]he task of boiling the answers down to colored bars often wastes days, compressing most of the useful meaning out of the inquiry.

I can’t speak to Laura’s and Arlene’s experiences directly, but I know they are not uncommon. Unfortunately, logic models that are rife with imprecision, questionable assumptions, and inappropriate associations are more frustrating to work with than no logic model at all—and it’s not as easy as it looks to create logic models that are free of these flaws. Such problems are magnified when logic models are treated as edicts sent down from on high rather than the learning, living documents that they are intended to be.

In her post, Laura presents an alternative formulation of a logic model that describes her theory of change for creative placemaking: artists + love + authenticity -> creative placemaking. While I’d classify this as more of a definition of creative placemaking than a logic model, it goes a long way toward illustrating my point that we all have latent logic models in our head that are just waiting to be expressed as such. Laura writes, “there’s no logic model in the world that can capture how a crazy parade [the annual MayDay parade in Minneapolis] can restore my faith in humanity.” I couldn’t disagree more – in fact, I made one, relying solely on the way Laura describes the parade in her post. Here you go:

For the past few months, I’ve been researching impact assessment methods used across the social sector in connection with some evaluation work we’re doing here at Fractured Atlas. Seemingly every year, someone comes up with a new way of evaluating impact, whether it’s for social purpose investing, choosing grants, or measuring externalities. I’m not done yet, but what I’ve found so far has only reinforced my appreciation for the logic model. The beauty of logic models is that, because they relate so directly to the fundamental elements of strategy, they are endlessly adaptable to almost any situation. I actually find it kind of funny when people call logic models too rigid, given the alternatives – especially considering how much of our lives is ruled by the granddaddy of rigid, one-dimensional success metrics: money.

Awesome #3: Logic Models Are a Victory for Transparency

Hewlett Foundation Performing Arts Program strategic framework (co-developed in 2008 by yours truly)

One of the really powerful things about downloading the implicit strategy that exists in your head into a diagram is that it confronts you with gaps that may be present in that strategy and allows you to try and work through them. The Underpants Gnomes example that I used in my creative placemaking post is a great illustration of this. The Gnomes clearly felt that stealing underpants would lead to profit, but hadn’t clearly thought through the fuzzy middle part of the scheme. The Underpants Gnomes might seem like a fanciful exaggeration of the problem I’m talking about, but I would argue that there’s been more than one arts organization (and funding initiative) started with a similar lack of congruity between the proposed activities and intended results.

Diane Ragsdale is skeptical that logic models can serve this function, suggesting that funders might misuse logic models and turn them against their grantees. First, I think it’s important to make a distinction here: Diane and Arlene and Laura are all talking about logic models at the level of the individual organization/project. While I think these can be helpful, in my mind the most important logic model is the one for the funder itself. This is admittedly a rarer practice, but several foundations – such as Ford, Hewlett, McKnight, and Boston – have taken the steps of not only developing a logic model to describe their grantmaking strategies, but sharing that logic model publicly. Those that don’t publish at least sometimes make them available to peers outside the organization. Once a logic model is “out there,” there’s no taking it back. Therefore, logic models both pull back the curtain on a funder’s current thinking and also make it harder to project the illusion after the fact that a funder knew how things were going to work out all along.

Even more importantly, though, logic models are a victory for transparency with oneself, not just with others. The most important part of any logic model creation process is the set of assumptions revealed about how your program or organization works, and what it needs to be successful. Sometimes these assumptions might seem like no-brainers, and other times they will seem as unproven as they are central. Being comfortable with naming your assumptions as such is not just good practice for ensuring that your organization is constantly learning and growing. It’s also extremely helpful on a psychological level for dealing with the specter of possible failure. Because logic models explicitly draw a distinction between program design and program execution, they acknowledge the very real possibility that you could be doing your job perfectly and your program could still fail, because its theoretical foundation rests upon faulty assumptions. This is an incredibly freeing realization, because it means that radically changing or even scrapping a program that isn’t working doesn’t necessarily have to mean changing program leadership.

One of the reasons people sometimes feel anxious about evaluation and measurement is because they’re afraid of being held accountable, especially to things that they don’t have full control over or to metrics that don’t seem relevant to what they’re trying to do. When that happens, there are enormous incentives on managers and their supervisors to “cook the books” or otherwise game the system to show results that look better than reality, because any failure—even failures that are no one’s fault—reflects on them personally. That’s the danger of trying to enforce a data-driven culture without first developing the theoretical frameworks that determine what data you’re trying to collect. Because logic models separate the person from the program, they can distinguish between lagging initiatives that might just need more time to prove themselves, and failures of design that can be transformed into productive learning opportunities.

A Note About Logic Models and “Proof”

One of the criticisms directed at logic models generally, and by ArtPlace at my post specifically, is that they promote an impossible standard for proof. Here’s what ArtPlace had to say about it:

A critical limitation of elaborate logic models of the style developed by Moss is that it is nearly impossible to quantify or measure all of the different factors and relationships proposed. While many of the asserted relationships are plausible…almost none are measured in practice. Many – if not most – of the variables…could only be gauged imprecisely at great expense or are not susceptible of measurement at all. Multiply this problem by the 50 variables the model uses and the dozens of relationships it asserts, and it’s clear that it is beyond anyone’s ability to actually prove or disprove the model for even a single metropolitan area, much less the nation.

In one sense this criticism is right: these are difficult research challenges that highly qualified professionals have been struggling to address for decades, some of the more promising approaches demonstrated at the recent NEA/Brookings Institution convening on economic development and the arts notwithstanding.

But contrary to popular perception, the term “scientific proof” is a misnomer: proof is a concept germane to mathematics, not science. (Admittedly, my previous post could have been clearer on this point.) Scientists, including social scientists, develop hypotheses about how the world works and then gather evidence to support or undermine those hypotheses. Whereas proof is black-and-white, evidence has shades of gray: it can be strong or weak, circumstantial or conclusive. My colleague Kiley Arroyo made a great courtroom analogy in response to my creative placemaking post: she wrote, “think of forensic analysis if you will. You’re not just going to look at where and how the bullet hit, but what it was shot from, where, by whom and why.” Our job as researchers is not to “prove” anything – instead, it’s to amass evidence in search of the truth.

The biggest problem that I see with most logic models is that they are too simple for what they are trying to describe, and thus consign us to amassing a whole bunch of weak evidence. Logic models are often developed as much for communication purposes as for research, and can thus face intense pressure to be “dumbed down” for public consumption. I frequently hear comments like, “make sure that there are no more than five categories, because after that you’re going to lose people,” and God help you if your diagram doesn’t all fit on one page. But think about it, would you apply this standard to a budget? To a work plan? For a major, mission-critical project with many moving parts? I’m not interested in logic models as a communication tool; I’m interested in them as a means to help us do our jobs better.

In the case of ArtsWave, we do intend to collect data to show progress towards the goals that have been established through the model. But the task is less daunting than it looks. Because we are not trying to “prove” the model, not everything has to be measured directly; indeed, not everything has to be measured at all! Instead, the model serves as a road map for us in considering what is most important to measure, given what we don’t know. Is it that assumption that the arts can differentiate Cincinnati from its peer cities in the minds of tourists and potential residents? Can we be confident that people from diverse backgrounds will interact with each other if they happen to come together for the same community-wide event? A smart research design will test the assumptions that are most in need of testing, and the purpose of the modeling exercise is to identify which assumptions those are.

It’s not like we are all alone in this effort, either. There is an ever-growing body of literature on the ways that the arts interact with communities, and there is no need for us to demonstrate yet again connections for which strong evidence exists in other contexts. Furthermore, since many of the data points involve stakeholders beyond the arts, there is an opportunity to collaborate with other local entities to share resources and develop knowledge infrastructure collaboratively. Cincinnati is home to the STRIVE initiative, which has been made famous in the broader social sector as the poster child for the “Collective Impact” concept as coined by consultants from FSG Social Impact Advisors. One of STRIVE’s chief accomplishments has been the development of a “adaptive learning system” for use by hundreds of education nonprofits in the region, which has helped align the efforts of those organizations around a common set of purposes and benchmarks. If it worked in education, and in the same city no less, why can’t it work in the arts?

But that being said, none of the three benefits that I’ve cited above—articulation, flexibility, and transparency—require “proving” the logic model. They all come, at least in part, just from creating one. And creating a logic model doesn’t have to be a tortured, involved process. It doesn’t have to cost hundreds of thousands of dollars. You can write them out on a back of a napkin (okay, for some things you might need a really big napkin). Creating a logic model for something that will require a lot of your time or money is one of the most highly leveraged activities you can possibly undertake. I hope more arts funders will undertake it.