Museum Camp is a creation of Nina Simon and the Santa Cruz Museum of Art & History. Createquity readers might recognize Nina and her fantastic work at Santa Cruz MAH from such Top 10 Arts Policy Stories posts as 2012‘s, not to mention many shout-outs before and since in blog posts here and there. Nina used to be a rockstar experience design consultant in the museum field and earned a measure of fame at the beginning of this decade as the author of The Participatory Museum, which you can read online for free. A couple of years ago, she decided to take the job as director of the Santa Cruz MAH, and she and her team have been up to amazing things since then, including a previous version of Museum Camp that sounded like pretty much the most fun anyone has had in a museum ever.

All that fun ultimately adds up to something significant, though, and it’s important to be able to describe what’s meaningful about what we do effectively and convincingly to people who weren’t there – not to mention ourselves. So my colleagues at Fractured Atlas and I are happy to be helping Nina bring a new edition of Museum Camp to life focused on social impact assessment, a three-day event in which small teams of people will develop creative ways to evaluate the work that diverse organizations are doing to transform communities. Our focus is on social impact in communities, and we will encourage teams to look at complex outcomes–like safety, cohesion, compassion, and identity–that are not commonly covered in standard evaluative practices. This is a learning experience with a heavy focus on actual doing throughout the event. In addition to representatives from Fractured Atlas and MAH, we’ll have “camp counselors” from the United Way, WolfBrown, Harder & Co., Animating Democracy, and more on hand to help attendees navigate the conceptual and practical issues associated with measuring what matters.

If you are interested in attending, you can fill out an application through February 28. Space is extremely limited, so the sooner the better. We look forward to seeing you!

“I want change” by m.a.r.c.

Toward the end of the summer, bioethicist Peter Singer raised the hackles of art lovers everywhere with a New York Times op-ed that considered a hypothetical dilemma: should you donate to a charity that combats blindness in the developing world or should you spend that money instead on an art museum? After running through a cost-benefit analysis of each option, he determined that the charity addressing blindness “offers [donors] at least 10 times the value” of the museum.

Ouch.

To no one’s surprise, the arts community didn’t exactly roll out the welcome mat for the piece, calling Singer’s argument “a shocker,” “absurd,” and “tyrannical.” Another round of alarm ensued recently when none other than megaphilanthropist Bill Gates threw his support behind Singer’s thesis. The responses from our field to date have generally coalesced around two broad counter-arguments:

•  Why does it have to be “either/or”? Why can’t we support both? Singer forces a false choice in “assuming charitable giving is a zero sum game.” Weighing the value of saving a life against the value of donating to an art museum is comparing apples to oranges when “both are essential, and if either disappeared you’d be in bad shape.” We need a holistic approach to ensure we don’t “solv[e] Third World crises at the expense of fostering crises right here at home.” Just as we have “multiple passions in [our] lives,” donors can and should target multiple causes and direct their charitable dollars in a “proportionally prioritized” manner. Anyway, we can’t really be sure than curing blindness is more important than inspiring the next Jackson Pollock, and even if we were, concentrating all our resources with one or two tried and true nonprofits runs counter to the “messiness and power of America’s [decentralized] approach to charity.”

• Saving lives is all fine and good – but only if those lives have meaning. If we’re so concerned with making sure that people can see, shouldn’t we also try to make sure they have beautiful things to look at? Singer’s logic is dangerous because he fails to acknowledge the “creative outlet[s] and emotional oas[e]s that only art museum[s] can provide.” If all philanthropic dollars were channeled toward alleviating disease and poverty, arts and culture would languish, society would become monochromatic and dull, and life would cease to be worth living.

As satisfying as these rebuttals may feel to arts advocates, they unfortunately miss the point. The crucial assumptions behind Singer’s argument are that

1. “there are objective reasons for thinking we may be able to do more good in one [sector] than in another,” and
2. we have a moral obligation to make choices that do as much good as possible.

It’s important to understand this perspective in the context of “effective altruism,” a relatively nascent but growing area of applied ethics that has been featured more than once on this blog, not to mention a recent edition of This American Life. Besides Gates, fellow philanthropic heavyweight and past Hewlett Foundation President Paul Brest has declared himself a fan. “Effective altruists,” or EAs, are on a quest to “do good” by way of hard-nosed rationality. “Doing good” doesn’t mean recycling a little more, or occasionally doling out spare change to a beggar on the street. It doesn’t mean foregoing a high-powered corporate career to work for a nonprofit. It means taking the time to analyze how to do the most amount of good possible with the resources available – or, to use a more nerdy turn of phrase, to “[use] science and rational decision-making to help as many sentient beings” as they can.

Most funders are already in search of a big “bang for your buck,” but in trying to identify the objectively best causes to support, effective altruists stray from the conventional wisdom of mainstream philanthropy. EAs cast a global net when determining where to focus, and often settle on supporting causes in faraway parts of the world, the results of which they may never see in person. They also believe that while human lives are created equal, philanthropic causes are not. Those causes that can save or improve the most lives must take first priority.

How does this play out in practice? Let’s say you donate to the free medical clinic in your area. You do this for good reasons: you care about inequities in the American healthcare system, and want to give back to your community. You like the feeling you get when you walk by that clinic every day. Maybe you even know people who benefit from the services the clinic provides. The clinic gets its donation, and you get warm fuzzies. Everybody wins. Right?

Not so, an EA would counter. Despite your good intentions, your donation amounts to a near-waste of resources:

We understand the sentiment that ‘charity starts at home,’ and we used to agree with it, until we learned just how different U.S. charity is from charity aimed at the poorest people in the world. Helping people in the U.S. usually involves tackling extremely complex, poorly understood problems… In the poorest parts of the world, people suffer from very different problems…

We estimate that it costs [Givewell’s] top-rated international charity less than $2,500 to save a human life… Compare that with even the best U.S. programs… over $10,000 per child served, and their impact is encouraging but not overwhelming.

EAs advocate making evidence-based decisions even if they don’t resonate on an emotional or intuitive level:

Effective altruism is consistent with believing that giving benefits the giver, but it’s not consistent with making this the driving goal of giving. Effective altruists often take pride in their willingness to give (either time or money) based on arguments that others might find too intellectual or abstract, and their refusal to give suboptimally even when a pitch is emotionally compelling. The primary/driving goal is to help others, not to feel good about oneself.

If this approach leaves you with an empty feeling in the back of your throat, it is by design. “Opportunity costs” – the costs of choosing not to behave in a certain way – weigh heavily on EAs. Every time you make a donation, considering where your money could have gone is as important as considering where it will ultimately go (emphasis mine):

In the “Buy A Brushstroke” campaign, eleven thousand British donors gave a total of £550,000 to keep the famous painting “Blue Rigi” in a UK museum. If they had given that £550,000 to buy better sanitation systems in African villages instead, the latest statistics suggest it would have saved the lives of about one thousand two hundred people from disease…  Most of those 11,000 donors genuinely wanted to help people … But these people didn’t have the proper mental habits to realize that was the choice before them, and so a beautiful painting remains in a British museum and somewhere in the Third World a thousand people are dead.

Weighing choices isn’t limited to how we spend our money – it also applies to how we spend our time. Just as EAs dispute the notion that people should support whichever charities they feel “passionate” about, they question whether channeling those passions into a nonprofit or medical career is the best way to make a difference. Many suggest instead that people “earn to give,” saying they “might be better off…in a high-earning job and making a deliberate commitment to give a large portion of what [they] earn away.“ The organization 80,000 Hours, founded to “become the world’s number one source for advice on pursuing a career that truly makes a difference in an effective way,” elaborates,

Working at a non-profit can be a great way to make a difference. But it’s no guarantee. Amazingly, lots of non-profits probably have no impact. And do workers at [a] non-profit have more impact than the people who fund them? The researchers who push forward progress? The entrepreneurs who transform the economy? Policy makers? Maybe. No one stops to ask.

Putting ideas like these on the table is a great way to make those of us in the arts squirm. While there are echoes of the effective altruism movement in some recent trends within our field, like the “universal call” for better data on the impact of the arts and the pointed questions about who ultimately benefits from arts funding, the arts are chock-full of people – artists and arts administrators alike – who were drawn to their work by that same passion that EAs claim clouds our judgment. The idea of allowing cold rationality to dictate and limit our quest to “do good” flies in the face of our artistic sensibilities, and challenges the assumptions many of us made when we entered the nonprofit sector in the first place – even those of us who have a sincere desire to address social inequities.

Tempting as it may be, it would be short-sighted to dismiss the EA movement as the pet project of a bunch of aesthetically stunted curmudgeons. It’s hard to dispute the notion that we could improve the human condition if only we could get our act together and commit our resources to a data-driven approach. After all, the nonprofit darling of the moment, collective impact, is based on the same premise. What effective altruism does is counter our cause-specific argument for the arts with a dizzying moral appeal for cause agnosticism. And to be honest, it’s hard to see how the arts win if they play the game by the EAs’ rules. The “both/and” argument mentioned previously is unlikely to sway an effective altruist who weighs each decision as a choice between two different futures, one in which a museum gets funded and some lives get saved and one in which the museum struggles and more lives get saved. Even if the museum shut down completely, its patrons could probably find or create an alternative “creative outlet and emotional oasis,” while the people dying of malaria can’t very well make the mosquito nets themselves. The “we give lives meaning” argument likewise rings hollow when we’re talking about lending privileged lives (anyone living on more than $2 a day is privileged in a global context) a dose of incremental “meaning” at the expense of giving others a shot at basic survival. It also comes across as incredibly condescending to those others considering that they would likely never get the opportunity to visit or benefit from Singer’s hypothetical museum. In any case, art is hardly the only possible delivery mechanism for meaning. In the words of one effective altruist,

Trying to maximize the good I accomplish with both my hours and my dollars is an intellectually engaging challenge. It makes my life feel more meaningful and more important. It’s a way of trying to have an impact and significance beyond my daily experience. In other words, it meets the sort of non-material needs that many people have.

Whether the EA movement sputters or gathers steam, taking the time to engage with its principles, even critically, is a healthy exercise. The bottom line is that EAs may actually be onto something when they argue it’s possible to make a bigger dent in one sector than another. Rather than insisting otherwise or dodging the argument altogether, we could heed the call to examine how altruism really manifests in our work, particularly when examined through the lens of what benefits the people we engage, rather than what benefits our organizations or our donors. Might we, too, have objective reasons for thinking we may be able to do more “good” in one program, or with one population, than in another? Do we, too, have a moral obligation to maximize that good? How would that change how we operate and who we serve? Do we want to change how we operate?

If the effective altruism debate makes anything clear, it’s that to be able to make art, not to mention argue about it, is to be fortunate. Taking a hard look at our assumptions about what draws and keeps us to this work may not be easy, but if we squirm a little, so be it. In the grand scheme of things, a little squirming is a luxury too.

Kenyan shilling. Photo by Claudia Daggett.

A few years ago four grad students from Harvard and M.I.T. decided they wanted to use their brains and dollars to improve the lives of some of the poorest people in the world. They researched different strategies of philanthropy, looked at the data available, and based on the evidence they chose a novel approach. No microlending, school-building, or vaccination campaigns for them: they would just give away cash, no strings attached. They called their new charity, simply enough, GiveDirectly.

The Concept

This is how it works: money is transferred from the organization to pre-identified families in Kenya via cell phones. GiveDirectly’s selection of recipients is based solely on need as signaled by mud or thatch roofs, as opposed to more durable materials. The standard amount is $1,000 over one to two years, about as much as a poor Kenyan family might spend in a single year. There are no restrictions on what a family can buy with money from GiveDirectly. Unlike with more traditional philanthropic efforts, there are no mandatory health check-ups or vaccinations, no obligatory training programs, and no mandates of any kind. The cash is completely unfettered. It is a “UCT”: unconditional cash transfer.

One common use of GiveDirectly cash transfers is the purchase of a metal roof. Mud and thatch don’t hold up well to weather elements and must be repaired or replaced often. There are major savings to be had with the purchase of a metal roof, which typically lasts at least a decade.

That’s not the only way in which recipients spend the money, however. If you own a motorbike in Kenya that can withstand the terrain, won’t break down on long journeys, and can carry a passenger, you can be a taxi driver. With an influx of cash, you can literally buy yourself a livelihood.

These sorts of purchases improve quality of life and increase earning capacity, and some even have a ripple effect beyond a single family. A cow, for example, can provide milk for an entire village, an income stream for the owner, and can create more cows to continue to multiply the benefits.

The Logistics

Since a normal transfer from bank account to bank account isn’t an option, and transportation and distribution of thousands of dollars of physical cash would require security and increase liability for the charity, the transactions are carried out via cell phone. In Kenya the “mobile-money system” is called M-Pesa, and it’s one of the most successful of its kind; transfers are discreet, simple, and perfect for this atypical exchange.

One M-Pesa location in Kenya. Photo by Fiona Bradley.

Using this technology, GiveDirectly can keep track of hundreds of recipient families with a single spreadsheet and send them money each month safely and securely. The recipient gets a text message indicating that the money has arrived and goes to his or her local M-Pesa franchise to pick it up (the one described in the radio show “This American Life”’s GiveDirectly coverage is basically a VW bus turned bank with a ledger, a box of cash, and one staff person). It might sound dodgy to the Western world, but the M-Pesa system is dependable and ubiquitous in Kenya and saves people time and money previously spent traveling to traditional banks or delivering money to family in remote and inaccessible areas.

The Evidence 

As Jacob Goldstein for the New York Times writes, “At its most basic level… GiveDirectly’s work is an attempt to test one of the simplest ideas in economics — that people know what they need, and if they have money, they can buy it.” As radical as the approach may seem, it is grounded in a strong evidence base. According to Holden Karnofsky, co-founder of a charity that supports GiveDirectly, “cash transfers…happen to be the most extensively studied non-health intervention we know of.” Indeed, traditional charities often call the very act of transferring funds into the hands of low-income people success. In GiveDirectly’s model, by contrast, the impact measures are more nuanced. Money changing hands is the intervention, not the desired outcome.

GiveDirectly is committed to investigating and recording its own impact. The organization recently completed a randomized control trial in Kenya, which reveals that; “recipients are not just spending their transfers, providing a one-time boost to their consumption without affecting their overall well-being.” The trial shows that food-consumption increased 20 percent for transfer recipients and the value of recipients’ livestock increased by 50 percent. The study even showed that recipients’ stress levels improved – their actual stress hormone levels decreased. The full report and a summary are available here.

The Implications

Pondering the broader lessons of UTCs and GiveDirectly, I’m reminded of all of the nonprofit organizations out there for which an influx of funding could really change their organizational “standard of living.” What GiveDirectly is providing goes by another name in the nonprofit sector: general operating support.

General operating support is the holy grail of nonprofit fundraising, defined by the Foundation Center as “grants for the day-to-day operating costs of an existing program or organization; also called unrestricted grants.” There may be reporting requirements Kenyans don’t have to adhere to, and an application process with more vetting than GiveDirectly’s system of identifying families in need, but the concepts aren’t too far off. The similarities between general operating support to organizations and cash transfers to families, however, might not be as obvious for some in the nonprofit sector.

“We had conversations with people [in the non-profit sector] who said there was a lot of internal resistance to unconditional transfers,” Niehaus [one of GiveDirectly’s four founders] told [reporter Dana Goldstein]. “If this works, what are we all here for? Why do we have jobs? There’s an industry that exists that tries to make decisions for poor people and determine what’s best for them.”

Shouldn’t we want the same kind of aid for the poor that we in the nonprofit sector would want for ourselves?

One intersection of Data and art. Photo by Geoff Stearns.

For all of the predictions flying back and forth about what 2013 holds for the arts and culture sector in the United States, one of the few things we can say with near-certainty is that 2013 will be a year of major transition for the Cultural Data Project (CDP). Our sector’s largest-scale effort to quantify and streamline the “value” of arts and culture across different regions, the CDP is breaking off from its original home at the Pew Charitable Trusts to become a separate nonprofit. Until now, the CDP has been governed by a local Pennsylvania-based board, but it recently launched a national board and announced Beth Tuttle as their new CEO, who will lead a strategic planning process to further evolve its scope and impact. These shifts, according to the CDP, will “put [it] in an even stronger position to serve the arts and culture community” and its “increasingly large and diverse number of constituents.”

As arts organizations are increasingly pressed to demonstrate tangible benefits, this turning point in the CDP’s history provides an opportunity to look back on its trajectory and examine ways to measure the contributions of arts and culture to society.

For those unfamiliar with the CDP and how it came to be, here is a little refresher:

Background of the CDP

The CDP launched in Pennsylvania in 2004 thanks to the collaboration between the Pew Charitable Trusts and a number of local funders. Using the CDP platform, arts and cultural organizations create an annual Data Profile based on their financial audit and quantitative programmatic data, such as the number of exhibitions or workshops held. It is an online tool that aims to help arts and cultural organizations “improve their financial management and services to communities” by streamlining, storing and aggregating data on their financial and programmatic activities each year.

In grantseeking, it functions a little bit like the Common Application for undergraduate college admissions, which some funders have emulated through standardized application templates; once organizations create Data Profiles they can then submit customized reports to multiple funders. Arts organizations also have access to a variety of tools that illustrate trends in their performance from year to year, and can compare their data against other organizations in the region or across all states participating in the CDP.

The tools include:

• A Program Revenue and Marketing Expense Report, which explores the relationship between marketing expenses and attendance figures
• A Personnel Report, which outlines costs associated with staffing, salaries and benefits
• A Contributed Revenue and Fundraising Expense Report, which examines how changes in fundraising expenses effect contributed revenue
• The Financial Health Analysis, developed in partnership with the Nonprofit Finance Fund, which serves as a fiscal health “check up” and provides an overview of financial strengths, weaknesses and business dynamics

Users can tailor comparison reports according to detailed specifications. For example, if you’re a mid-sized theater company, you can see how your programming and marketing expenses compare to those of your peers.  You can also see how they compare against, say, all arts organizations that were founded within a certain time frame and target a specific constituent group. CDP Help Desk Staff assists in running those reports, and with cleaning and checking the data that goes into creating a Data Profile, which consists of nearly 300 questions and requires 15 to 30 hours to complete.

Why go through such a complex process? For one thing, funders get a standardized profile with grant applications that allows them to easily compare organizations. Accordingly, most funders who support the CDP require that nonprofits submit their Data Profiles and a customized report with their grant applications. Furthermore, many funders are interested in research, and the CDP provides a massive data bank. The more funders require their constituents to use the CDP, the bigger the ultimate payoff for researchers: a regional (and, perhaps one day, national) aggregate of information about arts and cultural organizations, updated annually.

As for the cultural organizations that have to complete the Data Profile, those applying to multiple grantmakers requiring CDP reports receive the benefit of being able to submit detailed financial information in one format. If they participate over several years and take advantage of the reports to track and compare their progress, they also gain a better picture of their own strengths. The drawback comes for organizations with limited capacity to devote to completing the Data Profile, or for those seeking to evaluate more qualitative elements of their work. But therein lie the possibilities.

The CDP’s Impact to Date: The Research Perspective

Thirteen states currently take part in the CDP, with sixteen more expressing interest. (A list of participating states with links to their CDP Web sites is available here.) In the near-decade since the CDP launched, how has it benefited the myriad cultural organizations that input data and the researchers who analyze it?

Most researchers to date have used CDP data to try to quantify the broad “impact” of the arts and culture sector in financial and programmatic terms, totaling up the number of people employed by arts and culture organizations in a given area, the number of public events, the number of attendees, etc.  A snippet from Cuyahoga County’s Strengthening Communities 2011 report provides one such example:

The ability to quantify the scope and variety of the nonprofit arts and culture communities has proven invaluable to several advocacy campaigns. The Greater Philadelphia Cultural Alliance used CDP data to lead constituents across the state to successfully defeat a proposed tax on arts and culture activities. ArtServe Michigan’s Creative State Michigan Report, comprised mainly of CDP data, prompted the state’s Republican governor to propose a threefold increase in state arts funding.

Regardless of whether studies like these demonstrate that the arts generate economic growth, research resulting from the CDP has undoubtedly helped make the case for investment in the arts and culture and tell a more concrete story about people served. These efforts have been somewhat limited, however, by the “one size fits all” nature of the CDP’s construction. After all, attributes like financial health and the quantity of public events would be more accurately described as outputs rather than outcome or impact measures. The fact that CDP profiles are optional to fill out for organizations whose funders don’t require it also creates challenges for researchers who wish to generalize from CDP data to all nonprofit arts and culture organizations, since the CDP constitutes a nonrandom sample of such organizations. Recently, funders in California and New York have begun piloting small grant programs to support research projects using the CDP in their states; time will tell if these opportunities lead to a more diverse range of inquiries using the data.

The CDP’s Impact to Date: Arts Organizations’ Perspective

The CDP has been a boon to research and advocacy efforts, but cultural organizations themselves don’t always hear about that work, or take full advantage of the CDP’s resources. In 2012, the CDP conducted a survey of over 1,800 arts organizations charged with filling out a Data Profile every year. Arin Sullivan, the CDP’s Senior Associate for State-Based Activities, said that the survey found that 68 percent of respondents had never read a report that includes CDP data, like Cuyahoga County’s. This implies that researchers, and the CDP itself, need to close the feedback loop between research and the constituents being studied. In addition, according to Sullivan, the survey revealed that more than 40% of participating organizations have never run an annual, trend, or comparison report.

There are a number of possible reasons for this. Organizations simply may not be aware of the CDP reporting features. They may have limited staff capacity to run and analyze reports, particularly after devoting significant time to completing their Data Profile. Or they are aware of the reporting features, but don’t think they are worth using.

No matter what the reasons, these results suggest a range of perceptions about the CDP. To some organizations, it is simply a profile to complete in order to get a grant. To others, it is a resource that can help better inform their practice. The same survey that found nearly half of organizations don’t use CDP reporting tools also found that 45% of participants understood their own finances better as a result of completing the Profile. Of those respondents that did use CDP reports, 40% said it resulted in better transparency, 45% said they had a better sense of their progress and goals, and 56% said they had a better sense of their organization over time.  These relatively low percentages suggest that even organizations taking full advantage of CDP reports do not always find them of substantial benefit.

Yet for some organizations, the CDP has been quite useful indeed. DanceWorks Chicago, for example, changed its entire accounting system to align with the format of the CDP. According to CEO Andreas Böttcher, many small organizations have little expertise in accounting, and until the advent of the CDP had no clear template to follow when setting up their books. Using the CDP was “onerous at first,” he said, “but once you have it, you can build a business plan based on what you have to report.”

DanceWorks has also made use of comparison reports. Böttcher said they’ve shown funders, for example, that not only is DanceWorks unique in its programming (which provides a laboratory for artistic growth to dancers, choreographers, and directors), but it is one of only a few organizations of its size to offer health insurance to dancers. Comparison reports also affirmed other things the company was doing well, despite its relative youth: its 80 individual donors reflect a larger funding pool than those of comparable, older organizations, and its web-based donations are 300% higher.

Other organizations have used the CDP to better track programmatic information, such as their most popular services or types of events. One such organization is the Rhode Island Historical Society, whose Executive Director, Morgan Grefe, used the CDP as an opportunity to make a greater commitment to program evaluation.  According to Grefe, having to complete the Data Profile made program evaluation “the thing you can’t ignore instead of the thing you always put off.” She credits the CDP process with the Society’s increased attentiveness to more qualitative program data, which has had an effect on the organization’s work.

For example, staff members started tracking how many visitors to different historic sites used those sites as a “visitor’s center,” asking for brochures and other tourist information, but not participating in a tour of the building.  Tracking such data enabled the Society to illustrate to public and private funders that the sites hold value to the tourist industry as well as to visitors concerned primarily with historic preservation. It also allows Grefe to make programmatic decisions, such as recruiting volunteers to fulfill visitor center functions or investing in more brochures.

An increasingly data-oriented attitude also led to the Society to launch the Rhode Island History Online Directory Initiative (RHODI). The RHODI Project will be “a comprehensive and detailed survey of Rhode Island’s history and heritage sector, [providing] not only trustworthy data on which to base future grant-funded proposals for such activities as collections cataloging, capacity building advice, preservation projects, educational programming, a virtual museum, but also the much needed impetus for synergies and collaborations.”

Beyond the Numbers: Looking Ahead as The CDP Expands

As these examples indicate, the CDP has shown potential in establishing and tracking organizational success measures that can encourage stronger business operations, advocate for the arts, and guide grantmaking. In order for that success to solidify, however, the CDP must gain buy-in from a broader group of stakeholders, and address the fact that its current ability to gauge the arts’ “impact” is limited.

Of course, the CDP can’t be expected to track and measure everything related to arts and culture. Perhaps in recognition of this fact, and in an effort to integrate CDP data with other available resources, Southern Methodist University recently announced a partnership with the CDP to launch the National Center for Arts Research (NCAR). Among other things, NCAR plans to launch an interactive “dashboard” where “arts leaders will be able to enter information about their organizations and see how they compare to the highest performance standards in areas such as community engagement, earned and contributed revenue and balance sheet health.” Assuming the effort manages to avoid redundancies with existing CDP infrastructure, it may fulfill its vision to become “a catalyst for the transformation and sustainability of the national arts and cultural community.”

At this point in the CDP’s history, it’s worth asking how it can further engage arts organizations, and what role it can play in evaluating more qualitative trends in cultural activities, such as audience loyalty and the evolution of programming. Since the organizations participating in the CDP range from zoos to art galleries, attempting to evaluate these aspects in any standardized way may be an exercise in insanity. What’s the dance education equivalent of a blockbuster museum exhibition? Nevertheless, there may be opportunities for the CDP to help organizations draw connections between CDP data and other reports, and to provide them with a broader toolkit of resources.

For example, for more than 20 years the League of American Orchestras has compiled a report of repertoire played by its member orchestras, including a list of the top 10 most frequently performed works, composers, and soloists. Though sadly only published in dense PDFs, one could track the repertoire of the top orchestras and determine if programming is, for example, becoming more conservative over time. Those same orchestras could make careful use of their own CDP data to make connections between programming choices and financial health.

The community and economic development sector, meanwhile, has attempted to build evaluative tools into a shared measurement system with a platform called Success Measures,  which is smaller in scope but has certain similarities to the CDP.  Success Measures provides its members, most of whom are organizations with little experience or capacity for program evaluation, with a set of evaluative tools (surveys, interview protocols, observational checklists, and so forth).  They receive training on how to use the tools and then, using an online system similar to the CDP’s, upload, clean and analyze their data.

Participation in Success Measures is optional. By contrast, in each state where CDP operates, many funders require it as part of the application process, effectively enforcing participation. Given the CDP’s reach and the strong level of technical support it already provides, it is well poised to develop and disseminate additional tools. Suppose an organization was hoping to understand its audience retention rates, and had the option of accessing standardized survey tools as a CDP participant. The CDP would provide the organization with technical assistance in using the tools, in exchange for the survey results being added to the aggregate database. Such a system could not only strengthen the value of participating in the CDP, but also take a step toward a deeper understanding of “impact.” New tools — even optional ones — can motivate organizations to take a closer look at their activities, perhaps discovering how cultural choices affect their bottom line over time, or if their audience development efforts are truly paying off.

No matter how (and if) it chooses to expand, a few things about the CDP are clear:

• It has propelled an ongoing conversation about the economic role of arts and culture
• It may yet build the capacity of the field as a whole if it more directly engages the arts and culture organizations it aims to serve
• To do so, it needs to clarify and draw greater attention to the existing benefits of using the system, and stay alert to organizations’ needs moving forward

With both for-profit and nonprofit sectors gravitating toward Big Data (and seeking more and more people to help them make sense of that data), the CDP can take a lead in ensuring that aggregate information is not only useful to researchers studying the arts and culture sector, but also to the organizations — large and small, vibrant and struggling — that are its engine.

(Talia Gibas is Manager, Arts for All at the Los Angeles County Arts Commission and a past Createquity Writing Fellow. Amanda Keil is a mezzo-soprano and writer based in New York City.)

Since I didn’t write out the speech in advance, I don’t have a transcript for it. However, below is a reconstruction of the new material from my notes, so you can get a taste for it if you don’t have time to watch the whole thing right now. (You’ll notice I make a number of generalizations in the speech about the ways in which arts practitioners interact with research. These are based on observation and personal experience, and are best understood as my working hypotheses.)

*

[starting at 26:55]

Why is this integration between data and strategy important? Because research is only valuable insofar as it influences decisions. This is why logic models are awesome – they are a visual depiction of strategy. And there is no such thing as strategy without cause and effect. Think about that for a second. Our lives can be understood as a set of circumstances and decisions. We make decisions to try to improve our circumstances, and sometimes the circumstances of those around us. Every decision you make is based on a prediction, whether explicitly articulated or not, about the results of that decision. Every decision, therefore, carries with it some degree of uncertainty. This uncertainty can be expressed another way: as an assumption about the way the world works and the context in which your decision is being made. These assumptions are distinguished from known facts.

If you can reduce the uncertainty associated with your assumptions, the chances that you will make the right decision will increase. So, how do you reduce that uncertainty? Through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile. Alas, research and practice in our field are frequently disconnected in problematic ways. Six issues are preventing us from reaching our potential.

Issue #1: Capacity

Supply and demand apply as much to research as it does to artists. There are far more studies out there than a normal arts professional can possibly fully process. I wish I could tell you how many research reports are published in the arts each year, but nobody knows! To establish a lower bound, I went back over last year’s [2011] “around the horn” posts, which report new research studies that I hear about. I counted at least 41 relevant arts-research-related publications – a tiny fraction, I’m sure, of total output. To make matters worse, research reports are long, and arts professionals are busy. For the Createquity Writing Fellowship program, participants are required to analyze a work of arts research for the Createquity Arts Policy Library. I collect data on how long it takes to do this, and consistently, it requires 30-80 hours to research, analyze and write just one piece! Multiply this by the number of new studies each year, and you can start to see the magnitude of the problem.

Issue #2: Dissemination

Which research reports is an arts practitioner likely to even know about? Certainly not all of them, because there is almost no meaningful connection between the academic research infrastructure and the professional arts ecosystem. Lots of research relevant to the arts is published in academic journals each year, but unless the faculty member was commissioned to do their work by a foundation, we never hear about it. Academic papers are typically behind a pay firewall, and most arts organizations don’t have journal subscriptions. To give an example, after I wrote about Richard Florida’s Rise of the Creative Class, Florida pointed me to a study in two parts by two Dutch researchers. It’s one of the best resources I’ve come across for creative class theory, but I’ve never heard anyone even mention either study other than him and me.

Issue #3: Interpretation

Research reports inevitably reflect the researcher’s voice and agenda. This is especially true of executive summaries and press releases, which is often all anyone “reads” of research “reports.” Probably the most common agenda, of course, is to convey that the researcher knows what he/she is talking about. Another common agenda is to ensure repeat business from, or at least a continuing relationship with, the client who commissioned the study. The reality, however, is that research varies widely in quality. There’s no certification process; anyone can call themselves a researcher. But even highly respected professionals can make mistakes, pursue questionable methods, or overlook obvious holes in their logic. And, in my experience, the reality of any given research effort is usually nuanced – some aspects of it are much more valuable than others. Unfortunately, many arts professionals lack expertise to properly evaluate research reports, not having had even basic statistics training.

Issue #4: Objectivity

Research is about uncovering the truth, but sometimes people don’t want to know the truth. Advocacy goals often precede research. How many times have you heard somebody say a version of the following: “We need research to back this up”? That statement suggests a kind of research study that we see all too often: one that is conducted to affirm decisions that have already been made. By contrast, when we create a logic model, we start with the end first: we identify what we are trying to achieve and only then determine the activities necessary to achieve it.

Here are a bunch of bad, but common reasons to do a research project:

• To prove your own value.
• To increase your organization’s prestige.
• To advance an ideological agenda.
• To provide political cover for a decision.

There is only one good reason to do research, and that is to try to find out something you didn’t know before.

Issue #5: Fragmentation

The worst part of the problem I just described is that it drives what research gets done – and what doesn’t get done. There is no common research agenda adopted by the entire field, which is a shame, because collective knowledge is pretty much the definition of a public good: if I increase my own knowledge, it’s very easy for me to increase your knowledge too. The practical consequences of this fragmentation are severe. It results in a concentration of research using readily available data sources (ignoring the fact that the creation of new data sources may be more valuable). It results in a concentration of research in geographies and communities that can afford it, because people don’t often pay for research that’s not about them. And it results in a concentration of research serving narrow interests: discipline-specific, organization-specific, methodology-specific. My biggest pet peeve is that research is almost never intentionally replicated – everybody’s reinventing the wheel, studying the same things over and over again in slightly different ways. A great example of a research study crying out for replication is the Arts Ripple Effect report, which I talked about earlier. The results of that study are now guiding the distribution of millions of dollars in annual arts funding. Are those results universal, or unique to the Greater Cincinnati region? We have no way to know.

Issue #6: Allocating resources

Everyone knows there’s been a trend in recent years towards more and more data collection at the level of the organization or artist. Organizations, especially small ones, complain all the time about being expected to do audience surveys, submit onerous paperwork, and so forth. And you know what, I agree with them! You might be surprised to hear me say that, but when you’re talking about organizations that have small budgets, no expertise to do this kind of work, and the funder who is requesting the information is not providing any assistance to get it…just take a risk! You make a small grant that goes bad, so what? You’re out a few thousand dollars. The sun will rise tomorrow.

As an example of what I’m talking about, I participated in a grant panel recently. I enjoyed the experience, and am glad I did it, but there’s one aspect of the experience that is relevant here. There were seven panelists, and we were all from out of town. Each of us spent, I’d say, roughly 40 hours reviewing applications in advance of the panel itself. Then we all got together for two full days in person to review these grants some more and talk about them and score them. We did this for 64 applications for up to $5,000 each, and in the end, 92% 94% were funded.

So consider this as a research exercise. The decision is who to give grants to, and how much. The data is the grant applications. The researchers are the review panel. What uncertainty is being reduced by this process? How much worse would the outcome have been if we’d just taken all the organizations, put them into Excel, run a random number generator, and distributed the dollars randomly up to $5,000 per organization? And I’m not saying this to make fun of this particular organization or single them out, because honestly it’s not uncommon to take this kind of approach to small-scale grantmaking. And yet if you compare it to ArtPlace’s first round of grants, theoretically they had thousands of projects to choose from, and they gave grants up to $1 million for creative placemaking projects – but there was no [open] review process; they just chose organizations to give grants to. So there’s a bit of a mismatch in the strategies we use to decide how to allocate resources.

There’s a concept called “expected value of information” described in a wonderful book called How to Measure Anything, by Douglas W. Hubbard. It’s a way of taking into account how much information matters to your decision-making process. In the book, Hubbard shares a couple of specific findings from his work as a consultant. He found that most variables have an information value of zero; in other words, we can study them all we want, but whatever the truth is is not going to change what we do, because they don’t matter enough in the grand scheme of things. And he also found that the things that matter the most, the kinds of things that really would change our decisions, often aren’t studied, because they’re perceived as too difficult to measure. So we need to ask ourselves how new information would actually change the decisions we make.

There is so much untapped potential in arts research. But it remains untapped because of all the issues described above. So what can we do about it?

First, we need a major field-building effort for arts research. Connecting researchers with each other through a virtual network/community of practice would help a lot. So would a centralized clearinghouse where all research can live, even if it’s behind a copyright firewall. The good news is that the National Endowment for the Arts has already been making some moves in this direction. The Endowment published a monograph a couple of months ago called “How Art Works,” the major focus of which was a so-called “system map” for the arts. But the document also had a pretty detailed research agenda for the NEA, not for the entire field, that lays out what the NEA’s Office of Research and Analysis is going to do over the next five years, and two of the items mentioned are exactly the two things I just talked about: a virtual research network and a centralized clearinghouse for arts research.

This new field that we’re building should be guided by a national research agenda that is collaboratively generated and directly tied to decisions of consequence. The missing piece from the research agenda in “How Art Works” is the tie to actual decisions. Instead it has categories, like cultural participation, and research projects can be sorted under those buckets. But it’s not enough for research to simply be about something – research should serve some purpose. What do we actually need to know in order to do our jobs better?

We should be asking researchers to spend less time generating new research and more time critically evaluating other people’s research. We need to generate lots more discussion about the research that is already produced. That’s the only way it’s going to enter the public consciousness. Each time we fail to do that, we are missing out on opportunities to increase knowledge. It will also raise our collective standards for research if we are engaging in a healthy debate about it. But realistically, in order for this to happen, field incentives are going to have to change – analyzing existing research will need to be seen as equally prestigious and worthy of funding as creating a new study. Of course, I would prefer if people are not evaluating the work of their direct competitors – but I’ll take what I can get at this point!

Every research effort should take into account the expected value of the information it will produce. Consider the risk involved in various types of grants made. What are you trying to achieve by giving out lots of small grants, if that’s what you’re doing? Maybe measure the effectiveness of the overall strategy instead of the success or failure of each grant. This is getting into hypothesis territory, but based on what I’ve seen so far I would guess that research on grant strategy is woefully underfunded, while research on the effectiveness or potential of specific grants is probably overfunded. We probably worry more than we need to about individual grants, but we don’t worry as much as we should about whether the ways in which we’re making decisions about which grants to support are the right ways to do that.

Finally, we should be open-sourcing research and working as a team. I’m talking about sharing not just finished products and final reports, but plans, data, methodologies as well. I’m talking about seeking multiple uses and potential partners at every point for the work we’re doing. This would make our work more effective by allowing us to leverage each other’s strengths – we’re not all experts at everything, after all! And it would cut down on duplicated effort and free up expensive people’s time to do work that moves the field forward.

I thank everyone for their time, and I’d love to take any questions or comments on these thoughts about the state of our research field.

How does scale influence impact in the arts? In 2007, back when I was a fresh-faced grad student, I actually addressed this question head on in the eighth post ever published on Createquity. I argued pretty strongly that scale in the arts was a myth, or at least not salient to the same extent as in other fields:

It’s not that I don’t think large arts organizations do good work, or that they don’t deserve to be supported. What I’m going to argue instead is that there is a tendency among many institutional givers to direct their resources toward organizations that have well-developed support infrastructure, long histories, and vast budgets, and in a lot of ways it’s a tendency that doesn’t make much sense (or at the very least, could use some balance).

For one thing, those well-developed support infrastructures don’t come cheap. Consider the case of Carnegie Hall… [snip]

In contrast, small arts organizations are extraordinarily frugal with their resources, precisely because they have no resources to speak of. It’s frankly amazing to me what largely unheralded art galleries, musical ensembles, theater companies, dance troupes, and performance art collectives are able accomplish with essentially nothing but passion on their side. A $5,000 contribution that would barely get you into the sixth-highest donor category at Carnegie might radically transform the livelihood of an organization like this. Suddenly, they might be able to buy some time in the recording studio, or hire an accompanist for rehearsals, or redo that floor in the lobby, or even (gasp) PAY their artists! All of which previously had seemed inconceivable because of the poverty that these organizations grapple with.

The literature on scaling impact in the social sector tends to take for granted that scale is a good thing—that services are provided more effectively when centralized under a strong leader and when efficiencies can be exploited across functions and sites. This logic makes sense when the goal is to solve a systemic problem that is evident in many different  contexts, such as physical places. If you’ve come up with a solution that works in Chicago, why wouldn’t you want to bring it to New York and DC? Arts service organizations, in fact, can likely benefit from economies of scale. Fractured Atlas has certainly been able to accomplish a lot more because its focus is national and cross-disciplinary than would have been the case otherwise, and scale has no doubt been a motivating factor behind Americans for the Arts’s many mergers.

But when you get to talking about arts producers and presenters, which I think is what most people mean when they say “the arts,” the conversation about scale becomes very different. What problem, exactly, is being solved here? It seems like the whole point of the nonprofit arts is to add to the aesthetic diversity that would otherwise exist in the marketplace for creative expression. If the point is diversity, how is that goal served by attempting to scale up institutions? The very commercial marketplace to which the nonprofit arts strive to provide an alternative loves scale – it thrives on it, because scale begets market power, which begets revenue, which begets profit. (Profits worth talking about, anyway.)

Leaving our cultural lives in the hands of commercial entities, many theorists have worried, will result in a boring sameness, an attempt to feed the world’s aesthetic appetite with the equivalent of TV dinners every day.* Our sector takes it on faith that there are forms of artistic expression that have clear cultural value and relevance even if replicating them widely is not practical. I suppose if you believe that these forms are specific and identifiable in nature (e.g., classical music, plays by Henrik Ibsen), then scaling them to help them compete with commercial cultural products would make sense. But if you believe, as I do, that their value comes in large part from the diversity they add to our collective palate, it’s much better to spread the subsidy around.

On a purely theoretical level, my view hasn’t changed that much in the five years since I wrote that thought piece. However, having become more closely involved with several grantmakers (including serving on a couple of grant panels) since then, I’ve developed a newfound appreciation for what large organizations can accomplish with scale. The scale that institutions traffic in does not have to do with the creation or presentation of work, but rather the audiences reached by that work. There are arts consumers – plenty of them, in fact – who simply will never frequent a show or exhibition by a smaller, experimental group or venue unless they personally know someone in it. But give that experimental group an institution’s stamp of approval, and those audience members are all over it. That’s got to count for something, and speaks volumes of the curatorial role that large institutions have in the broader ecosystem.

That said, one thing I still don’t see much of on the part of arts funders is a willingness to consider the transformative potential (or lack thereof) of grants. Some years ago, the Hewlett Foundation developed a simple yet very clever rubric for grant selection called Expected Return. One of the ways in which Expected Return is clever is that it accounts for the proportion of a project’s success in an ideal world that can be attributed to the grant you made. The less of the budget you’re responsible for, the less of a difference you’re really making. As I wrote then and still believe now, “Foundations concerned with ‘impact’ should remember that it’s far easier to have a measurable effect on an organization’s effectiveness when the amount of money provided is not dwarfed by the organization’s budget.”

*Defenders of pop culture will no doubt cite the many creative achievements of the entertainment industry as evidence against this point – and they are certainly right to celebrate The Wire, Radiohead, and The Lord of the Rings. But for every groundbreaking artist who succeeds in the market economy, there are dozens more who don’t, and plenty of mediocre talents gumming up our headphones and screens instead.

“There is nothing worse than a sharp image of a fuzzy concept.” -Ansel Adams
Photo by beast love

(If you don’t know the name Ann Markusen, you should. As professor and director of the Project on Regional and Industrial Economics at the University of Minnesota Humphrey School of Public Affairs, Ann has become one of the most respected and senior voices in the arts research community over the past decade. Among her best-known recent efforts was her authorship, with Anne Gadwa Nicodemus, of the original Creative Placemaking white paper published by the NEA prior to the creation of the Our Town grant program and ArtPlace funder collaborative. So when she approached me to offer a guest post on evaluation challenges for creative placemaking, building on previous coverage of the topic here at Createquity, I could hardly say no. I hope you enjoy Ann’s piece and I look forward to the vigorous discussion it will no doubt spark. -IDM)

*

Creative placemaking is electrifying communities large and small around the country. Mayors, public agencies and arts organizations are finding each other and committing to new initiatives. That’s a wonderful thing, whether or not their proposals are funded by national initiatives such as the National Endowment for the Arts’s Our Town program or ArtPlace.

It’s important to learn from and improve our practices on this new and so promising terrain. But efforts based on fuzzy concepts and indicators designed to rely on data external to the funded projects are bound to disappoint. Our evaluative systems must nurture rather than discourage the marvelous moving of arts organizations, artists and arts funders out of their bunkers and into our neighborhoods as leaders, animators, and above all, exhibitors of the value of arts and culture.

In our 2010 Creative Placemaking white paper for the NEA, Anne Gadwa Nicodemus and I characterize creative placemaking as a process where “partners… shape the physical and social character of a neighborhood, town, city, or region around arts and cultural activities.” A prominent ambition, we wrote, is to “bring diverse people together to celebrate, inspire, and be inspired.”  Creative placemaking also “animates public and private spaces, rejuvenates structures and streetscapes, (and) improves local business viability and public safety,” but arts and culture are at its core. This definition suggests a number of distinctive arenas of experimentation, where the gifts of the arts are devoted to community liveliness and collaborative problem-solving and where new people participate in the arts and share their cultures.

And, indeed, Our Town and ArtPlace encourage precisely this experimental ferment. Like the case studies in Creative Placemaking, each funded project is unique in its artistic disciplines, scale, problems addressed and aspirations for its particular place. Thus, a good evaluation system will monitor the progress of each project team towards its stated goals, including revisions made along the way. NEA’s Our Town asks grant-seekers to describe how they intend to evaluate their work, and ArtPlace requires a monthly blog entry. But rather than more formally evaluate each project’s progress over time, both funders have developed and are compiling place-specific measures based on external data sources that they will use to gauge success: the Arts and Livability Indicators  in the case of the NEA, and what ArtPlace is calling its Vibrancy Indicators.

Creative placemaking funders are optimistic about these efforts and their usefulness. “Over the next year or two,” wrote Jason Schupbach, NEA’s Director of Design, last May, “we will build out this system and publish it through a website so that anyone who wants to track a project’s progress in these areas (improved local community of artists and arts organizations, increased community attachment, improved quality of life, invigorated local economies) will be able to do so, whether it is NEA-funded or not. They can simply enter the time and geography parameters relevant to their project and see for themselves.”

Over the past two years, I have been consulting with creative placemaking leaders and given talks to audiences in many cities and towns across the country and abroad. Increasingly, I am hearing distress on the part of creative placemaking practitioners about the indicator initiatives of the National Endowment for the Arts and ArtPlace. At the annual meetings of the National Alliance for Media Arts and Culture last month, my fellow Creative Placemaking panel members, all involved in one or more ArtPlace- or Our-Town-funded projects, expressed considerable anxiety and confusion about these indicators and how they are being constructed. In particular, many current grantee teams with whom I’ve spoken are baffled by the one-measure-fits-all nature of the indicators, especially in the absence of formal and case-tailored evaluation.

I’ll confess I’m an evidence gal. I fervently believe in numbers where they are a good measure of outcomes; in secondary data like Census and the National Center for Charitable Statistics where they are up to the task; in surveys where no such data exist; in case studies to illuminate the context, process, and the impacts people tangibly experience; in interviews to find out how actors make decisions and view their own performance. My own work over the past decade is riddled with examples of these practices, including appendices intended to make the methodology and data used as transparent as possible.

So I embrace the project of evaluation, but am skeptical of relying on indicators for this purpose. In pursuing a more effective course, we can learn a lot from private sector venture capital practices, the ways that foundations conduct grantee evaluations, and, for political pitfalls, defense conversion placemaking experiments of the 1990s.

Learning from Venture Capital and Philanthropy

How do private sector venture capital (VC) firms evaluate the enterprises they invest in? Although they target rates of return in the longer run, they not do resort to indicators based on secondary data to evaluate progress. They closely monitor their investees—small firms who often have little business experience, just as many creative placemaking teams are new to their terrain. VC firms play an active role in guiding youthful companies, giving them feedback germane to their product or service goals. They help managers evaluate their progress and bring in special expertise where needed.

Venture capital firms are patient, understanding realistic timelines. The rule of thumb is that they commit to five to seven years, though it may be less or more. Among our Creative Placemaking cases, few efforts succeeded in five years, while some took ten to fifteen years.

VC firms know that some efforts will fail. They are attentive to learning from such failures and sharing what they learn in generic form with the larger business community. Both ArtPlace and the NEA have stated their desire to learn from success and failure. Yet generic indicators, their chosen evaluation tools, are neither patient or tailored to specific project ambitions. Current Our Town and ArtPlace grant recipients worry that the 1-2 years of funding they’re getting won’t be enough to carry projects through to success or establish enough local momentum to be self-sustaining. Neither ArtPlace nor Our Town have a realistic exit strategy in place for their investments, other than “the grant period’s over, good luck!”

Hands-on guidance is not foreign to nonprofit philanthropies funding the arts.  Many arts program officers act as informal consultants and mentors to young struggling arts organizations and to mature ones facing new challenges. My study with Amanda Johnson of Artists’ Centers shows how Minnesota funders have played such roles for decades. They ask established arts executive directors to mentor new start-ups, a process that the latter praised highly as crucial to their success. The Irvine and Hewlett Foundations are currently funding California nonprofit intermediaries to help small, folk and ethnic organizations use grant monies wisely. They also pay for intermediaries across sectors (arts and culture, health, community development and so on) to meet together to learn what works best.

The NEA has hosted three webinars at which Our Town panelists talk about what they see as effective projects/proposals, a step in this direction. But these discussions are far from a systematic gathering and collating of experience from all grantees in ways that would help the cohorts learn and contact those with similar challenges.

The Indicator Impetus

Why are the major funders of creative placemaking staking so much on indicators rather than evaluating projects on their own aspirations and steps forward? Pressure from the Office of Management and Budget, the federal bean-counters, is one factor.  In January of 2011, President Obama signed into law the Government Performance and Modernization Act (GPRA), updating the original 1993 GPRA, and a new August 2012 Circular A11 heavily emphasizes use of performance indicators for all agencies and their programs.

As a veteran of research and policy work on scientific and engineering occupations and on industrial sectors like steel and the military industrial complex, I fear that others will perceive indicator mania as a sign of field weakness. To Ian David Moss’s provocative title “Creative Placemaking has an Outcomes Problem,” I’d reply that we’re in good company. Huge agencies of the federal government, like the National Science Foundation, the National Institutes of Health and NASA, fund experiments and exploratory development without asking that results be held up to some set of external indicators not closely related to their missions. They accept slow progress and even failure, as in cancer research or nuclear fusion, because the end goal is worthy and because we learn from failure. Evaluation by external generic indicators fails to acknowledge the experimental and ground-breaking nature of these creative-placemaking initiatives and misses an opportunity to bolster understanding of how arts and cultural missions create public value.

Why Indicators Will Disappoint I: Definitional Challenges

Many of the indicators charted in ArtPlace, NEA Our Town, and other exercises (e.g. WESTAF’s Creative Vitality Index) bear a tenuous relationship to the complex fabric of communities or specific creative placemaking initiatives. Terms like “vitality,” “vibrancy,” and “livability” are great examples of fuzzy concepts, a notion that I used a decade ago to critique planners and geographers’ enamoration with concepts like “world cities” and “flexible specialization.” A fuzzy concept is one that means different things to different people, but flourishes precisely because of its imprecision. It leaves one open to trenchant critiques, as in Thomas Frank’s recent pillorying of the notion of vibrancy.

Take livability, for instance, prominent in the NEA’s indicators project. One person’s quality of life can be inimical to others’. Take the young live music scene in cities: youth magnet, older resident nightmare.  Probably no worthy concept, as quality of life is, has been the subject of so many disappointing and conflicting measurement exercises.

Just what does vibrancy mean? Let’s try to unpack the term. ArtPlace’s definition: “we define vibrancy as places with an unusual scale and intensity of specific kinds of human interaction.” Pretty vague and….vibrancy are places?  Unusual scale? Scale meaning extensive, intensive? Of specific kinds? What kinds? This definition is followed by: “While we are not able to measure vibrancy directly, we believe that the measures we are assembling, taken together, will provide useful insights into the nature and location of especially vibrant places within communities.”  If I were running a college or community discussion session on this, I would put the terms “vibrancy, places, communities, measures,” and so on up on the board (so to speak), and we would undoubtedly have a spirited and inconclusive debate!

And what is the purpose of measuring vibrancy? Again from the same ArtPlace LOI: “…the purpose of our vibrancy metrics is not to pronounce some projects ‘successes’ and other projects ‘failures’ but rather to learn more about the characteristics of the projects and community context in which they take place which leads to or at least seems associated with improved places.” Even though the above description mentions “characteristics of the projects,” it’s notable that their published vibrancy indicators only measure features of place.

In fact, many of the ArtPlace and NEA indicators are roughly designed and sometime in conflict. While giving the nod to “thriving in place,” ArtPlace emphasizes the desirability of visitors in its vibrancy definition (meaning outsiders to the community); by contrast, the NEA prioritizes social cohesion and community attachment, attributes scarce in the ArtPlace definitions. For instance, ArtPlace proposes to use employment ratio—“the number of employed residents living in a particular geography (Census Block) and dividing that number by the working age persons living on that same block” as a measure of people-vibrancy. The rationale: “vibrant neighborhoods have a high fraction of their residents of working age who are employed.” Think of the large areas of new non-mixed use upscale high-rise condos where the mostly young professional people living there commute daily to jobs and nightly to bars and cafes outside the neighborhood. Not vibrant at all. But such areas would rank high using this measure.

ArtPlace links vibrancy with diversity, defined as heterogeneity of people by income, race and ethnicity. They propose “the racial and ethnic diversity index” (composition not made explicit) and “the mixed-income, middle income index” (ditto) to capture diversity. But what about age diversity? Shouldn’t we want intergenerational activity and encounters too? It is also problematic to prioritize the dilution of ethnicity in large enclaves of recent immigrant groups. Would a thriving heavily Vietnamese city or suburb be considered non-vibrant because its residents choose to live and build their cultural institutions there, facing discrimination in other housing markets? Would an ethnic neighborhood experiencing white hipster incursions be evaluated positively despite decline in its minority populations that result from lower income people being forced out?

Many of the NEA’s indicators are similarly fuzzy. As an indicator of impact on art communities and artists, its August 2012 RFP proposes median earnings for residents employed in entertainment-related industries (arts, design, entertainment, sports, and media occupations). But a very large number of people in these occupations are in sports and media fields, not the arts. The measure does not include artists who live outside the area but work there. And many artists self-report their industry as other than the one listed above, e.g. musicians work in the restaurant sector, and graphic artists work in motion pictures, publishing and so on. ArtPlace is proposing to use very similar indicators—creative industry jobs and workers in creative occupations—as measures of vibrancy.

It is troubling that neither indicator-building effort has so far demonstrated a willingness to digest and share publicly the rich, accessible, and cautionary published research that tackles many of these definitions. See for instance “Defining the Creative Economy: Industry and Occupational Approaches,” the joint effort by researchers Doug DeNatale and Greg Wassall from the New England Creative Economy Project, Randy Cohen of Americans for the Arts, and me at the Arts Economy Initiative to unpack the definitional and data challenges for measuring arts-related jobs and industries in Economic Development Quarterly.

Hopefully, we can have an engaging debate about these notions before indices are cranked out and disseminated. Heartening signs: in its August RFP, the NEA backtracks from its original plan, unveiled in a spring 2012 webinar, to contract for wholesale construction of a given set of indicators to be distributed to grantees. Instead, it is now contracting for the testing of indicator suitability by conducting twenty case studies. And just last week, the NEA issued a new RFP for developing a virtual storybook to document community outcomes, lessons learned and experiences associated with their creative placemaking projects.

Why Indicators Will Disappoint II: Dearth of Good Data

If definitional problems aren’t troubling enough, think about the sheer inadequacy of data sources available for creating place-specific indicators.

For more than a half-century, planning and economic development scholars have been studying places and policy interventions to judge success or failure. Yet when Anne Gadwa Nicodemus went in search of research results on decades of public housing interventions, assuming she could build on these for her evaluation of Artspace Projects’ artist live/work and studio buildings, she found that they don’t really exist.

Here are five serious operational problems confronting creative placemaking indicator construction. First, the dimensions to be measured are hard to pin down. Some of the variables proposed are quite problematic—they don’t capture universal values for all people in the community.

Take ArtPlace’s cell phone activity indicator, for instance, which will be used on nights and weekends to map where people congregate. Are places with cell activity to be judged as more successful at creative placemaking? Cell phone usage is heavily correlated with age, income and ethnicity. The older you are, the less likely you are to have a cell phone or use it much, and the more likely to rely on land-lines, which many young people do without. At the November 2012 American Collegiate Schools of Planning annual meetings, Brettany Shannon of University of Southern California presented research results from a survey of 460 LA bus riders showing low cell phone usage rates among the elderly, particularly Latinos. Among those aged 18-30, only 9% of English speakers and 15% of Spanish speakers had no cell phone, compared with 29% of English speakers over age 50 and 54% of Spanish speakers.  A cell phone activity measure is also likely to completely miss people attending jazz or classical music concerts, dramas, and religious cultural events where cell phones are turned off. And what about all those older folks who prefer to sit in coffee shops and talk to each other during the day, play leadership roles in the community through face-to-face work, or meet and engage in arts and cultural activities around religious venues? Aren’t they congregating, too?

Or take home ownership and home values, an indicator the NEA hopes to use. Hmmm… home ownership rates—and values—in the US have been falling, in large part due to overselling of homes during the housing bubble. Renting is a just as respectable an option for place lovers, especially young people, retirees, and lower-income people in general. Why would we want grantees to aspire to raise homeownership rates in their neighborhoods, especially given gentrification concerns? Home ownership does not insulate you against displacement, because as property values rise, property taxes do as well, driving out renters and homeowners alike on fixed or lower incomes. ArtPlace is developing “measures of value, which capture changes in rental and ownership values…” This reads like an invitation to gentrification, and contrary to the NEA’s aspirations for creative placemaking to support social cohesion and community attachment.

Second, most good secondary data series are not available at spatial scales corresponding to grantees’ target places. ArtPlace’s vibrancy exercise aspires to compare neighborhoods with other neighborhoods, but available data makes this task almost impossible to accomplish at highly localized scales. Some data points, like arts employment by industry, are available only down to the county level and only for more heavily populated counties because of suppression problems (and because they are lumped together with sports and media in some data sets). Good data on artists from the Census (Public Use Microdata Sample) and American Community Surveys, the only database that includes the self-employed and unemployed, can’t be broken down below PUMA (Public Use Microdata Areas) of 100,000 people that bear little relationship to real neighborhoods or city districts (see Crossover, where we mapped artists using 2000 PUMS data for the Los Angeles and Bay Area metros).

Plus, many creative placemaking efforts have ambitions to have an impact at multiple scales. Gadwa Nicodemus’s pioneering research studies, How Artist Space Matters and How Art Spaces Matter II, looked in hindsight at Artspace’s artist live/work and mixed use projects where the criteria for success varied widely between projects and for various stakeholders involved in each.  Artists, nonprofit arts organizations, and commercial enterprises (e.g. cafes) in the buildings variously hoped that the project would an impact on the regional arts community, neighborhood commercial activity and crime rates, and local property values. The research methods included surveys and interviews exploring whether the goals of the projects have been achieved in the experience of target users. Others involve complex secondary data manipulation to come up with indicators that are a good fit. Gadwa Nicodemus’s studies demonstrate how much work it is to document real impact along several dimensions, multiple spatial scales, and a long enough time periods to ensure a decent test. Her indicators, such as hedonic price indices to gauge area property value change, are sophisticated, but also very time- and skill-intensive to construct.

Third, even if you find data that address what you hope to achieve, they are unlikely be statistically significant at the scales you hope for. In our work with PUMS data from the 2000 Census, a very reliable 5% sample, we found we could not make reliable estimates of artist populations at anything near a neighborhood scale. To map the location of artists in Minneapolis, we had to carve the city into three segments based on PUMA lines, and even then, we were pushing the statistical reliability hard (Artists’ Centers, Figure 3, p. 108).

Some researchers are beginning to use the American Community Survey, a 1% sample much smaller than the decennial Census PUMS 5%, to build local indicators, heedless of this statistical reliability challenge. ArtPlace, for instance, is proposing to use ACS data to capture workers in creative occupations at the Census Tract level. See the statistical appendix to Leveraging Investments in Creativity (LINC)’s Creative Communities Artist Data User Guide  for a detailed explanation of this problem. Adding the ACS up over five years, one way of improving reliability, is problematic if you are trying to show change over a short period of time, which the creative placemaking indicators presumably aspire to do.

Fourth, charting change over time successfully is a huge challenge. ArtPlace intends to “assess the level of vibrancy of different areas within communities, and importantly, to measure changes in vibrancy over time in the communities where ArtPlace invests.” How can we expect projects that hope to change the culture, participation, physical environment and local economy to show anything in a period of one, two, three years? More ephemeral interventions may only have hard-to-measure impacts in the year that they happen, even if they catalyze spinoff activities, while the potentially clearer impact of brick-and-mortar projects may take years to materialize.

We know from our case studies and from decades of urban planning and design experience that changes in place take long periods of time. For example, Cleveland’s Gordon Square Arts District, a case study in Creative Placemaking, required at least five years for vision and conversations to translate into a feasibility study, another few years to build the streetscape and renovate the two existing shuttered theatres, and more to build the new one.

Because it’s unlikely that the data will be good enough to chart creative placemaking projects’ progress over time, we are likely to see indicators used in a very different and pernicious way – to compare places with each other in the current time period. But every creative placemaking initiative is very, very different from others, and their current rankings on these measures more apt to reflect long-time neighborhood evolution and particularities rather than the impact of their current activities. I can just see creative placemakers viewing such comparisons and throwing their hands up in the air, shouting, “but.. but…but, our circumstances are not comparable!”

One final indicator challenge. As far as I can tell, there are very few arts and cultural indicators included among the measures under consideration. Where is the mission of bringing diverse people together to celebrate, inspire, and be inspired? Shouldn’t creative placemaking advance the intrinsic values and impact of the arts? Heightened and broadened arts participation? Preserving cultural traditions? Better quality art offerings? Providing beauty, expression, and critical perspectives on our society? Are artists and arts organizations whose greatest talents lie in the arts world to be judged only on their impact outside of this core? Though arts participation is measurable, many of the these “intrinsic” outcomes are challenging data-wise, just as are many of the “instrumental’ outcomes given central place in current indicator efforts. WolfBrown now offers a website that aims to “change the conversation about the benefits of arts participation, disseminate up-to-date information on emerging practices in impact assessment, and encourage cultural organizations to embrace impact assessment as standard operating practice.”

The Political Dangers of Relying on Indicators

I fear three kinds of negative political responses to reliance on poorly-defined and operationalized indicators.  First, it could be off-putting to grantees and would-be grantees, including mayors, arts organizations, community development organizations and the many other partners to these projects. It could be baffling, even angering, to be served up a book of cooked indicators with very little fit to one’s project and aspirations and to be asked to make sense out of them. The NEA’s recent RFP calls for the development of a user guide with some examples, which will help. Those who have expressed concern report hearing back something like “don’t worry about it – we’re not going to hold you to any particular performance on these. They are just informational for you.” Well, but then why invest in these indicators if they aren’t going to be used for evaluation after all?!

Second, creative placemaking grants create competitors, and that means they are generating losers as well as winners.  Some who aren’t funded the first time try again, and some are sanguine and grateful that they were prompted to make the effort and form a team. But some will give up. There are interesting parallels with place-based innovations in the 1990s. The Clinton administration’s post Cold War defense conversion initiatives included the Technology Reinvestment Project, in which regional consortia competed for funds to take local military technologies into the civilian realm. As Michael Oden, Greg Bischak and Chris Evans-Klock concluded in our 1995 Rutgers study (full report available from the authors on request), the TRP failed after just a few years because Members of Congress heard from too many disgruntled constituents. In contrast, the Manufacturing Extension Partnership, begun in the same period and administered by NIST, has survived because after its first exploratory rounds, it partnered with state governments to amplify funding for technical assistance to defense contractors struggling with defense budget implosion everywhere. States, rather than projects, then competed, eager for the federal funds.

Third, and most troubling, funders may begin favoring grants to places that already look good on the indicators. Anne Gadwa Nicodemus raised this in her GIA Reader article on creative placemaking last spring. ArtPlace’s own funding criteria suggest this: “ArtPlace will favor investments… and sees its role as providing venture funding in the form of grants, seeding entrepreneurial projects that lead through the arts and already enjoy strong local buy-in and will occur at places already showing signs of momentum….” Imagine how a proposal to convert an old school in a very low income and somewhat depopulated, minority neighborhood into an artist live/work, studio and performance and learning space would stack up against a proposal to add funding to a new outreach initiative in an area already colonized by young people from elsewhere in the same city. A funder might be tempted to fund the latter, where vibrancy is already indicated, over the other, where the payoff might be much greater but farther down the road.

In an Ideal World, Sophisticated Models

In any particular place, changes in the proposed indicators will not be attributable to the creative placemaking intervention alone. So imagine the distress of a fundee whose indicators are moving the wrong way and which place them poorly in comparison to others. Area property values may be falling because an environmentally obnoxious plant starts up. Other projects might look great on indicators not because of their initiatives, but because another intervention, like a new light rail system or a new community-based school dramatically changes the neighborhood.

What we’d would love to have, but don’t at this point, are sophisticated causal models of creative placemaking. The models would identify the multiple actors in the target place and take into account the results of their separate actions. A funded creative placemaking project team would be just one such “actor” among several (e.g. real estate developers, private sector employers, resident associations, community development nonprofits and so on).

A good model would account for other non-arts forces at work that will interact with the various actors’ initiatives and choices. This is crucial, and the logic models proposed by Moss, Zabel and others don’t do it. Scholars of urban planning well know how tricky it is to isolate the impact of a particular intervention when there are so many others occurring simultaneously (crime prevention, community development, social services, infrastructure investments like light rail or street repaving).

Furthermore, models should be longitudinal, i.e. they will chart progress in the particular place over time, rather than comparing one place cross-sectionally with others that are quite unlikely to share the same actors, features and circumstances. If we create models that are causal, acknowledge other forces at work, and are applied over time, “we’ll be able to clearly document the critical power of arts and culture in healthy community development,” reflects Deborah Cullinan of San Francisco’s Intersection for the Arts in a followup to our NAMAC panel.

Such multivariate models, as social scientists and urban planners call them, lend themselves to careful tests of hypotheses about change. We can ask if a particular action, like the siting of an interstate highway interchange or adding a prison or being funded in a federal program like the Appalachian Regional Commission, produces more employment or higher incomes or better quality of life for its host city or neighborhood when compared with twin or comparable places, as Andrew Isserman and colleagues have done in their “quasi-experimental” work (write me for a summary of these, soon to be published).

We can also run tests to see if differentials in city and regional arts participation rates and presence of arts organizations can be explained by differences in funding, demographics, or features of local economies. My teammates and I used Cultural Data Project and National Center for Charitable Statistics data on nonprofit arts organizations in California to do this for all California cities with more than 20,000 residents. Our results, while cross-sectional, suggest that concerted arts and culture-building by local Californians over time leads to higher arts participation rates and more arts offerings than can be explained by other factors. The point is that techniques like these DO take into account other forces (positive and negative) operating in the place where creative placemaking unfolds.

Charting a Better Path

It’s understandable why the NEA and ArtPlace are turning to indicators. Their budgets for creative placemaking are relatively small, and they’d prefer to spend them on more programming and more places rather than on expensive, careful evaluations.  Nevertheless, designing indicators unrelated to specific funded projects seems a poor way forward. Here are some alternatives.

Commit to real evaluation. This need not be as expensive as it seems. Imagine if the NEA and ArtPlace, instead of contracting to produce one-size-fits-all indicators, were to design a three-stage evaluation process.  Grantees propose staged criteria for success and reflect on them at specified junctures. Funding is awarded on the basis of the appropriateness of this evaluative process and continued on receipt of reflections. Funders use these to give feedback to the grantee and retool their expectations if necessary, and to summarize and redesign overall creative placemaking achievements. This is more or less what many philanthropic foundations do currently and have for many years, the NEA included. Better learning is apt to emerge from this process than from a set of indicator tables and graphics.  ArtPlace is well-positioned to draw on the expertise of its member foundations in this regard.

Build cooperation among grantees to soften the edge of competition for funds. Convene grantees and would-be grantees annually to talk about success, failures, and problems. Ask successful grantees to share their experience and expertise with others who wish to try similar projects elsewhere. During Leveraging Investments in Creativity’s ten-year lifespan, it convened its creative community leaders annually and sometimes more often, resulting in tremendous cross-fertilization that boosted success. Often, what was working elsewhere turned out to be a better mission or process than what a local group had planned. Again, ArtPlace in particular could create a forum for this kind of cooperative learning. And, as mentioned, NEA’s webinars are a step in the right direction. Imagine, notes my NAMAC co-panelist Deborah Cullinan of Intersection for the Arts, if creative placemaking funders invested in cohort learning over time, with enough longevity to build relationships, share lessons, and nurture collaborations.

Finally, the National Endowment for the Arts and ArtPlace could provide technical assistance to creative placemaking grantees, as the Manufacturing Extension Partnership does for small manufacturers. Anne Gadwa Nicodemus and I continually receive phone calls from people across the country psyched to start projects but needy of information and skills on multiple fronts. There are leaders in other communities, and consultants, too, who know how creative placemaking works under diverse circumstances and who can form a loose consortium of talent: people who understand the political framework, the financial challenges, and the way to build partnerships. Artspace Projects, for instance, has recently converted over a quarter century of experience with more than two -dozen completed artist and arts-serving projects into a consultancy to help people in more places craft arts-based placemaking projects.

Wouldn’t it be wonderful if, in a few years’ time, we could say, look!  Here is the body of learning and insights we’ve compiled about creative placemaking–how to do it well, where the diverse impacts are, and how they can be documented. With indicators dominating the evaluation process at present, we are unlikely to learn what we could from these young experiments. An indicators-preoccupied evaluation process is likely to leave us disappointed, with spreadsheets and charts made quickly obsolete by changing definitions and data collection procedures. Let’s think through outcomes in a more grounded, holistic way. Let’s continue, and broaden, the conversation!

(The author would like to thank Anne Gadwa Nicodemus, Deborah Cullinan, Ian David Moss, and Jackie Hasa for thorough reads and responses to earlier drafts of this article.)

Note to readers: In addition to the comments below, the National Endowment for the Arts and ArtPlace have now published official responses to this article. Read them here:

• Jason Schupbach and Sunil Iyengar, Our View of Creative Placemaking, Two Years In
• Carol Coletta and Joe Cortright, Understanding Creative Placemaking

“a double-blind study,” photograph by Casey Holford

On October 1 the science section of the New York Times ran two articles next to each other. One of them describes a recent study that concluded young children at play display behaviors similar to those of scientists, suggesting scientific inquiry is driven by human instinct. The other refers to the alarming extent to which that human instinct muddies scientific inquiry along the way.

Recently the scientific community has dealt with controversies cascading across many areas of research.  Most of them relate to a phenomenon known as publication bias.  Put simply, publication bias occurs when research journals prioritize studies with thought-provoking—and at the very least statistically significant—results. This makes sense; it’s hard to get excited about studies that don’t show anything conclusive. We crave good stories, stunning breakthroughs, and world-changing discoveries. Such desire has driven scientific (and artistic) innovation throughout history.

The dark underbelly of this lust for meaning, however, is something called “significance chasing.” Researchers know their chances of getting published – and advancing their professional status – hinge on getting statistically significant results.  They have a huge incentive to hunt for and read into anomalies in data – raising the possibility of over-interpreting those anomalies as due to something other than chance. An article in the journal Psychological Science illustrates this point eerily well.  As the authors point out,

It is common (and accepted practice) for researchers to explore various analytic alternatives, to search for a combination that yields ‘statistical significance,’ and then to report only what ‘worked’… This exploratory behavior is not the by-product of malicious intent, but rather the result of two factors: (a) ambiguity in how best to make these decisions and (b) the researcher’s desire to find a statistically significant result.

To compound the problem, many researchers do not openly share their full data sets or calculation methods, and have few incentives to challenge one another’s findings.  The Psychological Science article hammers the former point home with a simulated experiment that “shows” listening to a Beatles song makes you older.  That’s hooey, of course, but the authors’ point is that without stricter guidelines around how data sets are reported, nearly any relationship can be presented as statistically significant.

How big of a problem is this? In the medical community it has raised frightening questions about cancer studies that had been the basis for new treatments. It has caused an increase in the number of retractions issued in high-profile scientific journals – and a blog devoted to tracking them. And lest you think this concern is limited to the “hard” sciences, think again – it has already raised discussions of implications in humanitarian aid and in the more mainstream business community (the latter summing things up nicely with a headline, “Why You Can’t Trust Any of the Research You Read”).

Yikes.

The idea that the scientific method is easily mucked up opens up a whole host of mind-bending questions. (What if there’s a publication bias toward studies about publication bias?  Eeek…). It forces us to stop and think about the fledgling world of arts research – a world that has desperately wanted to find good, hard scientific evidence of impact for a long time. Randomized controlled trials, double-blind studies and other sophisticated research methods seemed like a holy grail, promising that if we could cleverly adapt them to meet our needs, we would have indisputable evidence of the importance of the arts, and good, hard data to guide how we direct our resources. In light of these controversies, should we question our desire to be better researchers?

No – but we should learn from others’ mistakes, and take a hard look at institutional issues common across our fields. Many of the problems the scientific community is experiencing aren’t about the tools scientists have at their disposal, but the cultures in which those tools are used. A few months ago the editors of two high-profile medical journals, Drs. Ferric Fang and Arturo Casadevall, put out a call for “structural reforms” to combat a “hypercompetitive” and “insecure” working environment they believe to be the heart of the issue. The structural flaws they identify include inadequate resources, a “leaky pipeline” of emerging talent, agenda-driven funding and administrative bloat.

Sound familiar?

The long-term implications on all research communities will unfold over time. Many of Fang and Casadevall’s recommendations are similar to those made within our own field: directing more funding toward salary support to increase job stability, streamlining grant application and reporting processes, and examining the strengths and weaknesses of peer grant review. A number of other ideas have been floated that may change established research practices. Creating a “journal of good questions” that decides which studies to publish before their results are known would reward researchers for their curiosity and the strength of their proposed methodology. Limiting the “degrees of freedom” researchers have in gathering additional data if their original data set does not yield anything “interesting” would limit significance chasing and, in theory, create a culture more tolerant of inconclusive results.

Regardless of which, if any, of these ideas stick, we need to acknowledge two things: a) our research is in all likelihood as prone, if not more prone, to these problems as the “hard sciences,” and b) the “best practices” we have been trying to emulate are not “fixed practices.” It’s often said that what arts researchers seek to measure is too squishy to fit into the traditional scientific process. If more and more people are realizing the process has a squish of its own – well then, maybe we don’t need to play “catch up” so much as try new things.

We may even come up with ideas useful to the more “established” fields we have been trying to emulate. The authors of the study in the first (less depressing) New York Times article concluded the preschoolers they observed behaved like scientists because they “form[ed] hypotheses, [ran] experiments, calculat[ed] probabilities and decipher[ed] causal relationships about the world.” I suspect that a group of arts researchers, observing the same group of children, would have interpreted those same behaviors as artistic. Human instinct drives scientific inquiry and artistic inquiry, and muddies both. Artists, one could argue, are a little more used to the mud.

• RIP Artnet Magazine; more here. I will always be grateful to Artnet’s Ben Davis for being just about the only arts journalist worth his salt during the whole Yosi Sergant debacle.
• Congratulations to GiveWell, which has announced a not-quite-merger with Good Ventures, an emerging foundation led by Cari Tuna and Dustin Moskovitz (the latter is one of the founders of Facebook). The blog post is a bit thin on details, but it sounds like this arrangement will ensure GiveWell’s financial security for some time to come while substantially enhancing its real-world impact.
• Indiana University is set to open the country’s first School of Philanthropy later this year. It’s early, of course, but these snippets from the article suggest to me that buyer beware: “As with any academic setting, funding is an issue….With the nonprofit sector roughly 5 percent of the nation’s gross domestic product and 10 percent of the workforce, such [a] school could be a profit-center for the university, Rooney said.”
• One of the NEA’s lesser known programs, the Citizens’ Institute on Rural Design, will now be a partnership between the NEA, the Department of Agriculture, Project for Public Spaces, the Orton Family Foundation, and CommunityMatters. CIRD facilitates and hosts workshops on community design in places with fewer than 50,000 people.

BIG IDEAS

• Michael Kaiser has a penchant for inciting digital controversy, and his recent two–part post calling bullshit on “new business models” was no exception. At the core of the debate is this central question: how much is the nonprofit arts sector going to change in the next 50 years? Kaiser says not so much; Adam Huttler, on the other hand, thinks quite a lot. Huttler’s second post on the subject, in particular, is one of his most thought-provoking and brilliant in quite some time. EmcArts’s Richard Evans and Sarah Lutman also weighed in.
• Whither the future of open data and philanthropy? The Knight Foundation is currently considering a proposal to digitize 10 years of IRS 990 nonprofit data and make it available to the public for free. GiveWell’s Alexander Berger, writing on his personal blog, argues that this presents a clear opportunity to GuideStar’s next president to reform its business model around open data. (GuideStar’s current president, Bob Ottenhoff responds in the comments.) And the Foundation Center’s Brad Smith makes a passionate case for data standards and greater transparency among foundations.
• We’ve now entered an era in which college-age students have never known what it’s like to have to pay for music. Casey Rae and J. Holtham have more.
• What is the future of museums?

BIG MONEY

• The Irvine Foundation has announced its first set of grants under its new arts strategy that emphasizes audience engagement.
• Jon Silpayamanant makes the interesting point that sports teams have a performance income gap (i.e., expenses that outpace ticket revenue) just like symphony orchestras do.
• Wait, nonprofits are allowed to have shareholders?
  

  The deal will settle a lawsuit the Koch brothers filed in February over shares that determine control of Cato. It results from the original division of shares between the two Koch brothers, Crane and late Cato Chairman William Niskanen. After Niskanen died of stroke complications in October, the Koch brothers claimed a founding shareholder agreement gave them the option to buy Niskanen’s shares. Crane held they should go to Niskanen’s widow, which would leave him in effective control of the organization.

  The settlement involves dissolving the shareholder agreement. In addition, Crane is expected to retire under an agreement that allows him to select his successor, though the Koch brothers could veto the hiring.

RESEARCH (AND EVALUATION) CORNER

• FSG’s Valerie Bockstette points out the dangers of measuring what’s easy to measure instead of what’s most important.
• The Colorado Health Foundation’s Anne Warhover describes her organization’s approach to impact assessment.
• If you thought the theory of change and measurement framework for ArtsWave was ambitious, just take a look at this new comprehensive sustainability plan for Rockford, IL, which intends to measure economic, social, and environmental outcomes in 16 categories including cultural life and the built environment. The transportation category alone tracks 43 indicators.
• Kudos to the Cultural Policy Center at the University of Chicago for the most blockbuster release of an arts research study so far this year. Called “Set in Stone: Building America’s New Generation of Arts Facilities 1994-2008,” the report takes a critical look at the billions of dollars thrown by arts institutions at new buildings, museum wings, expansions, renovations, etc. during the decade and a half in question. Authored by then-grad-student Joanna Woronkowicz (as her dissertation), Carrol Joynes, and about a half dozen others, “Set in Stone” argues that much of that building boom was of questionable wisdom. The report is available in full multimedia regalia, even including an animated video, and scored a feature in the New York Times, along with reactions from The Atlantic Cities, Lucy Bernholz, the Nonprofit Finance Fund, and Sunil Iyengar (now Woronkowicz’s boss at the NEA’s Office of Research and Analysis). Elizabeth Quaglieri has a helpful summary over at Technology in the Arts. Congratulations, Chicago, you sure know how to get our attention!

ETC.

• Umm, please apply for the Createquity Writing Fellowship, Delali Ayivor?

Now that the talk has subsided a bit, it’s a good opportunity to clarify and elaborate upon some of the subtler points that I was trying to make in that piece, which definitely left a bit of room for people to read in their own interpretations.  So, just to be clear: despite the provocative title, I wasn’t trying to slam the practice of creative placemaking itself, nor call it into question as a focus area for policy and philanthropy in general. As I wrote in response to one of the comments on the original post, I believe strongly in the power of the arts to have a role in revitalizing communities, and I view the desire to direct resources toward bringing such efforts to life as a very positive impulse on the part of funders and policymakers. Furthermore, although I agree with the point made by John Shibley and others that the arts may not be the best way to foment economic development, no one said that cities and regions can only use one strategy. Economic development is a complex beast, and intuition and common sense would hold that there are most likely some specific situations in which the arts can have a real, irreplaceable, and catalytic impact.

My critique is really about how we don’t have much information about what those situations are – nor about how infusions of philanthropic capital can make a difference in those situations. What’s more, I am not confident that the tools we’re currently developing, as useful as they may be for other purposes, will get us there on their own. My contention is that logic models and their conceptual cousins, theories of change, can be useful tools in filling this gap – by forcing us to articulate our assumptions about the way the world works, and by providing a framework that we can use to test those assumptions. The problem is that most of the logic models that I see aren’t worked out to the level of detail that I believe is necessary to gain really useful information about the dynamics of these complex processes. In my post, I provided a couple of examples of theories that, while surely far from perfect, at least attempt to recognize some of the numerous and interlocking assumptions embedded in grantmaking of the kind engaged in by today’s funders supporting creative placemaking.

It’s clear from some of the responses, however, that not everybody shares my optimism in the utility of logic models. Laura Zabel writes that she “hates” them for being too reductive. Diane Ragsdale, taking a cynical view, is worried that they may be misused by funders in order to make themselves seem smarter than they really are or blame grantees for failed strategies. ArtPlace’s response suggests that logic models raise the bar for research too high, and that because proving a causal connection between these investments and the change they produce (or don’t) is so difficult, we’re better off not trying. While I can sympathize with each of these critiques, I also think that they give logic models a bad rap. I feel that logic models are a tool of tremendous power whose potential is only beginning to be unlocked. It’s true that, just like philanthropy and policy, logic models can be done very badly. But that doesn’t mean there’s no gain for us in trying to do them well.

Before we get into all that, however, I’m guessing that some of you probably could use a refresher course on logic models and the terminology associated with them, which can be quite confusing. So let’s start with a little background on what this is all about.

What the Hell Is a Logic Model, Anyway?

Simply put, a logic model is a method of describing and visualizing a strategy. Logic models have their conceptual origin in the “logical framework approach” originally developed for USAID in 1969 by Leon J. Rosenberg of Fry Consultants. Their use was largely concentrated in the international aid arena until 1996, when United Way of America published a manual on program outcome measurement and encouraged its hundreds of local agencies and thousands of grantees to adopt logic models as a matter of course. Since then, large private funders such as the Kellogg and Hewlett Foundations have integrated logic models into their program design and execution, and the concept is commonly taught in graduate programs in public policy, urban planning, and beyond.

Even though logic models have achieved greater adoption over the past several decades, there is little standardization in the content, format, and level of ambition seen in professionally produced logic models for institutions large and small. Worse, everyone seems to want to come up with their own terms to describe features of the logic model, and as a result, you’ll notice a lot of variation in language as well. Below, I’ll do my best to isolate the elements that most of these efforts have in common.

Nearly all logic models contain the following fundamental elements. In combination, they describe a linear, causal pathway between programs or policy interventions and an aspirational end-state.

• Activities are actions or strategies undertaken by the organization that is the subject of the logic model. These activities usually take place in the context of ongoing programs, although they can also be one-time projects, special initiatives, or policies such as legislation or regulation.
• Outputs refer to observable indications that the above activities are being implemented correctly and as designed.
• Outcomes are the desired short-, medium-, or long-term results of successful program or policy implementation.
• Impacts (or Goals) represent the highest purpose of the program, policy, or agency that is the subject of the logic model. Sometimes you’ll find these lumped in with Outcomes.

Logic model for a bicycle helmet public information campaign, courtesy RUSH Project

Many realizations of logic models combine these essential elements with additional information that provides contextual background for this causal pathway. Several of these supplemental concepts are listed below in approximate order from most common to most obscure.

• Measures (or Indicators) for outputs, outcomes, and impacts are concrete, usually quantitative data points that shed light on the degree to which each result has been achieved.
• Inputs are resources available to the program or organization in accomplishing its goals.
• Assumptions are preconditions upon which the model rests. If one or more assumptions proves unsound, the integrity of the model may be threatened.
• Benchmarks extend the concept of measures to incorporate specific target goals (so, not just “# of petitions delivered to Congress” but “50,000 petitions delivered to Congress”).
• Target Population refers to the audience(s) for the activities listed in the logic model.
• Influential Factors are variables or circumstances that exist in the broader environment and could affect the performance of the strategy as designed (e.g., an upcoming election cycle whose outcome might change the underlying landscape in which the program operates).

What About Theories of Change?

A frequently-employed alternative logic model approach strips out this latter set of contextual elements and instead aims to visualize the linear causal chain at a finer grain of detail. This version of a logic model is typically referred to as a theory of change (or, sometimes, a program theory). A well-executed theory of change diagram “unpacks” the processes and factors that lead to successful outcomes, exposing relationships between isolated variables that can then become the subject of research or evaluation.

Partial theory of change from Project SuperWomen case study (ActKnowledge/Aspen Institute Roundtable on Community Change)

Sometimes logic models and theories of change are presented as distinct concepts, while other times they really refer to the same thing. This is because logic models and theories of change evolved out of distinct communities of practice, but the philanthropic field has not always respected the distinction in the terminology it’s adopted to describe these tools. In my own practice I prefer to use theories of change, but for the sake of simplicity and readability, in the rest of this article I’m going to use the term “logic model” inclusively to refer to any diagram that clearly shows some combination of activities and outcomes, regardless of what other elements it may include or the visual approach it takes.

*

OK, now that we have our definitions in order, we can start talking about what makes logic models so awesome.

Awesome #1: Logic Models Describe What’s Already Going On in Your Head

So, here’s the thing: the core questions involved in creating any logic model—What am I trying to do? Why am I trying to do it? How will I know if I’ve succeeded or not?—represent the very essence of strategy. As a rabbi might say, “the rest is commentary.” If you have a strong sense of what the answers to these questions are, then you have a logic model in your head whether you realize it or not. All the diagram does is make it explicit.

To illustrate this, we can look at a simple example. Let’s say I decide I’m done with this whole “arts” thing and I want to go to law school instead. I know, though, that in order to get into a good law school I need to get a good score on the LSAT. So, how can I make sure I get a good score? Intuitively, I decide that taking a test prep class is the way to go.

Why do I think taking a test prep class is a way to increase my score on the LSAT? Well, if my score isn’t as high as it could be, it’s probably due to some combination of two factors. First, I may not know the material well enough. So, if the class helps me to learn how to answer the test questions better, I’ll likely perform better on the test. Second, there may be a psychological factor as well. If I’m someone who gets nervous on tests, then my performance on them may suffer. The practice exams and deep engagement with the material that comes with a class could help me to get more comfortable with the idea of the LSAT and make it seem less intimidating, thus improving my performance.

Seems logical enough, right? And voila, it lends itself quite easily to a logic model:

The truth is that any decision you make, if it has any element of intentionality at all, can be diagrammed as a logic model. You might hate logic models with every fiber of your being and think they’re the stupidest thing ever created, but I’m telling you right now: if you believe in strategy, then you believe in logic models.

Awesome #2: Logic Models Are Incredibly Flexible

Now, there’s a difference between having a logic model in your head and having a good logic model in your head. The example above is simple, but it’s limited by that simplicity. It doesn’t explain why I might have decided to go to law school, or explore other ways that I could get into the school I want besides increasing my test score. In short, it’s pretty much just a straight-up mapping of a decision already made.

The best logic models don’t do that. Instead, they proceed with the end in mind (what is the goal we want to achieve?) and then methodically work backwards to understand what activities or strategies would be most appropriate to achieve that end. The ultimate outcome of this exercise may be a very different set of strategies than the ones you were originally contemplating or the programs you already have in place! Because of that, the logic model creation process can be great for opening up new ways of thinking about old problems or longstanding dreams, as well as clarifying what’s really important to you and/or your organization.

I mentioned earlier that not everyone is a fan of logic models. Here’s what Laura Zabel had to say about them in her post responding to mine:

I hate logic models. For me they are, somehow, simultaneously too reductive and too complex. Too simple, too linear for how I think the world works and too dry, too chart-y for how beautiful the world is. They make me irrationally grumpy.

Arlene Goldbard, in a 2010 essay, is similarly grumpy about logic models:

[R]equiring one of these charts as part of a grant proposal bears about as much real relationship to community organizations’ work as would asking each to weave a placemat…[T]he task of boiling the answers down to colored bars often wastes days, compressing most of the useful meaning out of the inquiry.

I can’t speak to Laura’s and Arlene’s experiences directly, but I know they are not uncommon. Unfortunately, logic models that are rife with imprecision, questionable assumptions, and inappropriate associations are more frustrating to work with than no logic model at all—and it’s not as easy as it looks to create logic models that are free of these flaws. Such problems are magnified when logic models are treated as edicts sent down from on high rather than the learning, living documents that they are intended to be.

In her post, Laura presents an alternative formulation of a logic model that describes her theory of change for creative placemaking: artists + love + authenticity -> creative placemaking. While I’d classify this as more of a definition of creative placemaking than a logic model, it goes a long way toward illustrating my point that we all have latent logic models in our head that are just waiting to be expressed as such. Laura writes, “there’s no logic model in the world that can capture how a crazy parade [the annual MayDay parade in Minneapolis] can restore my faith in humanity.” I couldn’t disagree more – in fact, I made one, relying solely on the way Laura describes the parade in her post. Here you go:

For the past few months, I’ve been researching impact assessment methods used across the social sector in connection with some evaluation work we’re doing here at Fractured Atlas. Seemingly every year, someone comes up with a new way of evaluating impact, whether it’s for social purpose investing, choosing grants, or measuring externalities. I’m not done yet, but what I’ve found so far has only reinforced my appreciation for the logic model. The beauty of logic models is that, because they relate so directly to the fundamental elements of strategy, they are endlessly adaptable to almost any situation. I actually find it kind of funny when people call logic models too rigid, given the alternatives – especially considering how much of our lives is ruled by the granddaddy of rigid, one-dimensional success metrics: money.

Awesome #3: Logic Models Are a Victory for Transparency

Hewlett Foundation Performing Arts Program strategic framework (co-developed in 2008 by yours truly)

One of the really powerful things about downloading the implicit strategy that exists in your head into a diagram is that it confronts you with gaps that may be present in that strategy and allows you to try and work through them. The Underpants Gnomes example that I used in my creative placemaking post is a great illustration of this. The Gnomes clearly felt that stealing underpants would lead to profit, but hadn’t clearly thought through the fuzzy middle part of the scheme. The Underpants Gnomes might seem like a fanciful exaggeration of the problem I’m talking about, but I would argue that there’s been more than one arts organization (and funding initiative) started with a similar lack of congruity between the proposed activities and intended results.

Diane Ragsdale is skeptical that logic models can serve this function, suggesting that funders might misuse logic models and turn them against their grantees. First, I think it’s important to make a distinction here: Diane and Arlene and Laura are all talking about logic models at the level of the individual organization/project. While I think these can be helpful, in my mind the most important logic model is the one for the funder itself. This is admittedly a rarer practice, but several foundations – such as Ford, Hewlett, McKnight, and Boston – have taken the steps of not only developing a logic model to describe their grantmaking strategies, but sharing that logic model publicly. Those that don’t publish at least sometimes make them available to peers outside the organization. Once a logic model is “out there,” there’s no taking it back. Therefore, logic models both pull back the curtain on a funder’s current thinking and also make it harder to project the illusion after the fact that a funder knew how things were going to work out all along.

Even more importantly, though, logic models are a victory for transparency with oneself, not just with others. The most important part of any logic model creation process is the set of assumptions revealed about how your program or organization works, and what it needs to be successful. Sometimes these assumptions might seem like no-brainers, and other times they will seem as unproven as they are central. Being comfortable with naming your assumptions as such is not just good practice for ensuring that your organization is constantly learning and growing. It’s also extremely helpful on a psychological level for dealing with the specter of possible failure. Because logic models explicitly draw a distinction between program design and program execution, they acknowledge the very real possibility that you could be doing your job perfectly and your program could still fail, because its theoretical foundation rests upon faulty assumptions. This is an incredibly freeing realization, because it means that radically changing or even scrapping a program that isn’t working doesn’t necessarily have to mean changing program leadership.

One of the reasons people sometimes feel anxious about evaluation and measurement is because they’re afraid of being held accountable, especially to things that they don’t have full control over or to metrics that don’t seem relevant to what they’re trying to do. When that happens, there are enormous incentives on managers and their supervisors to “cook the books” or otherwise game the system to show results that look better than reality, because any failure—even failures that are no one’s fault—reflects on them personally. That’s the danger of trying to enforce a data-driven culture without first developing the theoretical frameworks that determine what data you’re trying to collect. Because logic models separate the person from the program, they can distinguish between lagging initiatives that might just need more time to prove themselves, and failures of design that can be transformed into productive learning opportunities.

A Note About Logic Models and “Proof”

One of the criticisms directed at logic models generally, and by ArtPlace at my post specifically, is that they promote an impossible standard for proof. Here’s what ArtPlace had to say about it:

A critical limitation of elaborate logic models of the style developed by Moss is that it is nearly impossible to quantify or measure all of the different factors and relationships proposed. While many of the asserted relationships are plausible…almost none are measured in practice. Many – if not most – of the variables…could only be gauged imprecisely at great expense or are not susceptible of measurement at all. Multiply this problem by the 50 variables the model uses and the dozens of relationships it asserts, and it’s clear that it is beyond anyone’s ability to actually prove or disprove the model for even a single metropolitan area, much less the nation.

In one sense this criticism is right: these are difficult research challenges that highly qualified professionals have been struggling to address for decades, some of the more promising approaches demonstrated at the recent NEA/Brookings Institution convening on economic development and the arts notwithstanding.

But contrary to popular perception, the term “scientific proof” is a misnomer: proof is a concept germane to mathematics, not science. (Admittedly, my previous post could have been clearer on this point.) Scientists, including social scientists, develop hypotheses about how the world works and then gather evidence to support or undermine those hypotheses. Whereas proof is black-and-white, evidence has shades of gray: it can be strong or weak, circumstantial or conclusive. My colleague Kiley Arroyo made a great courtroom analogy in response to my creative placemaking post: she wrote, “think of forensic analysis if you will. You’re not just going to look at where and how the bullet hit, but what it was shot from, where, by whom and why.” Our job as researchers is not to “prove” anything – instead, it’s to amass evidence in search of the truth.

The biggest problem that I see with most logic models is that they are too simple for what they are trying to describe, and thus consign us to amassing a whole bunch of weak evidence. Logic models are often developed as much for communication purposes as for research, and can thus face intense pressure to be “dumbed down” for public consumption. I frequently hear comments like, “make sure that there are no more than five categories, because after that you’re going to lose people,” and God help you if your diagram doesn’t all fit on one page. But think about it, would you apply this standard to a budget? To a work plan? For a major, mission-critical project with many moving parts? I’m not interested in logic models as a communication tool; I’m interested in them as a means to help us do our jobs better.

In the case of ArtsWave, we do intend to collect data to show progress towards the goals that have been established through the model. But the task is less daunting than it looks. Because we are not trying to “prove” the model, not everything has to be measured directly; indeed, not everything has to be measured at all! Instead, the model serves as a road map for us in considering what is most important to measure, given what we don’t know. Is it that assumption that the arts can differentiate Cincinnati from its peer cities in the minds of tourists and potential residents? Can we be confident that people from diverse backgrounds will interact with each other if they happen to come together for the same community-wide event? A smart research design will test the assumptions that are most in need of testing, and the purpose of the modeling exercise is to identify which assumptions those are.

It’s not like we are all alone in this effort, either. There is an ever-growing body of literature on the ways that the arts interact with communities, and there is no need for us to demonstrate yet again connections for which strong evidence exists in other contexts. Furthermore, since many of the data points involve stakeholders beyond the arts, there is an opportunity to collaborate with other local entities to share resources and develop knowledge infrastructure collaboratively. Cincinnati is home to the STRIVE initiative, which has been made famous in the broader social sector as the poster child for the “Collective Impact” concept as coined by consultants from FSG Social Impact Advisors. One of STRIVE’s chief accomplishments has been the development of a “adaptive learning system” for use by hundreds of education nonprofits in the region, which has helped align the efforts of those organizations around a common set of purposes and benchmarks. If it worked in education, and in the same city no less, why can’t it work in the arts?

But that being said, none of the three benefits that I’ve cited above—articulation, flexibility, and transparency—require “proving” the logic model. They all come, at least in part, just from creating one. And creating a logic model doesn’t have to be a tortured, involved process. It doesn’t have to cost hundreds of thousands of dollars. You can write them out on a back of a napkin (okay, for some things you might need a really big napkin). Creating a logic model for something that will require a lot of your time or money is one of the most highly leveraged activities you can possibly undertake. I hope more arts funders will undertake it.