A graphic from the Cultural Data Project report, “New Data Directions for the Cultural Landscape.”

As any internet geek or high-priced consultant will be happy to tell you, we find ourselves today in the age of Big Data. You know, the era when science and numbers are supposed to solve all our problems forever? That one. And yet in the cultural sector, according to a report published earlier this year, we don’t have the data we need; we don’t know what to do with the data we have; and even if we did, we still wouldn’t use it to make decisions. (Okay, that may be oversimplifying things a bit…but not by much.)

So what are we supposed to do? That’s what the Cultural Data Project (CDP), which commissioned the report, wanted to find out. Many readers know the CDP as the folks behind those forms you have to fill out when you’re applying for grants. Its spinoff last year from a foundation-sponsored initiative housed within the Pew Charitable Trusts to an independent nonprofit prompted some organizational rethinking, and “New Data Directions for the Cultural Landscape,” by Slover Linett consultants Sarah Lee and Peter Linett, was one result. “New Data Directions” seeks to situate the CDP’s efforts within the larger context of data collection throughout the United States cultural sector.

The study synthesizes comments from an online forum that CDP hosted in late 2013 with a small group of cultural data experts drawn from academia and the consulting world. (Disclosure: Createquity’s Ian David Moss was one of the participants in this forum and is quoted a handful of times in the report.) It additionally draws from CDP’s internal strategic planning survey, a paper by Margaret Wyszomirski (not available online) that sought “to frame and inventory the cultural data landscape,” and the authors’ own interpretations and experiences.

After citing the benefits that data-informed decision making has provided in other fields, “New Data Directions” identifies a number of factors that it says are preventing us from reaping those benefits, namely:

• Poor accessibility, quality, and comparability of cultural data
• Norms about data collection and use, including low priority/importance assigned to the task of data collection in general
• Lack of coordination and standardization among existing data collection efforts
• Skill and resource capacity constraints among cultural nonprofits
• Organizations’ perceptions of the public and their audiences that inhibit the effective use of data in decision-making
• A paucity of vision and lack of role models regarding the successful use of data to drive decisions

To address these challenges, the report recommends some straightforward steps, including coordinating leadership on cultural data, engaging program and artistic staff in conversations about data, shifting the frame of data use from accountability to decision-making, developing a field-wide research and data collection agenda, developing data-related skills among organization staff, and improving the cultural data infrastructure.

Piece of cake, right? Though the report’s heavy reliance on the comments of a group of data experts might suggest that we’re off to a bad start in terms of “engaging program and artistic staff in the conversation about data,” Lee and Linett nevertheless surface several important and underaddressed issues facing the cultural data landscape. Perhaps the most crucial of these is a recognition that leading with data as the solution before carefully considering the problem it’s supposed to solve runs the risk of creating data that lacks a purpose. In many cases, arts organizations’ collection of data has been driven by the need to comply with funders’ reporting requirements rather than by a desire to collect information that could improve their future decision making. While the databases that have been generated through this process provide rich sources of information, it is not always clear what that information is good for, or how individual organizations can benefit from it. “New Data Directions” deftly calls attention to this “data first, questions second” mindset that appears to be so pervasive across the sector.

And yet to some extent, the report is itself a testament to the difficulty of escaping this mindset. By framing its entire exploration around data, “New Data Directions” inadvertently obscures the role that other forms of information and analysis, such as literature review, strategic goal setting, and probability assessment, can play in promoting better decisions. More to the point, if better decisions for better outcomes are what we’re really interested in, how much is suboptimal use of data really sabotaging that goal? The report seems simply to take it on faith that it is, without providing much evidence for that view or weighing how much other obstacles (e.g., management incentives) may be contributing to the problem. In part because of this focus on the activity rather than the goal, “New Data Directions” does a much better job of describing where we are and suggesting a path forward than imagining what we might find on the other side.

So it seems that last task is left to the rest of us. Without ever quite coming out and saying it, “New Data Directions” challenges all arts administrators to think purposefully about our role in addressing the situation mentioned at the top of this article. Are we ready to declare a crisis around data collection and use in the field? Do the observations and subjective impressions offered in the report resonate with your own experience? What would consistently effective use of data for decision making at the organizational and system-wide level look like in practice? What good would it do for the sector? And what would it take to make change in this particular way?

Your turn.

Cover image: “Data Represented in an Interactive 3-D Form,” courtesy of the Idaho National Laboratory via Flickr Creative Commons license.

If you’ve been paying any attention at all to technology trends the past few years, you know that we live in the era of Big Data. All of those videos we upload to YouTube, hard drives we fill with government secrets (or cat photos, take your pick), and tweets we awkwardly punch out on touchscreen keyboards add up to a whole lot of gigabytes, the bulk of which are stored by someone, somewhere, indefinitely. By some estimates, human beings generate more data every two days than we did in the entire history of civilization prior to 2003 – and that was as of three years ago!

Indeed, these are exciting times for data nerds, and data nerds in the arts are no exception. Initiatives such as the Cultural Data Project, Southern Methodist University’s National Center for Arts Research, and the Americans for the Arts National Arts Index seek to collect or organize relevant indicators pertaining to everything from arts organizations’ financial health to audience reach and characteristics to long-term trends for musical instrument purchases.

Fractured Atlas is no stranger to data initiatives in the arts. Our Archipelago data visualization software is one of the largest such efforts, bringing together information on arts nonprofits, for-profits, fiscally sponsored projects, funding, audience distributions, and community context all in one place in the service of better understanding the arts ecosystem in a region. Facilitating data-driven decisions is a major long-term objective of Artful.ly, our just-launched cloud-based arts management tool, and a present-day reality for Spaces, our venue listing and booking service that can promote spaces with last-minute availability to users. Through our research advisory services work, we’ve helped funders such as ArtsWave organize their entire grantmaking process around principles of data-driven decision-making in order to further their philanthropic objectives. Everyone benefits when funders, organizations and individuals in the arts ecosystem make thoughtful decisions about resource allocation, setting up and responding to incentives, and more. At Fractured Atlas, we believe that data can and should be a crucial input into that thoughtful decision-making process, and we’ve been increasingly vocal in evangelizing for data-driven decision making throughout the arts and cultural sector.

There’s just one problem. Up until now, Fractured Atlas has not had any formal guidelines in place to ensure that we use data in our own decision making, with the result that our internal decisions – relating to management, marketing, strategy, and the like – have been guided primarily by managerial intuition. In a “doctor, heal thyself!” moment, we’ve agreed that is time for our practices to reflect our preaching, at both the program and institutional levels. In 2013, the scope of our operations, the size of the community we serve, and the financial stakes in our work demand informed analysis at a level of rigor that we have not historically practiced. (This directive was immortalized by our fearless leader Adam Huttler in the organization’s annual Strategic Priorities Memo with the colorful title, “Eating Our Data-Driven Dog Food.”)

So between now and next summer, Fractured Atlas is embarking on a pilot initiative to explore how we can use data and evidence to improve our decision-making process at all levels. We’re calling it Fractured Atlas as a Learning Organization, and through this and future blog posts, we’re giving you the opportunity to be a fly on the wall as use this process as a way of grappling with issues of organization identity, strategy, culture, and impact.

What Is a Learning Organization?

As I define it*, a learning organization is one for which information and strategy are joined at the hip. It is, quite literally, an organization that has successfully forged a culture of learning and integrated that culture into its decision-making process at all levels.

Why is this integration between information and strategy important? Because every organization operates in an environment of uncertainty about what is going to result from its decisions, and every decision we make on behalf of an organization is based on a prediction, whether explicitly articulated or not, about the results of that decision.

If you can reduce the uncertainty associated with your decisions, the chances that you will make the right decision will increase. Of particular interest here  are what I call decisions of consequence: dilemmas for which the consequences of making the wrong decision and uncertainty about the nature of the right decision are both high. So, how do you reduce that uncertainty? Why, through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile.

In fact, I subscribe to the notion that research is only valuable insofar as it helps to answer a question that matters. I’m not the only one who thinks so, either: Jake Porway, the founder of a nonprofit that connects data scientists with social enterprises in need, wrote this past spring that “any data scientist worth their salary will tell you that you should start [a data project] with a question, NOT the data.” In fact, all of the excitement around Big Data notwithstanding, data divorced from strategy is not likely to be very useful.

A learning organization solves this problem by forging a powerful feedback loop between information and strategy, with each feeding the other and adapting in relation to the other. The more obvious implication of this symbiosis is that organizational decisions must adapt in response to new information, as discussed above. But the less obvious implication is no less important: information-gathering must be directed by the organization’s decision-making needs. Without that intimate connection, there are no real safeguards to prevent organizations from thinking they are making data-driven decisions without really putting much thought into either the data or the decisions.

More broadly, a learning organization develops a culture of seeking out and using information thoughtfully from the highest levels to the organization’s grassroots. The most effective organizations are conscious about the impact they are trying to achieve, and willing to be open-minded regarding the paths they take to maximizing that impact.

*Some readers may be familiar with the term “learning organization” as defined by MIT management scientist Peter Senge in his well-known 1990 book The Fifth Discipline. My use of the phrase is broadly in the same spirit as Senge’s, but he sets out a very specific formula for what constitutes a learning organization that I don’t make use of here.

Fractured Atlas as a Learning Organization

This fiscal year, which started in September and goes through next summer, we are undertaking a pilot project to put some of these principles into practice. The primary goal of the pilot is to develop a conceptual framework and a toolkit of situation-adaptable methods for reducing uncertainty about decisions of consequence. If we can reduce the uncertainty we have about those decisions through strategic measurement and information-gathering efforts, over time we’re likely to make better decisions that will in turn lead to better outcomes for Fractured Atlas and the people who benefit from our work.

As powerful as this idea is, it only works if we have a very concrete sense of what we’re trying to accomplish as an organization. While we’ve had a mission statement for some time now, the huge variety of programs and services Fractured Atlas offers is virtually impossible to fully capture in a single sentence. Accordingly, the first step in this process is to create a theory of change for every program at the organization, from which we’ll roll up an overall theory of change and logic model for the organization as a whole. This will allow us to define our overall goals as well as some key success metrics at various levels of operation, taking into account both Fractured Atlas’s mission objectives and its focus on developing programs that are sustainable with earned income.

Meanwhile, we’ve formed an internal task force to work on this project at a deeper level of engagement throughout the year. Affectionately called the Data-Driven D.O.G. Force (the “D.O.G.” stands for Data Over Gut), the group will meet every 6-8 weeks to receive calibrated probability assessment training, identify real-world decisions of consequence to use as case studies, and come up with measurement experiments to gather information relevant to those decisions. In doing so, we’ll be using a modified version of a methodology called Applied Information Economics as described in Douglas W. Hubbard’s book How to Measure Anything: Finding the Value of “Intangibles” in Business. One major advantage of AIE is that it explicitly takes into account the cost-benefit of measurement strategies by calculating something called the value of information, which we’ll be exploring further in a future post.

At the end, we’ll attempt to formalize a process for identifying decisions of consequence in the future and fitting measurement strategies to the situation at hand. We’ll also present some recommendations for building infrastructure in the form of ongoing data collection, to address those questions that are likely to be asked again and again. And through it all, I’ll be writing about it here – so that anyone who wants to can learn alongside us.

Learning in Context: Why Philosophy Matters as Much as Performance

Data-driven decision-making isn’t just about crunching numbers. It’s a practice that requires certain values in order to work. The hard part of being data-driven is not the “data” but the “driven” – you have to be willing to question your assumptions and actually change your behavior in response to the new information coming in. Put another way, a learning organization is, well, open to learning new things -even things that suggest that the way that we’re currently doing things isn’t working as well as it could, or that we’re missing important opportunities to increase our impact.

It’s much easier to attain that kind of open stance if we train ourselves to expect failure upfront. In general, organizations as well as people have a tendency to be far too risk-averse. Being a learning organization means embracing a culture of intentional experimentation and productive failure: we’re likely not going to hit upon the secret sauce the very first time we try something – or, sometimes, at all.

Being a learning organization similarly requires that we think about ourselves from a system perspective – how are we making a difference in light of what everyone else is doing? And how can our experiences shed light on those of others? That’s why we’re not just going down this path on our own and in private. If the specific activities of the pilot project turn out to be a big waste of time (and I can’t guarantee that they won’t), we won’t be able to hide that from you or the world. But even that would ultimately be a good thing – because, in true learning organization fashion, it would cause us to reconsider the limitations of a data-driven approach. Embracing change is hard, but one of the very best things about it is that it can allow us to extract just as much (if not more) value from failure as success.

For me, personally, this project is very exciting. Of course I’m eager to find out what we’ll learn. But more than that, Fractured Atlas as a Learning Organization is an opportunity for us to exercise leadership in a way that reaffirms our highest standards for ourselves and for the field. I’m looking forward to sharing our journey with you.

Since I didn’t write out the speech in advance, I don’t have a transcript for it. However, below is a reconstruction of the new material from my notes, so you can get a taste for it if you don’t have time to watch the whole thing right now. (You’ll notice I make a number of generalizations in the speech about the ways in which arts practitioners interact with research. These are based on observation and personal experience, and are best understood as my working hypotheses.)

*

[starting at 26:55]

Why is this integration between data and strategy important? Because research is only valuable insofar as it influences decisions. This is why logic models are awesome – they are a visual depiction of strategy. And there is no such thing as strategy without cause and effect. Think about that for a second. Our lives can be understood as a set of circumstances and decisions. We make decisions to try to improve our circumstances, and sometimes the circumstances of those around us. Every decision you make is based on a prediction, whether explicitly articulated or not, about the results of that decision. Every decision, therefore, carries with it some degree of uncertainty. This uncertainty can be expressed another way: as an assumption about the way the world works and the context in which your decision is being made. These assumptions are distinguished from known facts.

If you can reduce the uncertainty associated with your assumptions, the chances that you will make the right decision will increase. So, how do you reduce that uncertainty? Through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile. Alas, research and practice in our field are frequently disconnected in problematic ways. Six issues are preventing us from reaching our potential.

Issue #1: Capacity

Supply and demand apply as much to research as it does to artists. There are far more studies out there than a normal arts professional can possibly fully process. I wish I could tell you how many research reports are published in the arts each year, but nobody knows! To establish a lower bound, I went back over last year’s [2011] “around the horn” posts, which report new research studies that I hear about. I counted at least 41 relevant arts-research-related publications – a tiny fraction, I’m sure, of total output. To make matters worse, research reports are long, and arts professionals are busy. For the Createquity Writing Fellowship program, participants are required to analyze a work of arts research for the Createquity Arts Policy Library. I collect data on how long it takes to do this, and consistently, it requires 30-80 hours to research, analyze and write just one piece! Multiply this by the number of new studies each year, and you can start to see the magnitude of the problem.

Issue #2: Dissemination

Which research reports is an arts practitioner likely to even know about? Certainly not all of them, because there is almost no meaningful connection between the academic research infrastructure and the professional arts ecosystem. Lots of research relevant to the arts is published in academic journals each year, but unless the faculty member was commissioned to do their work by a foundation, we never hear about it. Academic papers are typically behind a pay firewall, and most arts organizations don’t have journal subscriptions. To give an example, after I wrote about Richard Florida’s Rise of the Creative Class, Florida pointed me to a study in two parts by two Dutch researchers. It’s one of the best resources I’ve come across for creative class theory, but I’ve never heard anyone even mention either study other than him and me.

Issue #3: Interpretation

Research reports inevitably reflect the researcher’s voice and agenda. This is especially true of executive summaries and press releases, which is often all anyone “reads” of research “reports.” Probably the most common agenda, of course, is to convey that the researcher knows what he/she is talking about. Another common agenda is to ensure repeat business from, or at least a continuing relationship with, the client who commissioned the study. The reality, however, is that research varies widely in quality. There’s no certification process; anyone can call themselves a researcher. But even highly respected professionals can make mistakes, pursue questionable methods, or overlook obvious holes in their logic. And, in my experience, the reality of any given research effort is usually nuanced – some aspects of it are much more valuable than others. Unfortunately, many arts professionals lack expertise to properly evaluate research reports, not having had even basic statistics training.

Issue #4: Objectivity

Research is about uncovering the truth, but sometimes people don’t want to know the truth. Advocacy goals often precede research. How many times have you heard somebody say a version of the following: “We need research to back this up”? That statement suggests a kind of research study that we see all too often: one that is conducted to affirm decisions that have already been made. By contrast, when we create a logic model, we start with the end first: we identify what we are trying to achieve and only then determine the activities necessary to achieve it.

Here are a bunch of bad, but common reasons to do a research project:

• To prove your own value.
• To increase your organization’s prestige.
• To advance an ideological agenda.
• To provide political cover for a decision.

There is only one good reason to do research, and that is to try to find out something you didn’t know before.

Issue #5: Fragmentation

The worst part of the problem I just described is that it drives what research gets done – and what doesn’t get done. There is no common research agenda adopted by the entire field, which is a shame, because collective knowledge is pretty much the definition of a public good: if I increase my own knowledge, it’s very easy for me to increase your knowledge too. The practical consequences of this fragmentation are severe. It results in a concentration of research using readily available data sources (ignoring the fact that the creation of new data sources may be more valuable). It results in a concentration of research in geographies and communities that can afford it, because people don’t often pay for research that’s not about them. And it results in a concentration of research serving narrow interests: discipline-specific, organization-specific, methodology-specific. My biggest pet peeve is that research is almost never intentionally replicated – everybody’s reinventing the wheel, studying the same things over and over again in slightly different ways. A great example of a research study crying out for replication is the Arts Ripple Effect report, which I talked about earlier. The results of that study are now guiding the distribution of millions of dollars in annual arts funding. Are those results universal, or unique to the Greater Cincinnati region? We have no way to know.

Issue #6: Allocating resources

Everyone knows there’s been a trend in recent years towards more and more data collection at the level of the organization or artist. Organizations, especially small ones, complain all the time about being expected to do audience surveys, submit onerous paperwork, and so forth. And you know what, I agree with them! You might be surprised to hear me say that, but when you’re talking about organizations that have small budgets, no expertise to do this kind of work, and the funder who is requesting the information is not providing any assistance to get it…just take a risk! You make a small grant that goes bad, so what? You’re out a few thousand dollars. The sun will rise tomorrow.

As an example of what I’m talking about, I participated in a grant panel recently. I enjoyed the experience, and am glad I did it, but there’s one aspect of the experience that is relevant here. There were seven panelists, and we were all from out of town. Each of us spent, I’d say, roughly 40 hours reviewing applications in advance of the panel itself. Then we all got together for two full days in person to review these grants some more and talk about them and score them. We did this for 64 applications for up to $5,000 each, and in the end, 92% 94% were funded.

So consider this as a research exercise. The decision is who to give grants to, and how much. The data is the grant applications. The researchers are the review panel. What uncertainty is being reduced by this process? How much worse would the outcome have been if we’d just taken all the organizations, put them into Excel, run a random number generator, and distributed the dollars randomly up to $5,000 per organization? And I’m not saying this to make fun of this particular organization or single them out, because honestly it’s not uncommon to take this kind of approach to small-scale grantmaking. And yet if you compare it to ArtPlace’s first round of grants, theoretically they had thousands of projects to choose from, and they gave grants up to $1 million for creative placemaking projects – but there was no [open] review process; they just chose organizations to give grants to. So there’s a bit of a mismatch in the strategies we use to decide how to allocate resources.

There’s a concept called “expected value of information” described in a wonderful book called How to Measure Anything, by Douglas W. Hubbard. It’s a way of taking into account how much information matters to your decision-making process. In the book, Hubbard shares a couple of specific findings from his work as a consultant. He found that most variables have an information value of zero; in other words, we can study them all we want, but whatever the truth is is not going to change what we do, because they don’t matter enough in the grand scheme of things. And he also found that the things that matter the most, the kinds of things that really would change our decisions, often aren’t studied, because they’re perceived as too difficult to measure. So we need to ask ourselves how new information would actually change the decisions we make.

There is so much untapped potential in arts research. But it remains untapped because of all the issues described above. So what can we do about it?

First, we need a major field-building effort for arts research. Connecting researchers with each other through a virtual network/community of practice would help a lot. So would a centralized clearinghouse where all research can live, even if it’s behind a copyright firewall. The good news is that the National Endowment for the Arts has already been making some moves in this direction. The Endowment published a monograph a couple of months ago called “How Art Works,” the major focus of which was a so-called “system map” for the arts. But the document also had a pretty detailed research agenda for the NEA, not for the entire field, that lays out what the NEA’s Office of Research and Analysis is going to do over the next five years, and two of the items mentioned are exactly the two things I just talked about: a virtual research network and a centralized clearinghouse for arts research.

This new field that we’re building should be guided by a national research agenda that is collaboratively generated and directly tied to decisions of consequence. The missing piece from the research agenda in “How Art Works” is the tie to actual decisions. Instead it has categories, like cultural participation, and research projects can be sorted under those buckets. But it’s not enough for research to simply be about something – research should serve some purpose. What do we actually need to know in order to do our jobs better?

We should be asking researchers to spend less time generating new research and more time critically evaluating other people’s research. We need to generate lots more discussion about the research that is already produced. That’s the only way it’s going to enter the public consciousness. Each time we fail to do that, we are missing out on opportunities to increase knowledge. It will also raise our collective standards for research if we are engaging in a healthy debate about it. But realistically, in order for this to happen, field incentives are going to have to change – analyzing existing research will need to be seen as equally prestigious and worthy of funding as creating a new study. Of course, I would prefer if people are not evaluating the work of their direct competitors – but I’ll take what I can get at this point!

Every research effort should take into account the expected value of the information it will produce. Consider the risk involved in various types of grants made. What are you trying to achieve by giving out lots of small grants, if that’s what you’re doing? Maybe measure the effectiveness of the overall strategy instead of the success or failure of each grant. This is getting into hypothesis territory, but based on what I’ve seen so far I would guess that research on grant strategy is woefully underfunded, while research on the effectiveness or potential of specific grants is probably overfunded. We probably worry more than we need to about individual grants, but we don’t worry as much as we should about whether the ways in which we’re making decisions about which grants to support are the right ways to do that.

Finally, we should be open-sourcing research and working as a team. I’m talking about sharing not just finished products and final reports, but plans, data, methodologies as well. I’m talking about seeking multiple uses and potential partners at every point for the work we’re doing. This would make our work more effective by allowing us to leverage each other’s strengths – we’re not all experts at everything, after all! And it would cut down on duplicated effort and free up expensive people’s time to do work that moves the field forward.

I thank everyone for their time, and I’d love to take any questions or comments on these thoughts about the state of our research field.

• The National Endowment for the Arts has spearheaded the formation of a new coalition of private funders to support its creative placemaking agenda. Called ArtPlace, the collaboration features Carol Coletta as its fearless leader, and has the backing of such significant national funders as the Ford, Knight, Kresge, Rockefeller, and Mellon Foundations. Grants and a $12 million loan fund are administered through the Nonprofit Finance Fund, a nonprofit lender and financial consulting organization. ArtPlace has already made a set of 34 grants in “record time” totaling $11.5 million to a range of projects in the Our Town vein (including one to Coletta’s former employer, CEOs for Cities, in an cool-looking collaboration with GOOD Design.). Thankfully, after a closed-door process for this first round, ArtPlace is opening up next year’s grants through a letter of inquiry which is due November 15. Coletta has more at the Art Works blog.

PHILANTHROPY

• A couple of weeks ago, Sean Stannard-Stockton asked a seemingly innocent question: who should be the Hewlett Foundation’s next president? I was surprised to see Sean wrote a follow-up in which he shares that “emails I’ve gotten from very senior members of the philanthropic community – people whose opinions I respect very much – suggest that my hosting this discussion is far more controversial than I might have guessed.” Apparently, according to these Very Senior People (none of whom, Sean notes, are Hewlett employees), speculating about who might ultimately be the driving force behind the distribution of hundreds of millions of tax-exempt philanthropic dollars a year should be off-limits to plebes who are not on the Hewlett Foundation Board. Thankfully, Sean elected not to listen to this silliness and has gone ahead and published the suggestions that have come in, which include some very interesting names.
• Speaking of Hewlett, Emiko Ono will be the new Program Officer for the Foundation’s Performing Arts Program, replacing Marc Vogl. Ono was Director of Grants and Professional Development for the Los Angeles County Arts Commission.
• Duke’s Fuqua School of Business has announced a first-of-its-kind “Initiative on Impact Investing.” Officials at the school’s Center for the Advancement of Social Entrepreneurship (CASE) will be developing a new course, case materials, and working with practitioners to explore impact investing in more depth in an academic setting.
• Ouch: the Center for Effective Philanthropy reports that community foundation leaders are far less strategic in their work than their rhetoric would suggest. In particular, “CEOs who are strategic in their donor work focus explicitly on how donor contributions will benefit the community. In comparison, nonstrategic CEOs focus on how donor contributions will continue to flow to the foundation.”

IN THE FIELD

• After a decade of planning and building, Kansas City’s $326 million Kauffman Center for the Performing Arts has opened – one of the last products of the performing arts building boom of the 1990s-2000s.
• Doug Borwick, the Association of Arts Administration Educators president who has a new blog on ArtsJournal called Engaging Matters, writes a love letter to the much-missed Community Arts Network.
• And here’s an inside look at Yerba Buena’s personalized membership program, YBCA: YOU, with more from Borwick.
• Words I’d never thought I’d write department: congratulations to Philly’s Mural Arts Program, which landed a cover story…in AAA New York Car & Travel magazine!
• Composer Nico Muhly offers an insider’s perspective on the byzantine restrictions faced by orchestral composers seeking access to recordings of their own work.
• Sally Gaskill, who runs the Strategic National Arts Alumni Project at Indiana University, interviews Angela Myles Beeching, director of the Center for Music Entrepreneurship at Manhattan School of Music, about preparing musicians for careers outside the academy.
• And speaking of professional training degrees for artists, excuse Poets & Writers magazine for trying to give people some sense of how to choose a creative writing MFA program. According to an angry group of writing faculty, the fact that the rankings take financial aid too heavily into account is enough to break out the pitchforks.

THOUGHT BUBBLES

• A very interesting interview with John Kreidler about his cultural policy simulation game, Medici’s Lever.
• Cool true-life story of the birth of an internet meme, conceived by the arts blogosphere’s own Chris Ashworth.
• Clay Lord offers a rare look at the neuroscience of audience response to theater.
• Arlene Goldbard offers her vision of how the arts could play a role in a new stimulus.
• Plagiarism appears to be on the rise in the internet age, even among doctoral students: a study of 120 dissertations in psychology turned up instances of plagiarism (defined as using 10 or more words from another source verbatim without attribution) in a shocking 80% of them. If even the future teachers are plagiarizing, what does that mean for the long term trend?
• Surprise, surprise: when you raise prices 60%, you might lose some customers.
• Two months ago, shoemaker Converse opened up a free recording studio in the ultra-hip neighborhood of Williamsburg, Brooklyn. The New York Times spends a day there talking to the artists taking advantage of the service. A competitive process is used to identify musicians, and as with Kickstarter’s “Projects We Love” (see below), artists are chosen “less for their talent than for their viral energies — their presence on MySpace or Facebook, their hustle in pursuing their careers.”
• I found this live improvised playwriting experiment between Neil Labute and Theresa Rebeck kind of cool, despite the hokey setup.
• I’ve never understood why anyone would want a tattoo, but it seems buyer’s remorse is at an all-time high. Unfortunately, tattoo-removal-seekers are finding that it’s not so easy to turn back time.

RESEARCH CORNER

• In connection with the Artist Revenue Streams project, Future of Music Coalition and the Field are co-hosting (along with a boatload of other organizations including Fractured Atlas) a workshop for musicians on “accounting for creatives” in NYC on Monday, September 19. Check the link for info.
• Nice to see an organization that just released a research report (the Center for Effective Philanthropy, in this case) openly discussing how response bias might have affected the results. We need to see more of this kind of transparency in reporting research results.
• I was intrigued to hear of the formation of a new music research and composition e-journal series on the Social Science Research Network database, led (mostly) by Boston University faculty.
• Andrew Taylor points us to a new book by the Curb Center’s Steven Tepper on protest and the arts.
• Two economists estimate the “psychic value” of a work of art (as distinct from its investment value) at 28% of its overall price.

LOOKING BACK

(the following are some “retro” links from the past 12 months that for one reason or another didn’t make it into the around the horn wrap-ups the first time around.)

• Joshua Phillips lays out a very serious and detailed proposal (and justification) for a public policy game show.
• It’s not too often that I see a coherent conservative case against arts funding, but here’s an example for those who might be curious. Main arguments: the evidence of the arts’ economic impact is flimsy, and government funding makes for worse art.
• It was hard to suppress a wry smile upon learning that Leona Helmsley’s precious dog Trouble, to which she left $12 million in her will (disowning two of her grandchildren in the process), has passed away. The funds held in Trouble’s trust have reverted to Helmsley’s charitable foundation, which is one of the largest in the world.
• Wondering where our nation’s sudden income inequality came from? Since 1992, super-wealthy Americans’ effective tax burden has plummeted by more than a third. Over the same time period, the effective tax rate for all taxpayers has dropped only 6%.
• Is subway pop-up theater the new flashmob-chorus/dance/opera-in-a-mall?
• I found this quote worth mulling over, from the NYTimes Magazine’s writeup of Kickstarter last month:
  

  I sat in on a meeting where the [“Projects We Love”] newsletter picks were made. During the half-hour or so Strickler and the team discussed the choices, I was struck by how often they talked not about the projects but about the pitches. “His video is so boring.” “What are the rewards?” “Why is this cool?” They were focused on the project ideas through the filter of “the Kickstarter project” as a form. “We have values,” Chen told me, and they boil down to prizing creators who respect its proc­ess. They favor creators who think through the rewards for backers, get the word out and engage an audience. In other words, the process doesn’t shape the aesthetic. It is the aesthetic.

I was a participant in a couple of conversations with fellow arts research nerds recently in which we discussed the notion of cause and effect. You remember that one from grade school, right? Well, it turns out that when it comes to research (and especially arts research), it’s not as simple as we all thought.

You see, in science, when we say that something caused something else, we tend to want to be sure. A common concept in statistics is “significance“: the idea that a meaningful connection exists between two variables that can’t be explained by random noise. If you’ve formed a hypothesis and designed your experiments correctly, and you get statistically significant results, you can be fairly confident that the results you’re looking at indicate something real and not merely an accident or coincidence.

Of course, it’s never possible to be entirely sure. But in some fields, you can get pretty close. The technical term for the degree of uncertainty we’re willing to tolerate in an analysis like this is the alpha: an alpha of 0.05 means that we’d like to be 95% sure that we can reject the null hypothesis (i.e., that we might just be looking at random noise) before we go ahead and report the result as meaningful. In some fields, it’s common to require an alpha of 0.001 or even lower – that is, a 99.9% certainty that random variation isn’t behind the results. And for something like testing a new drug, you most definitely want to be that sure that, for example, it doesn’t cause heart attacks as a side effect before you put that sucker on the market.

In the social sciences, though, which is where arts research generally lives, it’s much harder to be certain about your results. That’s not necessarily because it’s harder to get statistically significant results. It’s because it’s harder to design the models and experiments — in other words, your hypothesis as to what is happening and why, and the means you use to test it — with integrity.

In order for a model to work, it needs to account for everything that might affect the result that you’re looking for. For health sciences, this can be pretty simple: you give people a drug or administer them a treatment, and you measure whether they got better. You can collect other information about them, such as their race, age, gender, and so forth, in order to catch any differences along those axes, but otherwise it’s fairly straightforward. It’s also relatively routine (though sometimes complicated by ethical issues) to construct what’s known as a control group: a set of comparable individuals with the same problem who don’t receive the treatment.

In the arts, by contrast, both of these assumptions are challenged. First, when you’re looking at the impact of an arts program on things like, say, crime rates in a community, or educational outcomes for children, or hell, even just straight-up happiness for individual arts participants, it’s really difficult to isolate the unique contribution of the arts program from everything else that could be entering into the picture and affecting the results (e.g., the economy, quality parenting, or what they had to eat that day). And second, unlike in the case of giving somebody a pill, it’s hard to isolate the recipients or beneficiaries of arts programs from people who don’t benefit in a way that is scientifically rigorous – especially when the desired results concern whole communities or ecosystems.

*

It would be tempting, faced with that litany of challenges, to conclude that arts research simply isn’t worth the trouble. I would disagree with such a conclusion. Well-executed arts research, though rarely providing evidence beyond all doubt, can nevertheless help to illuminate some of the key assumptions that we make when we design arts programs. And boy, do some of those assumptionsneed illumination!

I view causality and arts research more generally through a frame of program evaluation and impact assessment. Fractured Atlas is currently undertaking an evaluation of one of its own programs and also helping an external client develop a framework to assess the impact of its grantmaking. One of the most important steps in that process is to identify the underlying assumptions upon which your program strategy rests. Every time we employ strategy to make decisions, no matter what the context, we are making assumptions. When we sign up for a test-prep course to improve our score on the GRE, for example, we assume that the instructor and materials are of a sufficient quality to help us learn, and also that we’re intellectually capable of achieving a better score with assistance. When we open a Twitter account to drive traffic to a website, we assume that we’ll have sufficient time and inspiration to generate content for the medium in a way that will gain traction over time. And when we start an organization whose goal is to bring about world peace through the arts, well, there are a LOT of assumptions that go into that one!

Where research can help us most is by telling us whether or not our assumptions are valid. We might feel more confident about our decision to sign up for the test prep class if we can first view data on how much improvement previous participants saw in their scores after taking it. Our organization’s decision to sign up for Twitter would be made easier if we had information on the trajectories of comparable peers’ tweet activity and followers over time along with measures of how much of a drain it was on staff resources. To me, research is not especially meaningful or worthwhile unless it has the potential to inform, either directly or indirectly, the decisions we make. But if it does, it can be very valuable indeed.

That’s because, unlike in health care or the pharmaceutical industry, in the arts we’re (usually) not dealing with life and death. It’s okay if we make a mistake once in a while; the world will continue on. So we don’t need to have 99.9% or even 95% certainty that the choices we make are the right ones before we move ahead. Indeed, as of now it’s likely that we make some decisions with virtually no certainty of their wisdom at all! To the extent that research can play a role in reducing the uncertainty we face in making decisions within a strategic framework, that research can provide real, quantifiable value to its users.

Let me elaborate on that last point. One of the most powerful tools I learned in business school was decision analysis, a conceptual approach useful for incorporating uncertainties into scenario planning. A common concept in decision analysis is what’s known as “the value of perfect information.” You know you have perfect information when there is absolutely no uncertainty in the outcomes that might result from an action or set of actions you take. The value of perfect information is the difference in your “expected value” (i.e., the result of the best possible strategy given the average of all possible outcomes, weighted by probability) with certainty and without. For example, if you’re only 60% sure that taking the test prep class will get you to the GRE score you need, there’s a 40% chance the amount you spend on the class will be a waste. In the language of decision analysis, that’s equivalent to saying that you can “expect” to lose 40% of your investment. With perfect information that taking the class will lead to the result you want, you have no risk of wasting that money. Thus, the value of perfect information in this case is 40% of the price of the class.

Research, especially research in the arts, can’t give us perfect information. But it can sure as hell give us better information than we already have. Even if it can reduce our uncertainty that our strategy is the right one from 40% to, say, 20%, that’s still quite a boost to our confidence. But the value of research is only as high as its quality. Badly designed or poorly executed studies can be next to useless in reducing uncertainty, or worse, can actually increase it by confusing the underlying issues. Unfortunately, no certification body currently exists to ensure the research conducted in the arts is of a sufficient quality to be helpful. The best way to make sure as a field that we don’t get taken in by low-quality work is to take some time to educate ourselves on good research practices. For a good, short primer, I recommend Evaluation Essentials by my own program evaluation teacher, Beth Osborne Daponte.

Next time, some thoughts on how Fractured Atlas puts these principles into practice.