If you’ve been paying any attention at all to technology trends the past few years, you know that we live in the era of Big Data. All of those videos we upload to YouTube, hard drives we fill with government secrets (or cat photos, take your pick), and tweets we awkwardly punch out on touchscreen keyboards add up to a whole lot of gigabytes, the bulk of which are stored by someone, somewhere, indefinitely. By some estimates, human beings generate more data every two days than we did in the entire history of civilization prior to 2003 – and that was as of three years ago!

Indeed, these are exciting times for data nerds, and data nerds in the arts are no exception. Initiatives such as the Cultural Data Project, Southern Methodist University’s National Center for Arts Research, and the Americans for the Arts National Arts Index seek to collect or organize relevant indicators pertaining to everything from arts organizations’ financial health to audience reach and characteristics to long-term trends for musical instrument purchases.

Fractured Atlas is no stranger to data initiatives in the arts. Our Archipelago data visualization software is one of the largest such efforts, bringing together information on arts nonprofits, for-profits, fiscally sponsored projects, funding, audience distributions, and community context all in one place in the service of better understanding the arts ecosystem in a region. Facilitating data-driven decisions is a major long-term objective of Artful.ly, our just-launched cloud-based arts management tool, and a present-day reality for Spaces, our venue listing and booking service that can promote spaces with last-minute availability to users. Through our research advisory services work, we’ve helped funders such as ArtsWave organize their entire grantmaking process around principles of data-driven decision-making in order to further their philanthropic objectives. Everyone benefits when funders, organizations and individuals in the arts ecosystem make thoughtful decisions about resource allocation, setting up and responding to incentives, and more. At Fractured Atlas, we believe that data can and should be a crucial input into that thoughtful decision-making process, and we’ve been increasingly vocal in evangelizing for data-driven decision making throughout the arts and cultural sector.

There’s just one problem. Up until now, Fractured Atlas has not had any formal guidelines in place to ensure that we use data in our own decision making, with the result that our internal decisions – relating to management, marketing, strategy, and the like – have been guided primarily by managerial intuition. In a “doctor, heal thyself!” moment, we’ve agreed that is time for our practices to reflect our preaching, at both the program and institutional levels. In 2013, the scope of our operations, the size of the community we serve, and the financial stakes in our work demand informed analysis at a level of rigor that we have not historically practiced. (This directive was immortalized by our fearless leader Adam Huttler in the organization’s annual Strategic Priorities Memo with the colorful title, “Eating Our Data-Driven Dog Food.”)

So between now and next summer, Fractured Atlas is embarking on a pilot initiative to explore how we can use data and evidence to improve our decision-making process at all levels. We’re calling it Fractured Atlas as a Learning Organization, and through this and future blog posts, we’re giving you the opportunity to be a fly on the wall as use this process as a way of grappling with issues of organization identity, strategy, culture, and impact.

What Is a Learning Organization?

As I define it*, a learning organization is one for which information and strategy are joined at the hip. It is, quite literally, an organization that has successfully forged a culture of learning and integrated that culture into its decision-making process at all levels.

Why is this integration between information and strategy important? Because every organization operates in an environment of uncertainty about what is going to result from its decisions, and every decision we make on behalf of an organization is based on a prediction, whether explicitly articulated or not, about the results of that decision.

If you can reduce the uncertainty associated with your decisions, the chances that you will make the right decision will increase. Of particular interest here  are what I call decisions of consequence: dilemmas for which the consequences of making the wrong decision and uncertainty about the nature of the right decision are both high. So, how do you reduce that uncertainty? Why, through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile.

In fact, I subscribe to the notion that research is only valuable insofar as it helps to answer a question that matters. I’m not the only one who thinks so, either: Jake Porway, the founder of a nonprofit that connects data scientists with social enterprises in need, wrote this past spring that “any data scientist worth their salary will tell you that you should start [a data project] with a question, NOT the data.” In fact, all of the excitement around Big Data notwithstanding, data divorced from strategy is not likely to be very useful.

A learning organization solves this problem by forging a powerful feedback loop between information and strategy, with each feeding the other and adapting in relation to the other. The more obvious implication of this symbiosis is that organizational decisions must adapt in response to new information, as discussed above. But the less obvious implication is no less important: information-gathering must be directed by the organization’s decision-making needs. Without that intimate connection, there are no real safeguards to prevent organizations from thinking they are making data-driven decisions without really putting much thought into either the data or the decisions.

More broadly, a learning organization develops a culture of seeking out and using information thoughtfully from the highest levels to the organization’s grassroots. The most effective organizations are conscious about the impact they are trying to achieve, and willing to be open-minded regarding the paths they take to maximizing that impact.

*Some readers may be familiar with the term “learning organization” as defined by MIT management scientist Peter Senge in his well-known 1990 book The Fifth Discipline. My use of the phrase is broadly in the same spirit as Senge’s, but he sets out a very specific formula for what constitutes a learning organization that I don’t make use of here.

Fractured Atlas as a Learning Organization

This fiscal year, which started in September and goes through next summer, we are undertaking a pilot project to put some of these principles into practice. The primary goal of the pilot is to develop a conceptual framework and a toolkit of situation-adaptable methods for reducing uncertainty about decisions of consequence. If we can reduce the uncertainty we have about those decisions through strategic measurement and information-gathering efforts, over time we’re likely to make better decisions that will in turn lead to better outcomes for Fractured Atlas and the people who benefit from our work.

As powerful as this idea is, it only works if we have a very concrete sense of what we’re trying to accomplish as an organization. While we’ve had a mission statement for some time now, the huge variety of programs and services Fractured Atlas offers is virtually impossible to fully capture in a single sentence. Accordingly, the first step in this process is to create a theory of change for every program at the organization, from which we’ll roll up an overall theory of change and logic model for the organization as a whole. This will allow us to define our overall goals as well as some key success metrics at various levels of operation, taking into account both Fractured Atlas’s mission objectives and its focus on developing programs that are sustainable with earned income.

Meanwhile, we’ve formed an internal task force to work on this project at a deeper level of engagement throughout the year. Affectionately called the Data-Driven D.O.G. Force (the “D.O.G.” stands for Data Over Gut), the group will meet every 6-8 weeks to receive calibrated probability assessment training, identify real-world decisions of consequence to use as case studies, and come up with measurement experiments to gather information relevant to those decisions. In doing so, we’ll be using a modified version of a methodology called Applied Information Economics as described in Douglas W. Hubbard’s book How to Measure Anything: Finding the Value of “Intangibles” in Business. One major advantage of AIE is that it explicitly takes into account the cost-benefit of measurement strategies by calculating something called the value of information, which we’ll be exploring further in a future post.

At the end, we’ll attempt to formalize a process for identifying decisions of consequence in the future and fitting measurement strategies to the situation at hand. We’ll also present some recommendations for building infrastructure in the form of ongoing data collection, to address those questions that are likely to be asked again and again. And through it all, I’ll be writing about it here – so that anyone who wants to can learn alongside us.

Learning in Context: Why Philosophy Matters as Much as Performance

Data-driven decision-making isn’t just about crunching numbers. It’s a practice that requires certain values in order to work. The hard part of being data-driven is not the “data” but the “driven” – you have to be willing to question your assumptions and actually change your behavior in response to the new information coming in. Put another way, a learning organization is, well, open to learning new things -even things that suggest that the way that we’re currently doing things isn’t working as well as it could, or that we’re missing important opportunities to increase our impact.

It’s much easier to attain that kind of open stance if we train ourselves to expect failure upfront. In general, organizations as well as people have a tendency to be far too risk-averse. Being a learning organization means embracing a culture of intentional experimentation and productive failure: we’re likely not going to hit upon the secret sauce the very first time we try something – or, sometimes, at all.

Being a learning organization similarly requires that we think about ourselves from a system perspective – how are we making a difference in light of what everyone else is doing? And how can our experiences shed light on those of others? That’s why we’re not just going down this path on our own and in private. If the specific activities of the pilot project turn out to be a big waste of time (and I can’t guarantee that they won’t), we won’t be able to hide that from you or the world. But even that would ultimately be a good thing – because, in true learning organization fashion, it would cause us to reconsider the limitations of a data-driven approach. Embracing change is hard, but one of the very best things about it is that it can allow us to extract just as much (if not more) value from failure as success.

For me, personally, this project is very exciting. Of course I’m eager to find out what we’ll learn. But more than that, Fractured Atlas as a Learning Organization is an opportunity for us to exercise leadership in a way that reaffirms our highest standards for ourselves and for the field. I’m looking forward to sharing our journey with you.

I was a participant in a couple of conversations with fellow arts research nerds recently in which we discussed the notion of cause and effect. You remember that one from grade school, right? Well, it turns out that when it comes to research (and especially arts research), it’s not as simple as we all thought.

You see, in science, when we say that something caused something else, we tend to want to be sure. A common concept in statistics is “significance“: the idea that a meaningful connection exists between two variables that can’t be explained by random noise. If you’ve formed a hypothesis and designed your experiments correctly, and you get statistically significant results, you can be fairly confident that the results you’re looking at indicate something real and not merely an accident or coincidence.

Of course, it’s never possible to be entirely sure. But in some fields, you can get pretty close. The technical term for the degree of uncertainty we’re willing to tolerate in an analysis like this is the alpha: an alpha of 0.05 means that we’d like to be 95% sure that we can reject the null hypothesis (i.e., that we might just be looking at random noise) before we go ahead and report the result as meaningful. In some fields, it’s common to require an alpha of 0.001 or even lower – that is, a 99.9% certainty that random variation isn’t behind the results. And for something like testing a new drug, you most definitely want to be that sure that, for example, it doesn’t cause heart attacks as a side effect before you put that sucker on the market.

In the social sciences, though, which is where arts research generally lives, it’s much harder to be certain about your results. That’s not necessarily because it’s harder to get statistically significant results. It’s because it’s harder to design the models and experiments — in other words, your hypothesis as to what is happening and why, and the means you use to test it — with integrity.

In order for a model to work, it needs to account for everything that might affect the result that you’re looking for. For health sciences, this can be pretty simple: you give people a drug or administer them a treatment, and you measure whether they got better. You can collect other information about them, such as their race, age, gender, and so forth, in order to catch any differences along those axes, but otherwise it’s fairly straightforward. It’s also relatively routine (though sometimes complicated by ethical issues) to construct what’s known as a control group: a set of comparable individuals with the same problem who don’t receive the treatment.

In the arts, by contrast, both of these assumptions are challenged. First, when you’re looking at the impact of an arts program on things like, say, crime rates in a community, or educational outcomes for children, or hell, even just straight-up happiness for individual arts participants, it’s really difficult to isolate the unique contribution of the arts program from everything else that could be entering into the picture and affecting the results (e.g., the economy, quality parenting, or what they had to eat that day). And second, unlike in the case of giving somebody a pill, it’s hard to isolate the recipients or beneficiaries of arts programs from people who don’t benefit in a way that is scientifically rigorous – especially when the desired results concern whole communities or ecosystems.

*

It would be tempting, faced with that litany of challenges, to conclude that arts research simply isn’t worth the trouble. I would disagree with such a conclusion. Well-executed arts research, though rarely providing evidence beyond all doubt, can nevertheless help to illuminate some of the key assumptions that we make when we design arts programs. And boy, do some of those assumptionsneed illumination!

I view causality and arts research more generally through a frame of program evaluation and impact assessment. Fractured Atlas is currently undertaking an evaluation of one of its own programs and also helping an external client develop a framework to assess the impact of its grantmaking. One of the most important steps in that process is to identify the underlying assumptions upon which your program strategy rests. Every time we employ strategy to make decisions, no matter what the context, we are making assumptions. When we sign up for a test-prep course to improve our score on the GRE, for example, we assume that the instructor and materials are of a sufficient quality to help us learn, and also that we’re intellectually capable of achieving a better score with assistance. When we open a Twitter account to drive traffic to a website, we assume that we’ll have sufficient time and inspiration to generate content for the medium in a way that will gain traction over time. And when we start an organization whose goal is to bring about world peace through the arts, well, there are a LOT of assumptions that go into that one!

Where research can help us most is by telling us whether or not our assumptions are valid. We might feel more confident about our decision to sign up for the test prep class if we can first view data on how much improvement previous participants saw in their scores after taking it. Our organization’s decision to sign up for Twitter would be made easier if we had information on the trajectories of comparable peers’ tweet activity and followers over time along with measures of how much of a drain it was on staff resources. To me, research is not especially meaningful or worthwhile unless it has the potential to inform, either directly or indirectly, the decisions we make. But if it does, it can be very valuable indeed.

That’s because, unlike in health care or the pharmaceutical industry, in the arts we’re (usually) not dealing with life and death. It’s okay if we make a mistake once in a while; the world will continue on. So we don’t need to have 99.9% or even 95% certainty that the choices we make are the right ones before we move ahead. Indeed, as of now it’s likely that we make some decisions with virtually no certainty of their wisdom at all! To the extent that research can play a role in reducing the uncertainty we face in making decisions within a strategic framework, that research can provide real, quantifiable value to its users.

Let me elaborate on that last point. One of the most powerful tools I learned in business school was decision analysis, a conceptual approach useful for incorporating uncertainties into scenario planning. A common concept in decision analysis is what’s known as “the value of perfect information.” You know you have perfect information when there is absolutely no uncertainty in the outcomes that might result from an action or set of actions you take. The value of perfect information is the difference in your “expected value” (i.e., the result of the best possible strategy given the average of all possible outcomes, weighted by probability) with certainty and without. For example, if you’re only 60% sure that taking the test prep class will get you to the GRE score you need, there’s a 40% chance the amount you spend on the class will be a waste. In the language of decision analysis, that’s equivalent to saying that you can “expect” to lose 40% of your investment. With perfect information that taking the class will lead to the result you want, you have no risk of wasting that money. Thus, the value of perfect information in this case is 40% of the price of the class.

Research, especially research in the arts, can’t give us perfect information. But it can sure as hell give us better information than we already have. Even if it can reduce our uncertainty that our strategy is the right one from 40% to, say, 20%, that’s still quite a boost to our confidence. But the value of research is only as high as its quality. Badly designed or poorly executed studies can be next to useless in reducing uncertainty, or worse, can actually increase it by confusing the underlying issues. Unfortunately, no certification body currently exists to ensure the research conducted in the arts is of a sufficient quality to be helpful. The best way to make sure as a field that we don’t get taken in by low-quality work is to take some time to educate ourselves on good research practices. For a good, short primer, I recommend Evaluation Essentials by my own program evaluation teacher, Beth Osborne Daponte.

Next time, some thoughts on how Fractured Atlas puts these principles into practice.

If nothing else, I knew I would come out of the experience with two things: (a) a piece of paper in my hand that said I was a smart person and (b) a whole lotta debt. Both of these things turned out to be true. But that was about the extent to which my premonitions and preconceptions held water. Time and again, something about the experience would surprise me, and I remain now in awe of not only how much my feelings about b-school evolved, but how much I evolved. In no particular order, then, I offer the most valuable concepts, ideas, skills, and learnings I take away from my four semesters in New Haven, CT:

• As math subjects go, an understanding of statistics must rank close behind basic algebra in highest usefulness-in-daily-life-to-mental-effort-required ratio. It’s amazing to read websites like fivethirtyeight.com now and actually understand how those graphs were produced. And if that’s not practical enough for you, decision analysis can shed light onto even the most intransigent dilemmas by identifying dominated (i.e., foolish) strategies and helping to organize one’s thinking. A tool that I will use for the rest of my life.
• Program evaluation is not rocket science. All it requires you to do is to apply logic relentlessly, a skill that does take some practice. A class on ethics and human behavior taught us that the only reliable way to counteract persistent biases, such as overconfidence, is to ask oneself at critical junctures: “how could I be wrong?” This is what program evaluators, good ones anyway, do at every step of the process. It’s so simple, but it can save you a world of remorse down the line.
• The teaching of introductory economics needs serious reform. I have written about this at length, but having taken additional coursework in the subject and witnessed a global financial meltdown since my original rants, I now feel quite confident in saying that externalities and behavioral analysis should occupy a much more prominent role in the discussion, and that normative judgments about policy should be taken out. Free markets with perfect information, perfect competition, and rational actors are the exception, not the rule.
• Most of us are far too risk-averse. As a simple example: have you ever been really attracted to someone but decided not to ask them out because you were afraid of getting rejected? It’s silly, right? What, exactly, are the consequences of getting rejected? How is it worse than not asking in the first place? Take this thought and apply it to your job, a fundraising ask, an application to school — you name it. Make sure the risks you’re avoiding are actually risks, and not just fears in disguise.
• When it comes to negotiation, it’s amazing what difference a little planning makes. Knowing exactly what your preferences are and thinking about what concessions you’re prepared to make–and not–beforehand will equip you with the tools you need to guide the conversation toward an optimal conclusion. This little lesson can be applied to almost any kind of negotiation, not just the usual suspects like big business deals or bargaining at the flea market.
• Presentation matters. It really does. Oh sure, it doesn’t matter as much as content, at least in the long run. The thing is, presentation is a part of your content. That’s why it matters.

As much as I’ve learned from business school, perhaps the most profound lesson I take away is how much I still don’t know. In 2006, I knew a hell of a lot about choral music, the history of experimental rock ensembles, who’s who in the American classical composer social hierarchy, and where to find hassle-free street parking in New York City. At the time, that seemed like a pretty decent chunk of stuff to know a hell of a lot about. I now know that it’s about 0.00000000001% of the full extent of human knowledge and achievement. After two years in business school, I can now claim a passing conversance with, oh, maybe 0.00000000003%. Which, you know, is triple what I knew before — and still BARELY ANYTHING.

On the other hand, I now have tools to help me make decisions and manage in situations even when I don’t have all the information. I know how to figure out what questions I must ask in order to get to the answers I need. I know how to surround myself with people who know more than I do about particular subjects so that I don’t have to keep reinventing the wheel. And most importantly, I can feel secure in the knowledge that, for as long as I live, I will never stop learning new things. And that is a great gift.