Photo by ed_needs_a_bicycle

(This is the third post in a series on Fractured Atlas’s capacity-building pilot initiative, Fractured Atlas as a Learning Organization. To read more about it, please check out Fractured Atlas as a Learning Organization: An Introduction.)

I don’t know about you, but I’ve always been a reluctant decision maker. When I go out to eat at a restaurant, I often drive my dinner mates crazy by asking the wait staff for recommendations and then just ordering what I was thinking about getting anyway. In high school, I used to agonize over what now seem like trivial choices like what topic I should choose for my English papers. Perhaps that’s why I’m so drawn to the science of decision-making. I’m somebody who likes options, who likes variety. As soon as I make a decision, I close myself off to a world of possibilities. So I want to be sure I’m making the right one!

As much as anything else, making the right decision has to do with accurately forecasting what will happen once you make it. The trouble is, a lot of times we are pretty uncertain about just what those consequences might be. In the previous post in this series, I talked about how we frequently use too narrow a range of possibilities when trying to make a prediction about something we’re not sure about. In part that’s because our society – and perhaps our brains – have trained us to think deterministically rather than probabilistically. What that means in plain English is that when we make a guess, we typically hone in on one option. If you’re a contestant on Jeopardy! and the category is Tom Cruise movies, you can’t chime in with “What is either Magnolia or Top Gun, I’m not sure?” You gotta pick one. And that’s true of decision making as well, isn’t it? In the end, there’s only so much you can hedge your bets. We only get one life to live (well, that we know about) and we are accustomed to choosing a single path from the many in front of us.

The thing is, though, if we want to make smarter decisions, we need to train ourselves to be a little more comfortable with ambiguity. Sure, that course you took last fall seems like a slam-dunk right choice in retrospect, but did it look that way before you decided to sign up? What if it hadn’t worked out the way you’d hoped? The funny thing about decisions is that sometimes you can make the right call and still have things turn out badly – or vice versa. Improving the decision-making process is a long game – it banks on the idea that, over time, and across many decisions, you’re going to come out ahead for approaching each one thoughtfully. And thoughtful decision making involves defining the assumptions and potential consequences of your decision carefully and considering the full range of possibilities associated with each.

Forecasters have long been using a tool called Monte Carlo simulations to help us do just that. The Monte Carlo method was invented by a physicist named Stanislaw Ulam and was originally used to help design atomic bombs. (The codename “Monte Carlo” comes from the legendary casino in Monaco, where Ulam’s uncle used to gamble.) More recently, the method, which typically involves running thousands (or sometimes tens or hundreds of thousands) of simulations on a model and then aggregating the results, has shown up in everything from computational biology to corporate finance. Data scientist and former New York Times blogger Nate Silver, who famously predicted the 2008 and 2012 presidential election results with near-perfect accuracy, relied on a sophisticated Monte Carlo model to do so.

Using Monte Carlo methods within a decision-making context enables us to make predictions about the consequences of those decisions, and thus better understand the opportunities and risks associated with them. At Fractured Atlas, where we’ve set up an internal team to experiment with this sort of decision modeling, we’ve begun to hone the process for bringing quantitative definition to our decision dilemmas and making predictive estimates. Below, I’ll lay out the steps we currently use to do this.

1. Define what your dilemma actually is. For many people, this exercise is fairly intuitive, but sometimes it takes a little bit of work to get your decision to a state where it can actually be modeled. Something like “what should I do with my life?” is not really at that place yet. But let’s say you’re deciding whether or not to move your office to a bigger space. That’s a very clear, specific dilemma that a process like this one can help resolve (with a bit of elbow grease).
2. Articulate what excites you, and conversely, what worries you about this decision. If we had perfect information all the time about what was going to happen in the future, decision making would never be a source of stress. In fact, it would be very boring! What causes us to worry about making the wrong decision is that hard decisions always carry some sort of risk, opportunity, or combination of the two. Translating your emotions about the decision into words will help you understand the criteria by which the decision will be judged successful or not, and what factors could affect the outcome.
3. Define the “central question” for the decision and identify the relevant variables affecting it. This central question needs to be understood in quantitative terms and usually boils down to some sort of cost-benefit equation: for example, will the amount of money I earn from the points on this credit card exceed the annual fee? Alternatively, the central question might weigh two options against each other: does a change in the business model for my program yield better results than the status quo? Once you know what your central question is and have identified the elements of the decision that you care about most, the next step is to define variables – things like the number of people attending your event, the expected cost of a new software feature, and the like. The variables should collectively work together to create an equation that ultimately answers the central question.
4. Estimate values and distributions for the variables. For each of the variables in the model, we need to give our model some sense of what the numbers might be. For example, you might be 90% confident that the proportion of folks who will buy a book after they hear you speak is somewhere between one-tenth and one-third. This is where the calibration training mentioned in the previous post comes in handy. If we’ve learned anything about estimating ranges, it’s that we need to make sure to make them wide enough!
5. Run the Monte Carlo simulation. We set up Excel to run 10,000 individual scenarios, each one a plausible real-life outcome according to the assumptions we’ve set up in our model. The model aggregates the results of each of those to tell us, out of those 10,000, what percentage ended up answering the central question a certain way. And now we have a prediction of how our decision is going to turn out!

I imagine this must all seem kind of abstract, so let’s run through the process again with a concrete example. Let’s say you run a rental space and you’re deciding whether or not to extend your hours later into the evening. You’re excited because a lot of customers have been requesting to rehearse at those hours, and there’s a reasonable expectation that the change is going to bring in more revenue. On the other hand, you’re worried that the fixed costs of staffing the space during the late-night hours, whether it’s being used or not, will outpace any incremental dollars you might bring in. The central question is one of simple costs and benefits: is extending the hours going to be worth it in the end?

Defining the incremental costs of the change should be fairly simple. Most of it will have to do with labor. If you pay your people hourly, it’s as simple as the additional hours you’ll have to pay out due to the extended schedule. You may want to account for the possibility that you’ll need to hire an extra person to handle the load; in that case the drain on your time to find that person should be considered as an upfront cost. If you want to get a little fancier, there may also be increased utility costs (electricity, etc.) that you’ll incur as a result of the increased usage of the space, as well as increased wear-and-tear that could be expressed as faster depreciation.

Defining the benefit side of the equation happens much the same way. Do you have only one rental rate for all your customers? Then it’s just a matter of estimating how many new customer-hours will be enabled by the schedule extension and calculating how much they will pay for those hours. Things get a little more interesting if you have lots of different rates for different customers and uses – e.g., rehearsal vs. recording, non-profit vs. for-profit, etc. In that case, you’ll want to segment those customers out by their different properties, and calculate how much new revenue is generated by each type.

(click to enlarge)

Ultimately, we end up with a set of estimates for the potential costs and benefits of following through with your decision. The Monte Carlo simulation can then calculate, out of those tens of thousands of scenarios, how often the decision turned out to be the right one. Basically, it’s offering you a recommendation on which path to choose – kind of like the waiter who confidently tells you to get the saltimbocca. Just don’t be like me and order the chicken anyway!

If you’d like to see the formulas and play with the numbers yourself in the example above, you can download a simple version of it here.

Measuring the universe (Roman Ondak) — photo by flickr user Alexandre Dulaunoy

(This is the second post in a series on Fractured Atlas’s capacity-building pilot initiative, Fractured Atlas as a Learning Organization. To read more about it, please check out Fractured Atlas as a Learning Organization: An Introduction.)

Last fall, we put together a group of six people (henceforth referred to as the Data-Driven D.O.G. Force) to collectively develop a set of decision-making frameworks to help us resolve so-called decisions of consequence – situations for which the level of uncertainty and the cost of being wrong are both high. To do this, we’ve been taking inspiration from Doug Hubbard’s book How to Measure Anything, which introduces a concept he invented called “applied information economics,” or AIE. AIE is a formalized method of building a quantitative model around a decision and analyzing how information can play a role in making that decision. You can read much more about it in Luke Muehlhauser’s excellent summary of How to Measure Anything for Less Wrong.

One of the central tenets of AIE is that we can only judge the value of a measurement in relation to how much it reduces our uncertainty about something that matters. (More on that in a future post!) In order to know that, though, we have to have some sense of how much uncertainty we have now.

This concept of uncertainty is one that we understand on an intuitive level – I might be much more confident, say, predicting that I’ll be hungry at dinner-time tonight than predicting what I’ll be doing with my life 10 years from now. But most people don’t have a lot of experience quantifying their uncertainty. And yet, as forecasting experts from Hubbard to Nate Silver tell us, the secret to successful predictions (or at least less terrible predictions) is thinking probabilistically.

What does this mean in practice? Picture yourself at Tuesday trivia night at your favorite local pub. There you are with your teammates, you’ve come up with some ridiculous name for yourselves (like, I don’t know, the “Data-Driven D.O.G. Force”), and the round is about to begin. The emcee calls out the question: “the actor Tom Cruise had his breakout role in what 1983 movie?” Your friend leans over and says, “It’s Risky Business. I’m like 99% sure.”

Anyone who’s done time at trivia night will probably recognize something like that sequence. What I can virtually guarantee you, though, is that your friend in this situation hasn’t thought very hard about that 99% figure. Is it really 99%? That’s awfully confident – it implies that if your friend were to answer 100 questions and was as confident about every one of the answers as she was about this one, she would be right 99 times.

I’d be willing to bet that if you recorded the number of times people said they were “99% sure” about something and kept track of how often they were actually right, it would be significantly less than 99% of the time. That’s because as human beings, we tend to be overconfident in our knowledge in all sorts of ways, and this exact effect has been documented by psychologists and behavioral economists in experiment after experiment for decades.

This is why any AIE process involves something called calibration training. Overconfidence is an endemic and hard-to-escape problem, but if you practice making predictions and confront yourself with feedback about the results of those predictions, you can get better. In How to Measure Anything, Hubbard provides a number of calibration tests essentially consisting of trivia questions like the one above – except that instead of naming a specific movie or person, we’re asked to provide a ranged estimate (for numbers) or a confidence rating in the truth or falsehood of a statement. So for example, you might find yourself guessing what year Risky Business came out, or whether it’s true or false that it was Tom Cruise’s first leading role.

The six of us on the D.O.G. Force took a number of these calibration tests, and I’m gonna be honest with you – we were pretty awful. We got the hang of the binary (true/false) predictions relatively quickly, but the ranged estimates proved exceedingly difficult for us. In four iterations of the latter test across six individuals, only one of us ever managed to be right more often than we said we would be. You can see this in the results below (red colors and negative numbers mean that we were overconfident, green colors and positive numbers underconfident, and yellow/zero right on the money).

We were able to make good progress in the last round of the test (“Range supplemental 2”), though, primarily by focusing on making our ranges wide enough when we really had no idea what the right answer was. What’s the maximum range of a Minuteman missile? Well, if you don’t even know what a Minuteman missile is, your range should be wide enough to cover everything from a kid’s toy to an ICBM. It can feel incredibly unsatisfying to admit that the range of possibilities is so wide, but in order to construct an accurate model of the state of your knowledge, right now, you need to be able to articulate what “I have no idea” really means.

So why spend valuable company time working through a bunch of trivia questions? Because when we find ourselves needing to make estimates about, say, how much a new software feature might cost, or the number of people who might be reached when we speak at a conference, we suffer from the same disease of overconfidence if we don’t do something about it. What happens as a result is that we make predictions that are reassuringly precise in the moment, but might well end up far off from reality down the road. And when we use those inaccurate assumptions and predictions in our decision-making, there’s a good chance – so to speak – that we’re setting ourselves up for later regret.

Next up: how this all fits in with grand strategy!

[UPDATE: If you want to try a range test for yourself, Fractured Atlas’s rockstar Community Engagement Specialist and D.O.G. Force member Jason Tseng has created an arts-specific one! Here are the questions and here are the answers (don’t peek!).]

If you’ve been paying any attention at all to technology trends the past few years, you know that we live in the era of Big Data. All of those videos we upload to YouTube, hard drives we fill with government secrets (or cat photos, take your pick), and tweets we awkwardly punch out on touchscreen keyboards add up to a whole lot of gigabytes, the bulk of which are stored by someone, somewhere, indefinitely. By some estimates, human beings generate more data every two days than we did in the entire history of civilization prior to 2003 – and that was as of three years ago!

Indeed, these are exciting times for data nerds, and data nerds in the arts are no exception. Initiatives such as the Cultural Data Project, Southern Methodist University’s National Center for Arts Research, and the Americans for the Arts National Arts Index seek to collect or organize relevant indicators pertaining to everything from arts organizations’ financial health to audience reach and characteristics to long-term trends for musical instrument purchases.

Fractured Atlas is no stranger to data initiatives in the arts. Our Archipelago data visualization software is one of the largest such efforts, bringing together information on arts nonprofits, for-profits, fiscally sponsored projects, funding, audience distributions, and community context all in one place in the service of better understanding the arts ecosystem in a region. Facilitating data-driven decisions is a major long-term objective of Artful.ly, our just-launched cloud-based arts management tool, and a present-day reality for Spaces, our venue listing and booking service that can promote spaces with last-minute availability to users. Through our research advisory services work, we’ve helped funders such as ArtsWave organize their entire grantmaking process around principles of data-driven decision-making in order to further their philanthropic objectives. Everyone benefits when funders, organizations and individuals in the arts ecosystem make thoughtful decisions about resource allocation, setting up and responding to incentives, and more. At Fractured Atlas, we believe that data can and should be a crucial input into that thoughtful decision-making process, and we’ve been increasingly vocal in evangelizing for data-driven decision making throughout the arts and cultural sector.

There’s just one problem. Up until now, Fractured Atlas has not had any formal guidelines in place to ensure that we use data in our own decision making, with the result that our internal decisions – relating to management, marketing, strategy, and the like – have been guided primarily by managerial intuition. In a “doctor, heal thyself!” moment, we’ve agreed that is time for our practices to reflect our preaching, at both the program and institutional levels. In 2013, the scope of our operations, the size of the community we serve, and the financial stakes in our work demand informed analysis at a level of rigor that we have not historically practiced. (This directive was immortalized by our fearless leader Adam Huttler in the organization’s annual Strategic Priorities Memo with the colorful title, “Eating Our Data-Driven Dog Food.”)

So between now and next summer, Fractured Atlas is embarking on a pilot initiative to explore how we can use data and evidence to improve our decision-making process at all levels. We’re calling it Fractured Atlas as a Learning Organization, and through this and future blog posts, we’re giving you the opportunity to be a fly on the wall as use this process as a way of grappling with issues of organization identity, strategy, culture, and impact.

What Is a Learning Organization?

As I define it*, a learning organization is one for which information and strategy are joined at the hip. It is, quite literally, an organization that has successfully forged a culture of learning and integrated that culture into its decision-making process at all levels.

Why is this integration between information and strategy important? Because every organization operates in an environment of uncertainty about what is going to result from its decisions, and every decision we make on behalf of an organization is based on a prediction, whether explicitly articulated or not, about the results of that decision.

If you can reduce the uncertainty associated with your decisions, the chances that you will make the right decision will increase. Of particular interest here  are what I call decisions of consequence: dilemmas for which the consequences of making the wrong decision and uncertainty about the nature of the right decision are both high. So, how do you reduce that uncertainty? Why, through research, of course! Studying what has happened in the past can inform what is likely to happen in the future. Studying what has happened in other contexts can inform what is likely to happen in your context. And studying what is happening now can tell you whether your assumptions seem spot on or off by a mile.

In fact, I subscribe to the notion that research is only valuable insofar as it helps to answer a question that matters. I’m not the only one who thinks so, either: Jake Porway, the founder of a nonprofit that connects data scientists with social enterprises in need, wrote this past spring that “any data scientist worth their salary will tell you that you should start [a data project] with a question, NOT the data.” In fact, all of the excitement around Big Data notwithstanding, data divorced from strategy is not likely to be very useful.

A learning organization solves this problem by forging a powerful feedback loop between information and strategy, with each feeding the other and adapting in relation to the other. The more obvious implication of this symbiosis is that organizational decisions must adapt in response to new information, as discussed above. But the less obvious implication is no less important: information-gathering must be directed by the organization’s decision-making needs. Without that intimate connection, there are no real safeguards to prevent organizations from thinking they are making data-driven decisions without really putting much thought into either the data or the decisions.

More broadly, a learning organization develops a culture of seeking out and using information thoughtfully from the highest levels to the organization’s grassroots. The most effective organizations are conscious about the impact they are trying to achieve, and willing to be open-minded regarding the paths they take to maximizing that impact.

*Some readers may be familiar with the term “learning organization” as defined by MIT management scientist Peter Senge in his well-known 1990 book The Fifth Discipline. My use of the phrase is broadly in the same spirit as Senge’s, but he sets out a very specific formula for what constitutes a learning organization that I don’t make use of here.

Fractured Atlas as a Learning Organization

This fiscal year, which started in September and goes through next summer, we are undertaking a pilot project to put some of these principles into practice. The primary goal of the pilot is to develop a conceptual framework and a toolkit of situation-adaptable methods for reducing uncertainty about decisions of consequence. If we can reduce the uncertainty we have about those decisions through strategic measurement and information-gathering efforts, over time we’re likely to make better decisions that will in turn lead to better outcomes for Fractured Atlas and the people who benefit from our work.

As powerful as this idea is, it only works if we have a very concrete sense of what we’re trying to accomplish as an organization. While we’ve had a mission statement for some time now, the huge variety of programs and services Fractured Atlas offers is virtually impossible to fully capture in a single sentence. Accordingly, the first step in this process is to create a theory of change for every program at the organization, from which we’ll roll up an overall theory of change and logic model for the organization as a whole. This will allow us to define our overall goals as well as some key success metrics at various levels of operation, taking into account both Fractured Atlas’s mission objectives and its focus on developing programs that are sustainable with earned income.

Meanwhile, we’ve formed an internal task force to work on this project at a deeper level of engagement throughout the year. Affectionately called the Data-Driven D.O.G. Force (the “D.O.G.” stands for Data Over Gut), the group will meet every 6-8 weeks to receive calibrated probability assessment training, identify real-world decisions of consequence to use as case studies, and come up with measurement experiments to gather information relevant to those decisions. In doing so, we’ll be using a modified version of a methodology called Applied Information Economics as described in Douglas W. Hubbard’s book How to Measure Anything: Finding the Value of “Intangibles” in Business. One major advantage of AIE is that it explicitly takes into account the cost-benefit of measurement strategies by calculating something called the value of information, which we’ll be exploring further in a future post.

At the end, we’ll attempt to formalize a process for identifying decisions of consequence in the future and fitting measurement strategies to the situation at hand. We’ll also present some recommendations for building infrastructure in the form of ongoing data collection, to address those questions that are likely to be asked again and again. And through it all, I’ll be writing about it here – so that anyone who wants to can learn alongside us.

Learning in Context: Why Philosophy Matters as Much as Performance

Data-driven decision-making isn’t just about crunching numbers. It’s a practice that requires certain values in order to work. The hard part of being data-driven is not the “data” but the “driven” – you have to be willing to question your assumptions and actually change your behavior in response to the new information coming in. Put another way, a learning organization is, well, open to learning new things -even things that suggest that the way that we’re currently doing things isn’t working as well as it could, or that we’re missing important opportunities to increase our impact.

It’s much easier to attain that kind of open stance if we train ourselves to expect failure upfront. In general, organizations as well as people have a tendency to be far too risk-averse. Being a learning organization means embracing a culture of intentional experimentation and productive failure: we’re likely not going to hit upon the secret sauce the very first time we try something – or, sometimes, at all.

Being a learning organization similarly requires that we think about ourselves from a system perspective – how are we making a difference in light of what everyone else is doing? And how can our experiences shed light on those of others? That’s why we’re not just going down this path on our own and in private. If the specific activities of the pilot project turn out to be a big waste of time (and I can’t guarantee that they won’t), we won’t be able to hide that from you or the world. But even that would ultimately be a good thing – because, in true learning organization fashion, it would cause us to reconsider the limitations of a data-driven approach. Embracing change is hard, but one of the very best things about it is that it can allow us to extract just as much (if not more) value from failure as success.

For me, personally, this project is very exciting. Of course I’m eager to find out what we’ll learn. But more than that, Fractured Atlas as a Learning Organization is an opportunity for us to exercise leadership in a way that reaffirms our highest standards for ourselves and for the field. I’m looking forward to sharing our journey with you.