Author(s): Sarah Lee and Peter Linett

Publisher: Cultural Data Project

Year: 2013

URL: http://www.culturaldata.org/wp-content/uploads/new-data-directions-for-the-cultural-landscape-a-report-by-slover-linett-audience-research-for-the-cultural-data-project_final.pdf

Topics: research, data

Methods: Theory/assertion, informed by synthesis of comments from a CDP-hosted online forum of researchers (disclosure: I was one of them), results from CDP’s internal strategic planning survey, and a paper by Margaret Wyszomirksi (not available online) “to frame and inventory the cultural data landscape.”

What it says: “New Directions” was commissioned by the Cultural Data Project in connection with that organization’s transition from a foundation-housed initiative to an independent nonprofit, and the strategic planning process that followed. The report was intended to inform that process by situating CDP’s efforts within the larger context of data collection throughout the United States cultural sector. It notes a growing abundance of and interest in arts and cultural data, but identifies six factors that “may be limiting the sector from effectively incorporating data into decision-making processes.” The six factors [paraphrased] are:

• Poor accessibility, quality, and comparability of cultural data (stemming from a decentralized infrastructure)
• Norms about data collection and use, including low priority/importance assigned to the task of data collection in general
• Lack of coordination and standardization among existing data collection efforts
• Skill and resource capacity constraints among cultural nonprofits
• Organizational culture dynamics that inhibit thoughtful decision-making
• A paucity of vision and case studies regarding the successful use of data to drive decisions

To address these challenges, “New Directions” recommends coordinating leadership on cultural data, engaging program and artistic staff in conversations about data, shifting the frame from accountability to decision-making, developing a research and data collection agenda, developing data-related skills among organization staff, and improving the cultural data infrastructure.

What I think about it: While the work relies heavily on the impressions of a small number of experts, Lee and Linett make a number of good and important points. I particularly agree with the notion that what the world needs is not more data collection, but rather better skills and filters to apply to the data and research that’s already there. “New Directions” also calls out the “data first, research questions later” approach adopted by so many cultural institutions (including, arguably, the CDP itself) as an unhelpful norm, and makes veiled but unmistakable reference to the widespread practice of letting advocacy goals take priority over strong methodological standards. That said, the paper’s persistent focus on data rather than the broader concept of research ends up causing it to miss the forest for the trees in some respects. Despite acknowledging in places that new data collection isn’t always the most promising route to greater wisdom, it neglects to consider the role that literature review, calibrated probability assessment, and other approaches not involving primary data collection can play in informed decision-making. The paper focuses on challenges and strategies without the same level of attention to desired outcomes. Because of this, I felt excited by the direction “New Directions” was taking me, but frustrated that it didn’t map out more of the journey.

What it all means: Are we ready to declare a crisis in the field around its data collection practices? Are people who commission or carry out new research without thinking either about its direct tie to decision-making or its strategic place in the literature not just failing to add value, but doing the arts an active disservice? There’s an argument to be made that organizations shouldn’t be asked to collect any data at all, except insofar as it serves their strategic purposes and they know what they’re doing; that, instead, all data intended to serve the needs of the sector should be collected by knowledgeable third parties working under a clear and coordinated research agenda. While “New Directions” declines to do so for us, it’s interesting to imagine what a vastly improved cultural data infrastructure would actually look like, along with how we might get there.

A graphic from the Cultural Data Project report, “New Data Directions for the Cultural Landscape.”

As any internet geek or high-priced consultant will be happy to tell you, we find ourselves today in the age of Big Data. You know, the era when science and numbers are supposed to solve all our problems forever? That one. And yet in the cultural sector, according to a report published earlier this year, we don’t have the data we need; we don’t know what to do with the data we have; and even if we did, we still wouldn’t use it to make decisions. (Okay, that may be oversimplifying things a bit…but not by much.)

So what are we supposed to do? That’s what the Cultural Data Project (CDP), which commissioned the report, wanted to find out. Many readers know the CDP as the folks behind those forms you have to fill out when you’re applying for grants. Its spinoff last year from a foundation-sponsored initiative housed within the Pew Charitable Trusts to an independent nonprofit prompted some organizational rethinking, and “New Data Directions for the Cultural Landscape,” by Slover Linett consultants Sarah Lee and Peter Linett, was one result. “New Data Directions” seeks to situate the CDP’s efforts within the larger context of data collection throughout the United States cultural sector.

The study synthesizes comments from an online forum that CDP hosted in late 2013 with a small group of cultural data experts drawn from academia and the consulting world. (Disclosure: Createquity’s Ian David Moss was one of the participants in this forum and is quoted a handful of times in the report.) It additionally draws from CDP’s internal strategic planning survey, a paper by Margaret Wyszomirski (not available online) that sought “to frame and inventory the cultural data landscape,” and the authors’ own interpretations and experiences.

After citing the benefits that data-informed decision making has provided in other fields, “New Data Directions” identifies a number of factors that it says are preventing us from reaping those benefits, namely:

• Poor accessibility, quality, and comparability of cultural data
• Norms about data collection and use, including low priority/importance assigned to the task of data collection in general
• Lack of coordination and standardization among existing data collection efforts
• Skill and resource capacity constraints among cultural nonprofits
• Organizations’ perceptions of the public and their audiences that inhibit the effective use of data in decision-making
• A paucity of vision and lack of role models regarding the successful use of data to drive decisions

To address these challenges, the report recommends some straightforward steps, including coordinating leadership on cultural data, engaging program and artistic staff in conversations about data, shifting the frame of data use from accountability to decision-making, developing a field-wide research and data collection agenda, developing data-related skills among organization staff, and improving the cultural data infrastructure.

Piece of cake, right? Though the report’s heavy reliance on the comments of a group of data experts might suggest that we’re off to a bad start in terms of “engaging program and artistic staff in the conversation about data,” Lee and Linett nevertheless surface several important and underaddressed issues facing the cultural data landscape. Perhaps the most crucial of these is a recognition that leading with data as the solution before carefully considering the problem it’s supposed to solve runs the risk of creating data that lacks a purpose. In many cases, arts organizations’ collection of data has been driven by the need to comply with funders’ reporting requirements rather than by a desire to collect information that could improve their future decision making. While the databases that have been generated through this process provide rich sources of information, it is not always clear what that information is good for, or how individual organizations can benefit from it. “New Data Directions” deftly calls attention to this “data first, questions second” mindset that appears to be so pervasive across the sector.

And yet to some extent, the report is itself a testament to the difficulty of escaping this mindset. By framing its entire exploration around data, “New Data Directions” inadvertently obscures the role that other forms of information and analysis, such as literature review, strategic goal setting, and probability assessment, can play in promoting better decisions. More to the point, if better decisions for better outcomes are what we’re really interested in, how much is suboptimal use of data really sabotaging that goal? The report seems simply to take it on faith that it is, without providing much evidence for that view or weighing how much other obstacles (e.g., management incentives) may be contributing to the problem. In part because of this focus on the activity rather than the goal, “New Data Directions” does a much better job of describing where we are and suggesting a path forward than imagining what we might find on the other side.

So it seems that last task is left to the rest of us. Without ever quite coming out and saying it, “New Data Directions” challenges all arts administrators to think purposefully about our role in addressing the situation mentioned at the top of this article. Are we ready to declare a crisis around data collection and use in the field? Do the observations and subjective impressions offered in the report resonate with your own experience? What would consistently effective use of data for decision making at the organizational and system-wide level look like in practice? What good would it do for the sector? And what would it take to make change in this particular way?

Your turn.

Cover image: “Data Represented in an Interactive 3-D Form,” courtesy of the Idaho National Laboratory via Flickr Creative Commons license.

Measuring the universe (Roman Ondak) — photo by flickr user Alexandre Dulaunoy

(This is the second post in a series on Fractured Atlas’s capacity-building pilot initiative, Fractured Atlas as a Learning Organization. To read more about it, please check out Fractured Atlas as a Learning Organization: An Introduction.)

Last fall, we put together a group of six people (henceforth referred to as the Data-Driven D.O.G. Force) to collectively develop a set of decision-making frameworks to help us resolve so-called decisions of consequence – situations for which the level of uncertainty and the cost of being wrong are both high. To do this, we’ve been taking inspiration from Doug Hubbard’s book How to Measure Anything, which introduces a concept he invented called “applied information economics,” or AIE. AIE is a formalized method of building a quantitative model around a decision and analyzing how information can play a role in making that decision. You can read much more about it in Luke Muehlhauser’s excellent summary of How to Measure Anything for Less Wrong.

One of the central tenets of AIE is that we can only judge the value of a measurement in relation to how much it reduces our uncertainty about something that matters. (More on that in a future post!) In order to know that, though, we have to have some sense of how much uncertainty we have now.

This concept of uncertainty is one that we understand on an intuitive level – I might be much more confident, say, predicting that I’ll be hungry at dinner-time tonight than predicting what I’ll be doing with my life 10 years from now. But most people don’t have a lot of experience quantifying their uncertainty. And yet, as forecasting experts from Hubbard to Nate Silver tell us, the secret to successful predictions (or at least less terrible predictions) is thinking probabilistically.

What does this mean in practice? Picture yourself at Tuesday trivia night at your favorite local pub. There you are with your teammates, you’ve come up with some ridiculous name for yourselves (like, I don’t know, the “Data-Driven D.O.G. Force”), and the round is about to begin. The emcee calls out the question: “the actor Tom Cruise had his breakout role in what 1983 movie?” Your friend leans over and says, “It’s Risky Business. I’m like 99% sure.”

Anyone who’s done time at trivia night will probably recognize something like that sequence. What I can virtually guarantee you, though, is that your friend in this situation hasn’t thought very hard about that 99% figure. Is it really 99%? That’s awfully confident – it implies that if your friend were to answer 100 questions and was as confident about every one of the answers as she was about this one, she would be right 99 times.

I’d be willing to bet that if you recorded the number of times people said they were “99% sure” about something and kept track of how often they were actually right, it would be significantly less than 99% of the time. That’s because as human beings, we tend to be overconfident in our knowledge in all sorts of ways, and this exact effect has been documented by psychologists and behavioral economists in experiment after experiment for decades.

This is why any AIE process involves something called calibration training. Overconfidence is an endemic and hard-to-escape problem, but if you practice making predictions and confront yourself with feedback about the results of those predictions, you can get better. In How to Measure Anything, Hubbard provides a number of calibration tests essentially consisting of trivia questions like the one above – except that instead of naming a specific movie or person, we’re asked to provide a ranged estimate (for numbers) or a confidence rating in the truth or falsehood of a statement. So for example, you might find yourself guessing what year Risky Business came out, or whether it’s true or false that it was Tom Cruise’s first leading role.

The six of us on the D.O.G. Force took a number of these calibration tests, and I’m gonna be honest with you – we were pretty awful. We got the hang of the binary (true/false) predictions relatively quickly, but the ranged estimates proved exceedingly difficult for us. In four iterations of the latter test across six individuals, only one of us ever managed to be right more often than we said we would be. You can see this in the results below (red colors and negative numbers mean that we were overconfident, green colors and positive numbers underconfident, and yellow/zero right on the money).

We were able to make good progress in the last round of the test (“Range supplemental 2”), though, primarily by focusing on making our ranges wide enough when we really had no idea what the right answer was. What’s the maximum range of a Minuteman missile? Well, if you don’t even know what a Minuteman missile is, your range should be wide enough to cover everything from a kid’s toy to an ICBM. It can feel incredibly unsatisfying to admit that the range of possibilities is so wide, but in order to construct an accurate model of the state of your knowledge, right now, you need to be able to articulate what “I have no idea” really means.

So why spend valuable company time working through a bunch of trivia questions? Because when we find ourselves needing to make estimates about, say, how much a new software feature might cost, or the number of people who might be reached when we speak at a conference, we suffer from the same disease of overconfidence if we don’t do something about it. What happens as a result is that we make predictions that are reassuringly precise in the moment, but might well end up far off from reality down the road. And when we use those inaccurate assumptions and predictions in our decision-making, there’s a good chance – so to speak – that we’re setting ourselves up for later regret.

Next up: how this all fits in with grand strategy!

[UPDATE: If you want to try a range test for yourself, Fractured Atlas’s rockstar Community Engagement Specialist and D.O.G. Force member Jason Tseng has created an arts-specific one! Here are the questions and here are the answers (don’t peek!).]