Art Appreciation – by Flickr user Dusty J, Creative Commons license

Large-scale, randomized-control studies – the gold standard of science, regarded as the best way to demonstrate that one thing actually causes another – are extremely rare in the arts education field, given ethical restrictions on experimenting on kids and the subjectivity of so many things related to the arts. So when researchers working with the Crystal Bridges Museum of American Art announced the results of such a study last fall, the art and mainstream media all rightly sat up and took notice. “Art Makes You Smart,” declared the New York Times headline, barely resisting an exclamation point and, perhaps inevitably, getting the findings wrong – though the enthusiasm for a thoughtfully designed study is certainly justified.

Crystal Bridges is an ambitious new museum of five centuries of American art located in Bentonville, Arkansas and funded primarily by Alice Walton, heir to the Walmart fortune. When the museum opened in 2011, it invited elementary schools in the area to apply for all-expenses-paid class field trips to the museum, including structured tours focusing on five paintings and an hour’s worth of instructional materials for classroom use by the teachers. All of the eligible schools were given trips – but they were assigned by lottery to time slots over the course of a year, which gave University of Arkansas scholars Jay P. Greene, Brian Kisida, and Daniel H. Bowen a top-shelf research opportunity. They gave the same assessment of cognitive and interpersonal skills to two groups of students: one that had already taken their field trips, and another that would take their field trips the following semester. By matching similar groups to one another and using regression analysis to control for variables like demographics and poverty, Greene and company were able to isolate the effect of the visits more rigorously than is usually possible, and for a population of thousands of students.

So what good does a field trip to an art museum do? Researchers catalogued a range of benefits for those who went, ranging from critical thinking skills to measures of tolerance and “historical empathy,” but perhaps the most striking finding is how magnified those benefits were across the board for children from schools in rural areas and serving low-income populations, with effect sizes double or triple those of other students. In fact, most of the positive impact from these field trips accrued to students who had never been to Crystal Bridges before, whereas “much smaller or null effects” were observed in relatively more privileged students who presumably have easier access to enriching experiences like this one.

The report’s strongest finding suggests that it boosts attentiveness to visual detail, at least for a little while. As part of the assessment, students in both treatment and control groups were shown this rather eerie painting, which they hadn’t been exposed to as part of the study, and asked to write about what they thought was going on. Their essays were scored independently by two researchers for evidence of critical thinking using a method designed by scholars at the Isabella Stewart Gardner Museum; basically, the readers added up instances of elements like “observing,” “interpreting,” “evaluating,” and “flexible thinking.”

For rural and poor students in particular, the field trip was correlated with higher critical thinking scores, and the increase was largely driven by more instances of “observing,” meaning that these kids cited more details in their analyses of the painting. The researchers bill this effect bluntly as enhanced critical thinking, but their other findings, which were less pronounced, suggest that something more specific may be going on. Students who had visited the museum were also more likely to try to imagine what people depicted in art were thinking (part of the authors’ measure of “historical empathy”) and less likely to want to censor anti-American art (“tolerance”). At least for a few weeks after visiting, when the assessments were administered, museums seem to encourage students to take art on its own terms: they look more closely; enter the world of a painting more fully; and suspend their prejudices more effectively. This is the state of mind that makes critical thinking – not to mention understanding and appreciation of alternative viewpoints – possible.

And from all appearances, kids love it. Not only were students who had been on a field trip more likely to say that they find art museums interesting and fun, they actually acted on this belief. All of the students were given coupons to attend a special exhibition at the museum with their families for free in the six months following the study; having visited with their class made them 18% more likely to take advantage.

Crystal Bridges now has a $10 million endowment for school visits. Rigorous studies along these lines would be a worthwhile use for what would amount to a very modest portion of the funding that this and other museums set aside for arts education. Art may or may not make us smart, but research can certainly make us smarter.

(Read our full capsule review of “The Educational Value of Field Trips” in Createquity Insider.)

Author(s): Jay P. Greene, Brian Kisida and Daniel H. Bowen (“Educational Value of Field Trips”); Jay P. Greene and Brian Kisida (“Supplemental Study”); Jay P. Greene (“Methodological Appendix”).

Publisher: EducationNext

Year: 2013

URL: http://educationnext.org/the-educational-value-of-field-trips/; http://educationnext.org/supplemental-study-long-term-benefits-of-field-trips-to-the-walton-arts-center/; http://educationnext.org/methodological-appendix-for-the-crystal-bridges-experimental-study/

Topics: field trips, museums, arts education, experimental designs, visual art

Methods: Randomized controlled trial. Researchers worked with Crystal Bridges to assign spots to matched pairs of applicant groups. Treatment group received a tour that semester and control group was deferred to the next semester. Surveys were administered to the treatment group – 489 teachers and 10,912 students – around 3 weeks after their tour. The surveys assessed “knowledge about art as well as measures of critical thinking, historical empathy, tolerance, and sustained interest in visiting art museums.” One of the survey items involved writing a response to a work of art unfamiliar to the students; 3811 of these essays were graded for critical thinking skills according to a methodology developed by researchers at the Isabella Stewart Gardner Museum. 750 essays were coded twice with a high degree of inter-coder reliability. For the analysis, regression models controlled for gender, grade level, and matched pair group, while “standard validity tests confirmed that the survey items employed to generate the various scales used as outcomes measured the same underlying constructs.” A different version of the survey was administered to students in kindergarten through second grade.

What it says: Crystal Bridges is a new art museum in Arkansas that opened in 2011. As the only major art museum in the area, it offers nearby schools all-expenses paid tours for their students. These field trips were very popular on opening and the excess demand created an opportunity for a randomized study. The field trips involved an hourlong tour of the museum during which the students typically discussed five paintings, along with an additional hour or so of classroom instruction with the assistance of materials provided to the teachers. Students who participated in the field trips were able to recall factual details about the paintings at a fairly high rate, display improved critical thinking skills in evaluating a work of art they hadn’t yet seen, manifest increased “historical empathy” and tolerance for viewpoints different than their own, and report a higher interest in visiting museums in the future. All of these effects were stronger among students from high-poverty schools and especially pronounced (with effect sizes sometimes doubled or even tripled compared to the overall control group) among students from rural schools. In addition, the researchers set up a behavioral measure to test willingness to visit a museum by handing out vouchers for free admission to Crystal Bridges to the treatment and control groups; despite receiving 49% of the vouchers, 58% of the visits came from the treatment group – a particularly impressive result given that these students had just recently visited the museum and their appetite could have been satiated by the experience. It appears that to a large degree, the higher effect sizes seen among disadvantaged students can be explained by the fact that this was their first time visiting Crystal Bridges, as students in the treatment group who were new to the museum showed similarly impressive improvements. The K-2 students, who are also less likely to have visited a museum before, likewise exhibited especially strong gains. Notably, students from low-poverty schools and large towns appeared to benefit little or not at all from the exposure.

A second, supplemental study took advantage of a natural experiment in school zoning that resulted in some students making more field trips to performances at the Walton Arts Center than others. This study finds beneficial outcomes for students who took more field trips, including increased desire to attend cultural events or programs, increased desire to participate in choir or drama themselves, increased (self-reported) empathy, and increased (self-reported) tolerance. Notably, these effects are present even though the field trips took place over a period of years.

What I think about it: Overall, the main study exhibits an impressive degree of thoughtfulness in its design that allows for quite a bit of credence in its findings. The questions, about the educational value of one of the most common arts experiences that children have, are quite relevant to practice, and the findings regarding the value of field trips for first-time museum attendees are particularly provocative. For all of the study’s strengths, at many points the effects of attending Crystal Bridges specifically are conflated with the effects of visiting any museum or having any culturally enriching experience, and the authors could have done more to distinguish these by, for example, tracking what other cultural activities the students in the treatment and control groups had been or were being exposed to. (The supplemental study does address this in part by changing the focus to performing arts, but the design is not as tight in this case and the effects may well be attributable to the school environment in general rather than to the field trips specifically.)

In addition, I wonder to what degree some of the results observed are the effect of priming rather than the actual museum experience, particularly survey questions about things like freedom of expression in museums. It would have been interesting to divide the control group and have some students be told or reminded shortly before the survey that they are going to be visiting Crystal Bridges the following semester while other students are not given this stimulus. One important limitation to note is that the study measures only the short-term impacts of the field trips, and it’s unclear if the noted improvement in critical thinking skills would carry over to non-arts contexts. Again, the supplemental study addresses long-term impacts obliquely, but not directly.

What it all means: The Crystal Bridges experimental study strongly suggests that facilitated field trips have benefits to elementary school students, especially those from disadvantaged backgrounds and who have not had the opportunity to visit a museum before. The experience promotes prosocial attitudes and at least one means of skill-building in the form of visual attention to detail, though it is not clear if this carries over to other kinds of critical thinking tasks. While the effect sizes are fairly small on the whole, they appear to be real and are impressive considering the unsustained nature of the field trip experience. All in all, this is strong evidence of the importance and value of “common” opportunities to participate in the arts even beyond their role of sparking self-actualization through making or performing art.

Recently, Createquity waded back in to the smart-giving waters after an op-ed by bioethicist Peter Singer comparing donating to a museum to donating to a blindness charity understandably didn’t sit well with the museum community. Singer’s argument had its roots in an emerging area of applied philosophy called “effective altruism,” which argues that we have a moral imperative to do the most good we possibly can and use objective criteria to figure out what that good is. GiveWell has indicated its support for the effective altruist movement, so I thought it was high time to catch up with them to figure out where the arts fit in to all of this.

What was interesting was that the GiveWell folks seemingly came into this experience with a genuine desire to learn from my perspective as much as I was eager to learn from theirs. So at various points I found myself as suddenly the one answering questions, and in particular being challenged to articulate what funding opportunities might exist within the arts that self-aware philanthropists should be paying attention to.

This is a long but rewarding read. Tim and Elie were gracious enough to talk with me for over an hour, and the conversation will be of interest to anyone thinking seriously about philanthropy, advocacy, or research in the arts. That said, simply reproducing the whole thing verbatim here would make for by some margin the wordiest-ever post on Createquity (and that is really saying something), so rather than subject you to that, I’m sharing some of the highlights, condensing and moving things around a bit for the sake of readability.

On Where the Arts Fit in to GiveWell’s World

IDM: GiveWell hasn’t historically given a whole lot of attention to the arts, although I know the arts have been among a broader list of causes considered by the organization. I’m wondering if you can talk briefly about GiveWell’s current orientation to the arts, if any.

EH: There’s two main things I’d tell you about the arts and how they relate to the work that GiveWell is doing. For a long a time GiveWell was almost entirely focused on what we’ve termed evidence-backed, cost-effective, internationally-focused interventions. The arts really didn’t fit into the frame of GiveWell’s research process as it was originally constituted. More recently, as we’ve been working on this broader-scoped research that we call GiveWell Labs, I think it’s not as clear where the arts fit.

One of the things that we’ve always done at GiveWell is research the causes that we collectively, meaning our staff, are most interested in supporting. Early on when GiveWell just started, [it] was just Holden and Elie thinking about where we would give charity. I think now that’s broadened out to the staff we have. My impression is, and I’m certainly speaking for myself, but I think for other staff, that we tend to be more engaged in questions of giving to the causes that we’re currently researching, causes focused on international aid or US policy or scientific research, rather than the arts. And so to some extent those personal interests drive the research we’re doing.

One of the main reasons that we’ve done this is we’ve found that when we are trying to answer the question [of] where would we give our own funds, we tend to do better research then where we’re trying to answer something that I’d say is perhaps more of an intellectual question, which is where would I give if I were interested in something else? So that’s one part of the answer. The other thing I think is just important to ask, and it’s one of the questions that we’re asking for all the causes that we’re currently considering, is to what extent does this field have sufficient funding, versus not? I can’t say that I’m familiar enough with all of arts funding to know exactly how it stacks up, but [I have] sort of a superficial impression that there’s lots of ways in which people can get funding for the arts, whether through, let’s say, privately funded entertainment or government grants or otherwise, and there’s a lot of interest among philanthropists in providing that funding. And so one of the questions that we would have if we were to be involved in this area is what part of this field seems to be under-invested in. I think that question of where additional funding or current funding is not quite meeting the needs is one of the main ways that we’d think about this…[but] in many ways, because of the first point I made I don’t think we’re particularly well positioned to answer [it].

On Prioritizing Basic Versus Higher-Order Needs

IDM: Is it fair to say that GiveWell prioritizes serving the bottom of Maslow’s pyramid or hierarchy of needs? I’m wondering if those concepts of Maslow figure into any of your conversations or thinking about values, or if it’s more coming from an intuitive sense that poverty is central.

TTL: Yeah, I’d say we aren’t just focusing, and don’t want to just focus, on the bottom third or some tier of Maslow’s hierarchy. Traditionally, all the recommendations that we’ve made to date, as you point out, have been in global health and direct aid to people that have dire needs or needs that are different than the needs of people in developed countries.

When we were first deciding what causes we wanted to work on, we wanted to limit it to just causes that had really good evidence of effectiveness, and we found pretty quickly that the types of causes that had really good evidence were interventions in global health and developing countries and direct aid such as using bednets to prevent malaria deaths. There’s been over 20 randomized controlled trials that have connected the properties of bednets to reduce malaria and reduce malaria rates [and] deaths of, especially, people under 5 years old. There are very few interventions available to philanthropists out there that can claim that level of evidence. That was one of the big reasons for our historical focus on global health and direct aid interventions.

What we’ve been working to do recently is also open that up to a broader range of possible causes to look at, and that’s the project we’ve been calling GiveWell Labs, which still hasn’t made any recommendations yet. The causes we’re considering within GiveWell Labs include things that are not just focused in the same areas and includes things like trying to understand if there are ways that a philanthropist can improve scientific research or can change aspects of the political process in the US or elsewhere and a bunch of other causes as well.

We’re definitely very open to the idea that it’s possible to have more impact per dollar with things that are outside of developing health, or things that don’t just affect the bottom tier of Maslow’s hierarchy as you’re saying. But when there’s not as much academic literature on a specific intervention, it’s certainly a lot harder to understand that impact and it’s taking us a long time to try to understand.

IDM: Do you have a formal definition that you use, or even an informal definition, of what the good is that you guys are seeking to create in the world? Because I’m wondering when there are tradeoffs between those kinds of needs, how do you compare higher-level needs to lower-level needs in thinking about that hierarchy?

TTL: Yeah, I think this is a great question. It’s a hard one, and we have not formalized what values we are trying to maximize, if you will, or how to trade off the value of saving the life of someone that’s less than five years old versus maybe reducing the chance of mental development problems in another person, or improving the life of someone in a developed country, or maybe improving an institution like a government that will affect a whole lot of people.

EH: I think the main thing we’ve written that I would just point you to is this blog post [GiveWell co-founder] Holden [Karnofsky] wrote about a year ago called “Deep value judgments and worldview characteristics.” I care about self-actualization, so in some ways, I can easily imagine us being excited about things at the higher end of the hierarchy of needs, but I think it would really depend on the specifics of the circumstance.

One of the things that that blog post talks about is that we are not putting strong weights on achieving specific things in and of themselves – so some artistic endeavor as, like, some sort of achievement, as much as the broader impact that those types of activities could have on individual self-actualization. And so again, I think that one of the challenges for us in engaging with a type of philanthropy that we’re not particularly involved in now is understanding how the activities fund and would contribute to the types of goals that we would value.

On Effective Altruism and Strategic Cause Selection (aka Can You Work in the Arts and Still Be an Effective Altruist?)

IDM: I loved that you guys published a roundup of the GiveWell staff’s personal donation decisions this past December. It was super interesting. One thing I noticed was that there were a couple of staff who chose not to allocate all their charitable dollars to GiveWell-recommended charities. [But] some of the logic that we hear from a theoretical standpoint from effective altruists has to do with the idea of concentrating resources on high-impact opportunities rather than spreading the wealth around.

I’m wondering if you could talk a little bit about the balance between personal passions and feeding those through charitable activities on the one hand, and on the other hand, the moral imperative that a lot of people involved with this movement do lay out around the idea that you really should maximize the expected amount of good that you can do in your life.

TTL: The way I think about this broadly is that it’s important to me to have as big of an impact as possible and to approach that question sort of systematically. For me, not surprisingly, GiveWell is my primary resource for figuring out how to do that with the bulk of my funds – and I guess on top of that, it’s also how I’ve chosen to try to do that through my career – but then there [are] a bunch of reasons why maybe I should give in ways that aren’t just GiveWell top charities. I think you saw a bunch of these in the staff giving profile but, you know, it includes things like, well, if you have particular or special knowledge of a particular area then that might be a really good reason to expect that you might have a really good giving opportunity even if the broader community or GiveWell in particular hasn’t discovered it and developed the same sort of public degree of confidence that you have privately.

Additionally, for me, I think that certain types of heuristics in terms of one’s giving habits or patterns can be really useful even if they can’t quite be justified in this typical sort of straight-line effective altruist or consequentialist type perspective. Even if you can’t prove or you have no expectation that this marginal dollar if given by anyone would be best spent in this particular way, maybe if it’s related to something that you care a lot about or you use as a service yourself. Then that is an additional reason to value it, or to value the principle in general that people using that service might contribute to it to some extent.

IDM: In my professional life, I work with a lot of people who are very cause-centric, right? [Laughs] People care a lot about the arts. And so I’m wondering if you feel that there are principles from effective altruism, or from your general approach to giving, that could be applied even within a cause? As background, I’ll just tell you that when we were working on our effective altruism article for Createquity, we had a lot of debate internally about whether the idea of effective altruism in the arts is an oxymoron because of that cause-agnostic nature of effective altruism.

TTL: I don’t think it’s an oxymoron. I think that it’s totally possible to – if you can restrict the set of possibilities to some subset before, and then even within that subset, there [are] going to be causes that have more of the impact you’re looking for or less of the impact you’re looking for per dollar.

And so I absolutely don’t think it’s an oxymoron. I think that if I had some pot of money that was going to be dedicated towards the arts, then I would definitely be interested to know what are the opportunities to make changes out there, which of the opportunities seem to be most effective could actually be scaled up with more money, versus they might be really effective but giving them more money won’t allow them to do more of the same work, and other related questions.

EH: Yeah. I mean, I think there [are] a lot of the sort of questions and tools that we ask that I can easily imagine applying well to the arts. I think one of the main questions I’d have is, how does the arts funding ecosystem work, and what types of activities or outputs are for whatever reason not valued by the current funding infrastructure, but they appear to achieve the same types of goals, or the goals that one has as an arts funder or an artist?

Those are the types of things that I think come out of what I would characterize as the broad goals of an effective altruist, trying to use the part of your time or charitable funds that is being directed towards altruistic rather than perhaps personal goals as effectively as possible.

While I think people will reach different conclusions about which causes they are excited to work on, there is nothing that seems particularly problematic to me about someone saying, “the way in which I think that I can best contribute to the world is via the arts and, therefore, I’m going to try and maximize in some broad sense the impact that I have in that domain.”

On How to Think About Giving to the Arts

EH: Sorry, just to follow up actually I have a question for you if that’s okay. I mean, I think one of the questions that I would have when thinking about the arts is, what is the problem that additional funding could solve? I think that would help me because I think I have a relatively superficial understanding of what the problem might be, but I would characterize it in such a naïve way that I’m not sure it’s particularly helpful. So my naïve characterization might be something like, we could fund more art than we are currently funding, and the thing that would start to help me think this through more carefully would be, you know, what are we not funding that we should be, and how bad is that, and how much funding would it require? And I guess, then, ultimately, what could that mean to the development of a more complete, richer world arts community? Those are some of the things that I think I would want to ask when starting to think about this question.

IDM: Yeah, so two things, I guess, on that. The first is that I think the arts in some ways have struggled with this tendency of the broader philanthropic and nonprofit or social sector community to frame things in terms of problems, because what I think a lot of people in the arts might say is that we’re not here to solve a problem, we’re here to create possibility. We’re here to sort of extend the universe of what it is possible for humans to do in a way.

And in some ways, what we do has more in common with something like higher education or even science then it does with international development or aid or things like that. With that being said, I think that your question is still valid and important, because you focused it specifically around the idea of, well, what are the opportunities that we’re missing specifically with respect to funding?

I think that there are a lot of potential ways to answer that, but the reason why I asked about Maslow’s hierarchy of needs is because if you think about where the arts kind of fit into that, you know, it seems pretty clear to me that where they slot in is in that top need of self-actualization. The arts, creativity, and sort of related concepts – I don’t think anyone would argue that it’s the only form of self-actualization, but Maslow himself talks about that being one of the ways in which self-actualization manifests.

[Later on…]

EH: I do think there is this question about the arts, which I would be interested in hearing from people who are themselves very interested in providing charitable support there, answering the question of how those funds will make a difference. Because I guess I don’t want to, sort of let the arts off too easy relative to any other cause, and I’d be interested in this question of trying to determine what is not being funded that should be, and why. Because it strikes me that there are a lot of institutions and individuals who are interested in being part of the arts and funding the arts, and so there’s something of an obstacle to overcome in terms of convincing, me, let’s say, or other donors that additional funding is really what is most needed there.

IDM: So let me ask, do you think that the greater obstacle for you is more about the value of the arts in the abstract, compared to some of the other things that GiveWell focuses on? Or is it more about, as you kind of expressed just now, a lack of familiarity or confidence that, in GiveWell’s term, there is room for more funding in the arts?

EH: I think the issue is more a room for our funding issue, but I’ll try to explain what I mean by that and then let me know if this makes sense. Basically, I think a world – like, imagine you could just take all of the funding and time that goes into arts and totally take it away, and now it all goes to just, I don’t know, like poverty prevention programs.

I mean, that doesn’t strike me as the ideal balance for the world. You know, like absolutely no entertainment or literature or painting or music. I mean, that does not seem like a good world to live in and so, now, again, I’m just kind of giving you my own values and my impression, [but] I wouldn’t want to see a world where there was none of that. And so, therefore, to me the big question is, does this area have sufficient funding or insufficient funding to engage humanity as much as it potentially can or should, relative to the other needs that people have? That’s a very hard question to answer, but that’s the way that at least I personally look at it.

IDM: So, I think my readers might kill me if I didn’t at least attempt to hazard an answer to that question. I’ll preface this by saying there is no sort of canonical consensus around the answer to that question of, you know, what is it that philanthropic intervention in the arts is supposed to do? But a while back I articulated two ways of thinking about justifications for subsidy of the arts which are mine alone, but also do have antecedents and connections to other work that people have done.

I don’t think it’s realistic to imagine a world where there is literally no art or entertainment, or anything like that. Because it’s part of human expression and people find a way to make it happen, sometimes in very adverse conditions.

[But] if it were only up to the commercial marketplace to decide what art gets created and who gets to be an artist, there would be two things that would happen. In the long run, over time, on average, you would have art and cultural products that cater to a wide, broad-based audience, and so you’d lose some of the diversity of product. You would lose a lot of the most interesting kind of expressions of human creativity that you get, and there are plenty of examples of artists who are considered very famous or important today that basically survived to the present day entirely because of luck. If they survived because of luck, then how many other geniuses or brilliant contributions to the literature or to the set of human achievement were lost, because they were never created in the first place or because they were literally lost? That’s one kind of justification.

The other justification is – so, if we go back to this idea of self-actualization and sort of take it as a given that for at least some people, the path to that is through being an artist or through engaging with the arts in some really deep sustained way in order to have peak experiences, understand and really experience what it means to be alive in this very present and visceral way [such] that you could make a moral argument that everybody deserves to have that opportunity – people’s access to the arts is determined in many ways by the market. And there are many disparities in the level of access that is available to people in various ways, for example due to cuts in arts education funding, it’s much less common now for people from poor or minority communities to have access to arts education than was the case in the past. That’s not necessarily to say that they won’t come into contact with the arts outside of school, but it’s less likely that they will have these pathways into discovering themselves through this medium that is one way to kind of achieve one’s potential. That’s sort of the way that I’m currently thinking about it.

EH: Got it. Yeah, I mean so those two points, and I think maybe this is just something about definitions, but I think that this problem that people who are perhaps socioeconomically disadvantaged have less access to the arts, it’s something that I would almost categorize as part of the general cause of inequality in the rich world. That’s to just say that is broadly speaking how I mentally file this cause, and it would almost be outside of art specifically.

On the first point, you know, I think the place I start is I think the most recent Giving USA survey data says there was roughly $14 billion given to the arts in 2012 and $19 billion given to international aid. And so the question is, you know, we can all agree that here should be, or at least I’m willing to agree that there should be some level of non-market-based arts funding, and then the question is should it be equivalent, roughly speaking, to the amount going internationally or should it be more or should it be less. That seems like the major question to try to answer and it becomes difficult to answer what the appropriate level should be in some abstract sense.

And so that’s why the approach that we’ve taken, at least in the research we’re doing under the name GiveWell Labs, is trying to look for specific areas that where we’re seeing ideas or problems that don’t seem to be funded in the way that they should be, where you can almost see the full concept and idea behind a lack of funding in a particular area. And you can say, you know, this thing, it would cost X dollars and it appears to have insufficient funding, therefore, this is something that is worthy of serious consideration.

On Evaluating and Allocating Resources to Research

IDM: You guys have devoted quite a lot of resources over the last few years to reviewing research literature, often either in connection with GiveWell Labs or to develop a knowledgebase of evidence-backed [interventions] in international aid.

I’m curious if you could talk a little bit about how your process has evolved and changed since you first started. I’m especially interested in whether you feel like you’ve kind of hit upon the answer at this point to what an effective research process is in terms of just going into a completely new area and finding out as much information as you can about what the evidence base is for guiding philanthropic decisions, or if you feel like there is still inefficiencies and problems that you’re still trying to figure out.

EH: Yeah, so the short answer is we don’t have it all figured out yet and there’s a lot we’re still trying to figure out about the best research process. The longer answer is that I think that we have come to a reasonably good process for our traditional research on international aid organizations but even that, you know, is not particularly formulaic because it varies a lot based on the specifics of the intervention or the organization.

There’s two different ways that we’ll look at an intervention. One is the more traditional GiveWell focus, which is very specific interventions that have a great degree of rigorous evidence evaluating their effectiveness. Another type, I wouldn’t even call it an intervention as much as a charitable program area, you know, where one might say hey, we could have a big impact on the world if we were to increase labor mobility or have some sort of software patent reform. These are areas that I don’t think one could call the activity we undertake evidence review as much as trying to get a better sense of the area.

I think the first kind is one where we have a pretty standard process we go through of looking for research that evaluates the question we have. You know, do bednets work, how well do they work. Then we are trying to think of all the questions that we have of the ways that the program could fail and then looking for literature on those questions. So, in the case of bednets, just to play out this example, it would involve how often do people actually use the bednets and was it the case that they only used the nets in smaller, randomized trials but in a larger-scale government program they might not. Or what impact does insecticide resistance have. So then we just go about listing out the questions and trying to answer them.

IDM: So, there’s a piece of that that you’re glossing over a little bit that I’m really interested in. I have to imagine that in the area in which you’re looking, there are hundreds, maybe thousands of studies that are potentially relevant to the questions that you’re looking at. So what are the filters that you use to decide which studies you’re even going to take a look at in depth? And then do you sort of structure the process in such a way so that you are looking at some of them at a shallow level, some of them at a deeper level, and so forth?

EH: The biggest filter that is imposed in the health interventions is we give serious priority to randomized controlled trials, which are created explicitly to evaluate the causal relationship between the intervention and the outcome in a way that other study methodologies have greater challenges to overcome.

That said, we don’t only focus on randomized trials. There’s evidence in our reports that comes from other types of evaluations, other types of studies, but because other types of studies often are not created in such a way to answer causal questions as directly, as easily, and it’s really the causal question is the one that we have (meaning “what can we say generally about bednet effectiveness?” is a question of what the causal relationship is between distributing bed nets and cases of deaths from malaria), we tend to prioritize the randomized studies.

TTL: The other thing that can be really useful when there’s thousands of studies in a general area that you’re trying to understand is using other people’s literature, or in the case at least when there is a lot of randomized controlled trials, there’s some times meta-analyses that are done to try to combine the statistical power of many of these different studies.

Now, I don’t know if this actually applies in the arts. I don’t know how common randomized controlled trials are or whether there is …

IDM: They are not. And I’ll just tell you guys that it’s a little bit funny to hear you talk about how you have so many doubts about the room for more funding in the arts and the general impression that the arts are overfunded. I don’t think that you actually used those words, but the thing is that compared to, like, the NIH spending on research, the amount of resources that actually go into research on the arts is incredibly paltry.

It’s true that there are big, big sums of money spent on arts organizations and arts interventions, but a lot of times that goes to things like buildings, whereas only a tiny fraction of that amount might actually go into studying whether that building ever made a difference to anybody.

I think it’s interesting because, while I think there are lots of arguments that you can make about the relative proportion of funding in the arts versus other areas, I would imagine that the typical ratio of funding that is spent on research about the topic or evaluations of the topic compared to the amount that is actually spent on the program delivery is way, way, way lower in the arts than it is in a lot of other fields.

TTL: It sounds like you think there is a lot of, the research on arts effectiveness is very underfunded.

IDM: I think so, yeah, and it’s, and because of that, you know, by the kinds of standards that you guys are using, the overall quality of evidence in the arts is pretty poor. There’s just, there are a lot of things that haven’t been studied, or they have been studied but not with the kind of rigor that you guys are looking for in your process.

EH: You know, the reason that earlier I was trying to distinguish between sort of these evidence-backed interventions versus other types of research that we’re doing for GiveWell Labs is I really think the latter is the one that seems like an easier fit for the arts, and the one that makes more sense.

It’s almost like I think there needs to be something of a more qualitative case that some part of the arts is underfunded or there is some segment that should be funded to a greater extent than it already is. I wouldn’t expect that rigorous evaluations are the right fit for evaluating that type of activity because I’m not even sure that we could agree on what impact we’re trying to evaluate.

IDM: Right. That makes sense. Could [you] describe a little bit more what that more qualitative analysis looks like? And in particular, I’m curious, is that entirely or almost entirely a theoretical exercise, or are you drawing in research that maybe doesn’t reach the level of randomized controlled trials and is maybe a little bit less expensive or less ambitious as part of the background for information-gathering for that analysis?

EH: I think the best way to get an idea of how we do that research is, we have these web pages that we’ve published that we call GiveWell Labs investigations of new causes or also called shallow investigations. They’re our initial look into various different areas.

On each of these pages, we do our best to answer the questions that we have about that area. It’s kind of like the things that we want to know in 10 to 20 hours of investigation. The questions we’re trying to answer are, what is the problem, and as part of what is the problem, some sense of how big a problem this is in the scheme of things. I think we’ve taken a lot of different approaches to answering that question, but on some level, trying as much as we can to quantify the problem and when we can’t quantify anymore, trying to explain it more qualitatively.

So you’ll see that on these pages. The other question that we’re trying to answer is a question about tractability. We can define the problem, but what can be done, and how likely are these goals to be achieved? Again, these require, without a doubt, a large degree of qualitative judgment about what it is and is not feasible and what is and is not likely, and we largely form these conclusions through conversations with people in the field. In the issues that are listed on this page, the shallow investigations, maybe we have two or three conversations with people in the field. Then there are other investigations that are larger, we call them “medium investigations,” maybe there we’re talking to 25 or 30 people to just try and triangulate what we can understand about the area.

Then finally, we’re asking the question, how crowded is this area? Who else is working here? How much are they funding? What are they funding? Putting it all together, areas where the problem is large and seems particularly tractable, and there is relatively little philanthropic funding, or if there is funding, we can understand why it is focused on part A of the issue but not part B. Those are very attractive and areas that say seem less important, less tractable, but highly crowded are less attractive.

In practice, things don’t kind of fall out so nicely; like normally problems have some combination of these factors and ways that require some thinking about how exactly to prioritize them. Those are the types of questions we’re asking and the types of information that we’re trying to feed into our process as we think about what we’re doing. To me, you know, these are the questions that I would have about the arts. Are we talking about, I don’t know, large museums in major cities? It seems like there is a lot of funding that goes to the Met, and the Guggenheim or other museums like that. I’m sure I sound hopelessly naïve when talking about the arts but that’s one type of question.

IDM: You are stating fact, my friend.

EH: And then maybe on the other hand, you know, you say, well, really the issue is funding of arts access in poorer communities. You could do a little investigation of that area and try to determine, is this something that people focus on and to what extent do they? We would wonder, like, is it that there’s no funding from local government as part of schools? Is there just no interest from major donors? How much money really is there? What could we expect to happen if this were to go well?

Those are all the questions that we ask. One thing just to add, and I know I’ve gone on for a little while on this, but another broad type of activity we’re undertaking in this area is what we call the history of philanthropy project where we basically say we recognize that all of these areas that don’t have that same type of rigorous evidence so arts, but also policy, or even science – it’s harder to know what will work.

One of the things we’re trying to look at is just what has worked historically when philanthropy has been involved, and this is an area where there is very limited information available. The basic idea is to try and do something that is more like investigative reporting or journalistic reporting where you better understand the role philanthropy has played. And I could imagine that also being helpful in thinking about arts philanthropy, where you can look back and say you know, what did someone do 30 years ago and what impact does that seem to have had? It obviously can’t be quantified in the way that saving lives with bednets could be quantified, but it can perhaps offer a deeper picture of what role philanthropy plays in achieving some outcome.

• A consortium of City of Detroit creditors have made the first legal move towards pressuring the Detroit Institute of Arts to sell city-owned artworks to help pay for debts owed. Executive Vice President Annemarie Erickson defends the museum against Emergency Manager Kevyn Orr’s demand that the museum find one way or another to contribute $500 million in assistance to the bankrupt city.
• The California Arts Council will apply a $2-million funding windfall it received from Assembly member John Perez to several new initiatives in arts education and community improvement, including Creative California Communities, The Arts in Turnaround Schools, and Jump stARTS. In the face of a 7.6% budget cut handed down last year, the state arts council is taking a gamble on the success of these programs winning fresh credibility with policymakers and an increase in annual funding.

MUSICAL CHAIRS

• Jamie Bennett, chief of staff and director of public affairs at the NEA, will take over as executive director of the creative placemaking funder collaboration ArtPlace America starting in January. He succeeds ArtPlace’s founding director Carol Coletta, who joined the Knight Foundation back in March, and interim head Jeremy Nowak.
• After a decade serving Californians as president of the James Irvine Foundation, James E. Canales will step down in the spring to become the first president of another arts funder, Boston’s Barr Foundation.

• There has been some shuffling in the world of state and local arts councils. Ohio Arts Council ED Julie Henahan has retired after thirty years; Milton Rhodes, President of the Arts Council of Winston-Salem and Forsyth County in North Carolina, has retired and been succeeded by Jim Sparrow; and Glenda Toups was dismissed from her position as ED of the Houma Regional Arts Council in Louisiana in the wake of the discovery by the board that the Council was not in compliance with state reporting law.

• We’ve known for a while that Michael Kaiser is leaving his post as President of the Kennedy Center for the Performing Arts; now it turns out he’s taking the DeVos Institute of Arts Management with him. Both are moving to the University of Maryland, where Kaiser will be a professor of practice beginning in the fall, and hopes to expand the Institute to include a master’s program.
• Financial news giant Bloomberg has decided to discontinue its cultural journalism brand, Muse, in favor of focusing more on leisure and luxury. Along with the reassignment of Muse editor Manuela Hoelterhoff and a cadre of employees and contracted writers, the news outlet laid off theater critic Jeremy Gerard.

ALL ABOUT THE BENJAMINS

• The Hewlett Foundation has announced a rigorous new “Openness and Transparency” policy, which assumes from the outset that information the foundation creates should be made public to improve outcomes, spark debate, and foster collaboration. Hewlett’s President Larry Kramer offers context in a post on the foundation’s new blog; transparency watchdogs celebrate the policy.
• The D5 Coalition has released a scan of best practices and a guide to online resources for foundations wishing to advance diversity, equity, and inclusion at every stage of their work.

IN THE FIELD

• Eric Booth and Tricia Tunstall share profiles of El Sistema “encounters” in five of approximately 55 countries – Sweden, Austria, Korea, Japan, and Canada – that have borrowed from Venezuela’s seminal movement to realize youth development goals through “intensive investment in ensemble music.” The global umbrella for El Sistema has also released the first literature review of “research, evaluation, and critical debates” related to Sistema-inspired programs around the world.
• The Arts Council of Lawrence, New Jersey has shut down after 42 years, having, in the words of one member, “outlived [its] usefulness.” Originally formed by a group of female volunteers, the Council struggled to recruit younger members throughout the recession.
• The August Wilson Center for African American Culture in Pittsburgh is struggling mightily. After a struggle to find an audience and keep backers the organization has been forced to move further and further from its original intention to create a cultural home for the people portrayed in Wilson’s plays, working class African Americans. A conservator has been appointed to try to avoid liquidation.
• The Warehouse, an all-ages music venue in La Crosse, Wisconsin, has filed to become a nonprofit after 22 years as a for-profit, prompting some musicians to wax lyrical about their time there. Financial pressures were the primary impetus, but owner Steve Harm has indicated he will open the space to the local community in new ways to provide a public good.
• Fractured Atlas has added another tool to their encouraging-and-rewarding-arts-entrepreneurship tool belt. The Arts Entrepreneurs Awards will recognize artists and arts organizations who have “innovated new business practices or paradigms” or  “developed novel solutions to old problems.” Nominations will be accepted until December 22nd at 5:59pm.

BIG IDEAS

• A report from the Next Generation Evaluation Conference forecasts “game-changing” trends in impact evaluation, including shorter evaluation cycles and simpler measurement systems.
• “Is social practice gentrifying community arts out?” Arlene Goldbard parses the difference between the art world’s latest obsession and community cultural engagement.

RESEARCH CORNER

• Southern Methodist University’s National Center for Arts Research is about to release its inaugural report, drawing on what it calls the “most comprehensive set of data ever compiled” on arts organizations.  In addition to a statistical overview of the field – did you know that performance of an arts organization is lower in communities with a higher concentration of graduate degrees? – the report attempts to answer the question, “What makes one arts organization more successful than another?” The key turns out to be leadership.
• Speaking of data aggregation, Markets for Good has a progress report on the BRIDGE (Basic Registry of Identified Global Entities) project, an ambitious collaborative effort to identify and map philanthropic entities across the world.
• A new report by the Consumer Federation of America bashes “abuse of market power by a highly concentrated music sector,” argues against the need “to expand copyright holders’ rights,” and suggests that digital file-sharing (aka “piracy”) may, in some cases, actually be good for both artists and consumers. One well-circulated chart suggests that it is the proceeds of live performance, not recordings, that drives artists’ income.
• Gold standard at Crystal Bridges? In a rare, randomized, controlled (albeit “natural”) experiment on the effects of art on students, a single school-group visit to the major new museum appears to have raised students’ scores on vague but desirable traits such as critical thinking, social tolerance, historical empathy, and likelihood of future museum visits. It’s too soon to parse out the effect of contemporary art in particular.
• A study of STEM graduates from the Michigan State University’s Honors College found that graduates who went on to earn patents or start companies had more arts and crafts experiences than the average Americans – and believed their ability to innovate was influenced by that experience. (The paper itself is behind a paywall.)
• How “rampant” is gentrification? New research suggests that most urban areas experienced only “moderate” gentrification in the past decade, with significant variations across cities. Unsurprisingly, gentrification was most prevalent in large and dense metro regions with solid public transit infrastructure.