"The Bridge" by artist Elena Colombo, image courtesy of ArtsQuest.

“The Bridge” by artist Elena Colombo, image courtesy of ArtsQuest (MICD25 grantee)

(As expected, Ann Markusen’s article Fuzzy Concepts, Proxy Data: Why Indicators Won’t Track Creative Placemaking Success has provoked lots of discussion and reaction among the creative placemaking practitioner community. Much of this can be found in the comments to the original post; in particular I recommend John Carnwath’s extensive discussion of the costs and benefits of measurement in the arts. On top of that, we now have official responses from the two major national creative placemaking funders that were the subject of Ann’s critique. ArtPlace posted a detailed rebuttal last week on its website, and below we have an essay penned by National Endowment for the Arts Director of Design Jason Schupbach and Director, Office of Research and Analysis Sunil Iyengar. Enjoy. -IDM)

We continue to be grateful for the level of national discourse that has emerged since the National Endowment for the Arts’ introduction of Our Town, the federal government’s signature investment in creative placemaking. In particular, Createquity has published a number of blog posts that have provided us with valuable feedback. They have also raised insightful questions about the program resources and research needs for an initiative of this size and scale.

So much has been happening at the NEA  – and some of the most vibrant conversations have been based in part on incomplete or out-of-date information – that we thought it made sense to run through our accomplishments and goals, now that we are in the second year of Our Town grantmaking.  (If, after reading this post, you want to know even more about what is happening across the country, please take a look at the current issue of NEA Arts: “Arts and Culture at the Core: A Look at Creative Placemaking.”)



When Rocco Landesman arrived at the NEA in 2009, he put a name on something he saw happening all across this country, from the Little Haiti neighborhood in Miami, Florida, to the cultural district that sprang up around the Museum of Glass in Tacoma, Washington: cities and towns were using the arts to help shape their social, physical, and economic characters. We were rich in anecdotes, but individual communities and organizations lacked the opportunity to connect with others doing similar work.

Like any good producer, Rocco realized that we could not create a community of practice without a name for our shared endeavor, and so the phrase “creative placemaking” was introduced into our national lexicon. Two efforts quickly followed: a white paper by Ann Markusen and Anne Gadwa Nicodemus that defined this sector of work, and a national convening of 40 experts in the arts, community development, and research.  This diverse group launched a conversation about how to measure the presence and impact of the arts in U.S. communities.


The grant program

Both of these efforts helped inform the design of Our Town, which makes grants to partnerships among arts and design organizations and local governments to increase community livability through the arts. By framing the conversation around how communities can use the arts to contribute positively to shared priorities, rather than adopting the more traditional approach of simply stating what the arts organizations would like to do and asking for support to do it, Our Town projects have attracted an impressively diverse range of partners.  These have included social service agencies, botanic gardens, schools, religious institutions, scientific organizations, local businesses, and business improvement districts.

Through two rounds, we have now invested more than $11.5 million in Our Town grants to 131 communities in all 50 states and the District of Columbia. Along the way, we learned an important lesson: creative placemaking is a big and inclusive tent, and in order to make sense of this emerging sector, we need to look at the specific sub-communities it contains.  As grant administrators, we find that it helps to consider Our Town projects in terms of these sub-communities at different points of the award cycle.

From a grant-making point of view, for example, we sort applications into three subsets for review: arts engagement projects, cultural planning and design projects, and projects in non-metro and tribal communities.  This is far from a mutually exclusive / completely exhaustive taxonomy, but for our review panels, this division allows Our Town grant applications to be examined in clusters that share similar opportunities, challenges, and access to resources.

Once the grants have been made, and we move into the mode of grants stewardship, it has made more sense for NEA staff to look at the projects based on the specific activities being undertaken.  This list is bound to change or grow, but to date,  Our Town grants tend to fall into these distinct categories: creating and strengthening artists’ work spaces; asset mapping/cultural district planning; creative industries and entrepreneurship; creating and strengthening cultural facilities; investing in festivals, performances, and other innovative arts programming; reinventing public spaces through creative uses; and the planning and implementation of temporary and permanent public art.

To varying degrees, this taxonomy has subsequently guided our work in evaluating grant projects, conducting national-level research, and creating communities of practice.  Let’s take each of these in turn.


Grant evaluation

At the NEA, every grant program must help achieve one of five outcomes: creation, engagement, learning, research, or livability. The Our Town grants are all measured against livability, and grantees report to us through a final descriptive report form specific to this outcome.

Unlike private endowment-driven funders, the NEA’s budget is allocated annually by Congress.  Despite our name, the NEA does not, in fact, have an endowment, and we are mandated to make our grant decisions anew each year. These facts mean that the NEA cannot commit to funding specific projects over long periods of time, as is the practice with many foundations. (Organizations may, of course, re-apply to the agency.) The ability to make a multi-year commitment to a grantee is the moral prerequisite for doing a multi-year evaluation of that project. So we look at each grantee’s project on its own terms and measure it against its contributions to community livability.

These final descriptive reports allow the NEA to make an evaluation of each grant, but they are also a foundational element in fulfilling our other responsibilities, including both our national research into creative placemaking and our work to build communities of practice.


National research

Following publication of the Creative Placemaking white paper, several organizations and individuals  approached us, requesting cost-effective solutions for better understanding and communicating the value their work added to their communities. Almost all of these groups were more than adept at documenting their work with images, video, and anecdote, but they lacked easy access to quantitative information.

We felt that we could play a key role in building an infrastructure to address this need. In order to better articulate the concept of livability that underpins the Our Town program, we posited a hypothesis that almost any successful creative placemaking project would make a difference to its community in at least one of four ways: strengthening the infrastructure that supports artists and arts organizations; increasing community attachment; improving quality of life; and/or driving local economies.

These particular dimensions of livability emerged from a review of extant literature, consultations with the field, and an initial review of grant applications.  It also grew apparent that these outcomes would be profoundly difficult to measure.  So we decided that an appropriate next step would be to develop a framework of arts and livability indicators that would help the field think constructively about how these concepts might be reflected in data already being collected. The indicators are not intended to measure exactly what is happening in creative placemaking projects; they are instead – as the name implies – meant to indicate conditions on the ground that reflect important dimensions of livability and provide insights into relationships that might exist, thus highlighting areas for further research.

By tracking outcomes that are already publicly reported and widely available, we should be able to provide a reasonably reliable indicator of changes to a community’s overall livability.  Are all or even some of these changes necessarily due to the presence of creative placemaking activities? Absolutely not–but at least they are the kinds of community-wide outcomes that should matter most to people and groups engaged in creative placemaking. By allowing such outcomes to be tracked easily, the indicators system will bypass the need for elaborate and expensive data collection tools and analytics on a project-by-project basis.

How will we know that the indicators we choose are the right ones? Because they will be based on a series of hypotheses soon to be tested in real communities.  For instance, we hypothesize that one indicator of a strengthened arts infrastructure might be an increase in the number of employees at arts organizations. An indicator of community attachment might be the average length that a citizen has lived in a community. An indicator of quality of life might be lower crime rates. And an indicator of a strong local economy might be the number of valid business addresses in a community. Each of these example indicators is based on information already collected and made available by the Census Bureau, the FBI, and the U.S. Postal Service.

We need to test each hypothesis in multiple communities because a single indicator may not work the same way in every place. For instance, at the NEA, we have spent a lot of time internally debating whether “length of commute” is a potential indicator for increased quality of life. Not surprisingly, those of us who live in urban centers think shorter commute times equate with a higher quality of life; and those of us who live in the suburbs and have chosen homes specifically further way from work, feel that longer commute times better correlate with a higher quality of life.

We are working with a team of researchers from the Urban Institute to explore these kinds of nuances for every indicator, testing and validating each hypothesis in multiple use cases and documenting the ways in which a single indicator is and is not an effective proxy. We are also working with and learning from other federal agencies that are similarly building indicator systems from nationally available data sets. It is possible to use an indicators system very effectively, indeed, but it is also all too easy to misuse one – and we want to do everything we can to avoid such pitfalls.

Our team will also assess whether the appropriate data can be accessed at the geographical level of detail we require. Recently, Ann Markusen shared a summary from Arizona State University Professor Emily Talen that was circulated on a listserv for urban planning researchers. This sort of granular investigation into the data available from, in this case, the American Community Survey is exactly the next order of business for our indicators team.  So we are, yet again, indebted to Ann.

If we are successful in creating this indicators framework, then the nation’s arts organizations will have free and easy access to a system that helps them begin to visualize and report on some of the things happening alongside their creative placemaking projects.  From a social science perspective, will these metrics prove a causal relationship?  Again, absolutely not.  But for citizens, funders, civic officials, and business leaders, they will provide a good indication of what is happening.  And when viewed alongside qualitative data from the projects themselves, the indicators may provide sufficient evidence to satisfy stakeholders who seek assurance of the projects’ overall value. Others may wish to know more, and if so, the indicators and the qualitative data lay the foundation for further research and project-specific evaluation.

We believe this approach will help demystify data for organizations involved in creative placemaking.  An organization might be brilliant at developing an outdoor festival that would literally bring art into the center of the public square.  It might also excel at documenting the resulting changes it can observe in the surrounding neighborhood.  But it may not be skilled at identifying and analyzing data sets, and it may not have the time or the funds to undertake an expensive and exhaustive research project.  These organizations are exactly the target audience for our framework, since we will publish – in plain language ­– the data sets that pass our national validation tests and explain how to extricate only the data that are relevant to, in this case, innovative arts programming.

Photograph by Robert Allen, copyright Trey McIntyre Project, all rights reserved.

Photograph by Robert Allen, copyright Trey McIntyre Project (Our Town grantee).


Communities of practice

Taken together, these quantitative and qualitative impacts will allow the NEA to help connect and support communities of practice in creative placemaking.

We have issued an RFP for help in producing documentation that looks at each of the Our Town grantees and asks: what did you set out to do with your project; how did you go about doing your project; how do you know whether you succeeded; and what would you do differently having been through what you went through?

The field has been clamoring for “how-to” information. By combining the final descriptive reports from the Our Town grants, the indicators framework, and the in-depth documentation of each project, we will be able to play matchmaker.

A community that wants examples of successful, federally funded projects can comb through our analysis of the NEA’s final descriptive reports to learn which other communities have succeeded.

A community that would like to make a major investment in public art will be able to parse the in-depth descriptions of public art projects to see what lessons they can learn.

Even prior to all of those resources being available, we have started trying to create cross-community connections by having Our Town panelists share their insights and experiences in a series of archived webinars.  We will do even more of these in the coming months, featuring grantees.


Moving forward

We are really only two years into this work, and are proud of all that we have been able to accomplish. But we are also humbled by the work ahead. The good news is that there continues to be national energy and excitement around creative placemaking, and we are eager for any or all feedback.

We hope that there will continue to be a robust conversation in blogs, on listserves, and throughout the Twittersphere. And we also hope that people will continue to feel free to interact directly with the agency. We are always eager to hear from you at schupbachj@arts.gov or iyengars@arts.gov.

  • If we deconstruct the concept of the NEA’s Our Town program using Mr. Shupbach’s words, imagine trying to explain this idea to a board of directors or making a pitch for this concept to a funding source.
    The pitch goes like this.

    “We are a government run granting program but it’s run like a partnership between us, the arts endowment agency, and everything from social service agencies to botanical gardens. We are going to partner with everything from schools, scientific organizations, business improvement districts, local business, as well as churches. Everyone will be included.
    The program will give money to support “the arts” (let’s just assume everyone agrees what that is). The key difference between this program and our past programs of the NEA is that we no longer directly support artists but we can safely say we support creating and strengthening artists “work spaces”, in addition we will take on the social tasks of asset mapping (let’s just assume everyone agrees what that is) and cultural district planning, encouraging creative industries (let’s just assume everyone agrees what that is) and entrepreneurship. We will create cultural facilities, invest in festivals, performances, and other innovative arts program. We plan to reinvent public spaces through creative uses (let’s just assume everyone agrees what those are) and plan to implement temporary and permanent public art (let’s just assume everyone knows what that is).”

    “We will do all of this on a budget which is less than what it cost for us to buy one new fighter jet and we promise to measure (let’s just assume everyone agrees how that is done) and report the impact of everything we do by assessing what was created (let’s just assume everyone agrees how that will be done), who was engaged by the project, what was learned from the project and how exactly it improved everyone’s livability (let’s just assume everyone agrees what ‘livability’ is).”

    “All these factors of measurability will be based on indicators that are based on a series of hypotheses tested in “real” life based on data from the Census Bureau, FBI, and the Postal Service. All this governmental data, census information, and tested social indicators will be used to assess the quality of Art. (let’s just assume everyone agrees what that is).”

    I think I would be laughed out of the room.

    The descriptions above quantifies and qualifies, assesses and promotes everything politically safe and acceptable, everything entrepreneurially approved, everything socially bright-sided. But it fails to promote, assess and describe what is the Art, where does it exist, and how does it thrive in creative platemaking.

    • OK Richard, I’ll bite.

      Your critique is compelling on the surface, but rests on a few assumptions that I believe are untrue or exaggerated. The first is your assertion that “the key difference between this program and our past programs of the NEA is that we no longer directly support artists.” The NEA has not offered direct support to artists (other than literature fellowships) for many years. This is a relic of the culture wars, of course, and I doubt you’re going to see those reinstated anytime soon. I actually think it’s for the best – I think direct grants to artists are generally a bad idea for reasons detailed here, and it’s a particularly ill-advised role for the NEA given its national scope and the sheer number of artists it would have to consider. (You complain about money not going to artists – just think about how much staff time and overhead it would take to administer the tremendous number of review panels required to adjudicate between thousands of grant proposals in every discipline!) Anyway, regardless of what you think of artist fellowships, in practical terms the program that Our Town is replacing is essentially the Big Read, which still exists but has been displaced as Dana Gioia’s signature national initiative. The Big Read is even less artist-centric than Our Town, so I don’t think there’s much to complain about there. In short, the notion that Our Town is displacing money that otherwise would have gone directly to artists is simply false.

      But is it true for creative placemaking more generally? Another prevalent fallacy is the idea that creative placemaking is a major factor in American arts funding. Sure, it generates a lot of press, but when you look at the actual numbers, the grants for Our Town last year ($5 million) represented just 3.4% of the NEA’s budget and 0.04% of total giving to the arts last year. Add in the grants budget for ArtPlace and all other funders specifically focusing on creative placemaking, and you’re looking at maybe something like $30 million total, which is still a quarter of 1 percent of giving to the arts. Furthermore, the whole point of making creative placemaking the focal point of federal arts policy is to draw more money to the arts. It was chosen strategically at a time when our country’s economy was struggling to make art relevant to the many millions of Americans for whom it is at best an afterthought, including many politicians. We can argue about whether it has or will be successful in that gambit, but I guarantee you that, contrary to the claims in your last comment about a “capitalistic,” “conservative” conspiracy to “control culture,” creative placemaking advocates are sincere in this goal and the belief that it is possible.

      Finally, it’s not like artist workspace, festivals, and the like don’t help artists. I’m guessing from your comment that your space needs are met at the moment. I think if you were in need of a workspace, though, and plenty of artists are, you would really appreciate having one provided for you at preferential rates. Lowering artists’ costs is, in a very real sense, just like giving them money. And when funders support organizations for presenting art, like festivals, artists get paid by those organizations (or at least they should). Have you looked at ArtPlace’s guidelines? They won’t even consider you for a grant unless you can provide the bio of at least one specific artist you’re working with. Now, I do agree that there is a real conversation to be had about whether enough of the money that is raised for the arts, in aggregate, ends up in artists’ hands. But that phenomenon is WAY, WAY bigger than creative placemaking, and you’d hit much closer to the target directing your jaundiced eye at boards and donors of big institutions raising millions of dollars for capital campaigns than creative placemaking funders.

      So if your concern is that there’s not enough money going to artists, your ire at creative placemaking is a bit, uh, misplaced. This is now your fifth comment on Createquity in recent months promoting essentially the same message, often in not very respectful tones. I think that message has had plenty of opportunity to be heard. We’ll continue to publish articles on creative placemaking and measurement opportunities and challenges here, topics that many of our readers find to be of interest. If you don’t, you’re more than welcome to take advantage of the billions of other websites on the internet instead.

      • Ian,

        The fact that we as a people don’t directly support artists (why just writers?) but we do support arts organizations is not a ‘relic’ of the culture wars, it is the ‘result’ of a culture war still being fought today.
        We shouldn’t forget that the NEA had a fully functional system of awarding artist grants for over 20 years. Do you really believe that administering that system of direct support to artists was more burdensome and more wasteful then the countless number of national and state administrators and staff, and real estate that exists today?

        I believe the critique is this: more artists in the past were helped financially to develop the work they want to, to use the money as they deemed best, to pay rent, buy materials, put on performances, publish, go to the doctor, buy food, than under the current system today. And it was done cost effectively, and more importantly, was aesthetically productive.
        The reason we no longer have that system of support for the arts is not because it didn’t work, it’s because of conservative politics.

        I am concerned about creative placemaking because it is the most blatant attempt at directing and controlling what art does in our society.
        Don’t just take my ‘jaundiced’ view on the topic. Claire Bishop in her book ‘Artificial Hells’ (Verso Books) writes about financially concerned societies asking the question -what can the arts do for society?
        “The answers included increasing employability, minimising(sic) crime, fostering aspirations – anything but artistic experimentation and research as values in and of themselves. The production and reception of the arts was therefore reshaped within a political logic in which audience figures and marketing statistics became essential to securing public funding.”
        She suggests that our current focus on ‘creativity’, the ‘creative class’ and I would add, ‘creative placemaking’ blur the distinction between art and “creativity.”
        “Through the discourse of creativity , the elitist activity of art is democratized, although today this leads to business rather than Beuys. The dehierarchising rhetoric of artists whose projects seek to facilitate creativity ends up sounding identical to governmental cultural policy geared towards the twin mantras of social inclusion and creative cities.”

        Any frustration I have, comes from the possibility that in your promotion of creative placemaking and it’s metrics, you might not even have any grasp of what Bishop is referring to. I would suggest that if you really want to develop a system that measures the validity of what artists do, you show a little more willingness to listen to them rather than telling them to go away.

        • All right, now we’re getting somewhere.

          We shouldn’t forget that the NEA had a fully functional system of awarding artist grants for over 20 years. Do you really believe that administering that system of direct support to artists was more burdensome and more wasteful then the countless number of national and state administrators and staff, and real estate that exists today?

          I’m going to go ahead and say yes, I believe this. However, I would like to learn more first about how that system was administered in practice before I can be sure. It certainly is a legitimate question to consider.

          I believe the critique is this: more artists in the past were helped financially to develop the work they want to, to use the money as they deemed best, to pay rent, buy materials, put on performances, publish, go to the doctor, buy food, than under the current system today. And it was done cost effectively, and more importantly, was aesthetically productive.

          OK, that’s a fair claim. And it sounds pretty great for the artists who get that help. But my question to you would be this: how do we decide who deserves it? By almost any measure, the number of aspiring professional artists has increased dramatically in the past two decades, whereas the amount of charitable support for the arts has not. So do we just spread that money out to more people, or do we make our aesthetic criteria more stringent? And a related question: how do you know it has been aesthetically productive? (That’s a serious question, not a rhetorical one.)

          I’m having a bit of a hard time deciphering Claire Bishop’s concern from the short excerpt you gave. Is she arguing that art should be elitist (presumably on the basis of artistic excellence) and that democratization is to be frowned upon? Is it a fair paraphrase to say that you and she would like for art not to be judged by what it does for the public but rather solely by whether or not it’s good art?

      • Ian,

        Good fair questions.

        The early NEA grant panels were composed of people in the field of art, and other artists. In the 1990’s three influential conservative policy organizations began an attack on the NEA and public support for the arts. The Cato Institute, The Mackinac Center for Public Policy, and The Heritage Foundation all came out with positions calling for the elimination of public funding. The first capitulation to the right attack came when the make up of those panels was changed to include lay people from outside the field of art. Those lay people were selected by politicians. Later all direct funding for artists was removed in a bid to save any funding to the NEA.

        But let’s back up a bit to get to your questions regarding who is deserving and how should grants be awarded.

        It was President Kennedy that pursued a vision that was to become the NEA under the Johnson administration. If you read the history and wording of the National Foundation on the Arts and Humanities Act passed into law by Congress in 1965 it was an ideology that believed that artistic pursuit faced unique difficulties that needed both public and private support.
        In those first twenty years artists were awarded grants in the field the same way any other field of study or science selects people for projects deemed worthy, by peer review. Who best to decide what is an important direction, a tradition worth revisiting, or a ground breaking new avenue to award money to than those who work in their field. That system was compromised when people, selected by politicians, who may have had or may have not had any real knowledge about art, were put on those panels. 
Was it a perfect system. No. But would you suggest the public or a panel selected by politicians gets to decide what books are placed in a public library?

        And those early panels did produce results. All one has to do is go back and look at the NEA records to see the list of names of new and established artists that received support and then see how their careers benefited.

        The Bishop quote is from her book on socially engaged art practices. I would say she is definitely not arguing for or against a type of democratization( there is no such thing). I see her as referencing the language that fills the literature of advocacy today (like the NEA white paper on CP) which uses politically generated terms like ‘community’, ‘diversity’, ‘crowd sourcing’, ‘economic development’, ‘livability’. These terms have come to replace the words and concepts we use to use to talk about the intrinsic value of art.
        Bishop also lays out an informative history of how this came to be, beginning with the British New Labor Parties move from liberal socialism to capitalism, the writings of Charles Leadbeater, (who several years later Richard Florida would use for his pop sociology, Creative Class theories) to the political idea today that if art is worthwhile for society it needs to prove itself outside of the ‘intrinsic art’ argument. Today any public art is only good if it can be shown to be good for business, for community, for employment, etc. That is dangerous.

        It is dangerous because there is no longer a governmental system to support the making of just Art. An artist can only receive public money if they do something that can be shown to ‘engage’ people, provides a ‘learning’ experience, and improves ‘livability’ (whatever that is?). On top of this it has to be something that can be measured and confirmed in the NEA’s Final Descriptive Report. Have you tried to imagine filling that thing out?

        To answer your final question, good art has always been good for the public. Good art has always led society, sometimes kicking and screaming, but always to great heights that they could never imagine. What we have today is a system that says the public gets to decide what is good for them. That’s not supporting the arts.

  • John Carnwath

    Thanks for the post. If Schupbach and Iyengar are following this thread, I’d be interested to hear more about how you’re testing the indicators. It seems that the general hypothesis you want to test is something like, “The indicators we have chosen provide reasonably accurate measures of the livability in a given community.” In order to determine whether this is true or false, I imagine you’ll need to compare the indicator scores with some accepted measure of “livability.” Will that be the local population’s subjective evaluation of the livability in their community? I know you said you’re still working on the details, but I wonder if you could share what your general approach will be. Thanks.

    • Hi, John. We plan to use a combination of stakeholder interviews and analysis of extant data collections, at the local level, to validate our national indicators and to determine the scenarios under which they might prove useful, either in current or modified form. A research workplan is being developed with our team from the Urban Institute.

      • John Carnwath

        Thanks for the response. I really appreciate the NEA’s openness about how it’s designing and evaluating the Our Town project. Your plan for evaluating the livability indicators makes sense to me, but I’m not sure it’s what I would call “hypothesis testing.” To me, that term implies an objective test that clearly determines whether the hypothesis is accepted or rejected. In the “combination of stakeholder interviews and analysis of extant data” that you propose, it sounds like the outcome of the test—whether or nor the selected indicators are determined to be good measures of livability—will be negotiated rather than observed. I also don’t really see how the “testing” differs from the process you used to select the indicators in the first place. My critique (intended as constructive feedback more than anything else) is not so much directed toward your plan for assessing the indicators, as it is against raising a false sense (or expectation) of scientific objectivity. I think it’s OK to acknowledge that when we’re dealing with concepts such as “diversity,” “community,” “livability”, and “art” there’s bound to be some level of subjectivity involved. And perhaps acknowledging that will also make the indicators more palatable to those who fear that in the final analysis cold numbers and statistics will overshadow the art and humanity that is at the heart of the projects.

  • Ann Markusen responds to the NEA’s Sunil Iyengar and Jason Schupbach

    Thanks to Sunil Iyengar and Jason Schupbach for explaining the actual evaluation process that each grantee/project undergoes, where their reporting is scrutinized for contributions to livability. It’s good to remind us, too, of the one-year funding limitations of NEA’s appropriations. And thanks for your graciousness about the debate!

    I hope we can still debate the indicators effort. I remain skeptical that by “…tracking outcomes that are already publicly reported and widely available, we should be able to provide a reasonably reliable indicator of changes to a community’s overall livability.” One indicator? Many? The questions I’ve raised about defining livability remain on the table. Questions about whether these indicators are cross-sectional, i.e. one-time and compared with other places, or over time, are still on the table – the paragraph in the NEA response that begins “How will we know…” suggests that some (e.g. increase in # of employees in an art organization) are longitudinal, but others # of local businesses and lower crime rates (than elsewhere? than before?) are unclear in this regard. And, I second John Carnwath’s objection that this is not hypothesis-testing (which would also, as I have explained, need to take into account all the other factors causing change in the area).

    And I am still concerned about the NEA’s use of these indicators. Quoting, “But for citizens, funders, civic officials, and business leaders, they will provide a good indication of what is happening. And when viewed alongside qualitative data from the projects themselves, the indicators may provide sufficient evidence to satisfy stakeholders who seek assurance of the projects’ overall value.” As I wrote before, the problem is that an Our Town initiative might be making a very positive impact on its community but simultaneously other forces—a plant closing, a weather-related or environmental disaster, falling housing prices—may result in negative indicators. The Our Town project may moderate and compensate to some extent for the consequences of these other negatives in the same place, but the indicators won’t show that.

    I remain concerned with what seems an enamoration with data qua data here, as in “…an organization may not be skilled at identifying and analyzing data sets.” It’s not data sets per se that we need, but good complex causal models of change over time, at various spatial scales, which then can be tested with data appropriate to them. The NEA states that they will “publish – in plain language ¬– the data sets that pass our national validation tests and explain how to extricate only the data that are relevant to, in this case, innovative arts programming,” but as any good social scientist knows, we aren’t dealing with a laboratory here where the effects only of arts interventions can be isolated from other factors at work….Unless one does a multi-variate analysis.

    I applaud all the other ways that the NEA team is using to evaluate, and share evaluations among, fundees and future applicants: the descriptive reports, the case study work, the webinars. I am hoping that the NEA leadership team will decide to settle with these and not feel invested in the “indicators framework.” Otherwise, we’ll all be puzzling over and debating the packaged data sets that are sent out to fundees and what they really mean and reflect about each place and each project. Not really where we need to sink our scarce time and energies.

    If I didn’t love the NEA’s and ArtPlace’s creative placemaking initiatives so fervently and continue to speak publicly all over the US in favor of them, and how to do it, I wouldn’t be continuing to engage in the indicators debate!

  • Pingback: Articles of Interest – November 30, 2012 « National Creativity Network()