So, Tom Garvey’s takedown of Emily Glassberg Sands’s undergraduate thesis on sexism in theater is pretty much a must-read.

Now the ultra-articulate Sands had been in high gear from the very start of the conversation, but as I got closer to my concerns, she began to power-chatter at a nearly alarming rate. I kept trying to steer the conversation to what I thought should be the central question of her final chapter – were those closed shows actually more profitable than the male-written shows she was comparing them to? (Because profitability, or the lack thereof, is the reason producers close shows.) But every time I tried to phrase this question, Emily deflected it by claiming that I wouldn’t understand her even if she explained her method, that I wasn’t trained enough in statistics to comprehend what she was doing, etc., etc. Finally, I managed to blurt out the full question:

“Emily – do you or do you not have valid data on profitability?”

And suddenly the chatter ended abruptly; a long silence ensued. And ensued. For what seemed like minutes. “Emily? Emily, are you there?” I asked. “Yes, I’m still here,” she answered quietly, but said nothing more. “Can you answer my question?” I then ventured. “I’m still here,” she repeated. And again fell absolutely silent.

It gets more uncomfortable from there, in case you’re wondering. For those of you new to this, here’s the backstory: Emily Glassberg Sands, a May 2009 graduate of Princeton College, recently released the results of her undergraduate economics thesis, a study attempting to identify and quantify gender bias against female playwrights. Due to its topical content and a connection to Freakonomics author Steven D. Levitt through playwright Julia Jordan, her thesis has received quite a bit more attention than most of her peers’ do; for her part, Sands is starting a Ph.D. at Harvard in the fall.

I haven’t yet read Sands’s report, which can be downloaded here, and I’m not sure when I’ll have time to; I already have a bunch of studies on my plate for the Arts Policy Library, and clocking in at 173 dense pages, Opening the Curtain would not be an insignifcant addition to that pile. In the meantime, though, the back-and-forth about the study is piquing my interest quite a bit.

Garvey’s analysis is compelling because, from what I can tell, he is the only journalist who has actually taken the time to dig in to the study and try to understand its underlying methodology. (Frankly, he may well be the only journalist who’s actually read the whole thing.) Not having read the report myself, I can’t vouch for his analysis of her analysis, but he does seem to be asking the right questions. His investigation demonstrates the unfortunate truth about research studies: they’re a lot more subjective than we might like to believe. Everything from the way that the data is collected to the way it is analyzed to the way it is presented can color what the public eventually sees. And all too often, only the researcher is ever in a position to examine the process from start to finish. This is why the Givewell project is so valuable — it takes the time to separate meaningful, rigorous studies from those that merely claim to be. And that’s very much a part of what I hope to do with the Arts Policy Library.

On the other hand, Garvey’s post (which goes on to tell a story about Sands and her thesis that, while plausible, is almost completely speculative) and very aggressive questioning of Sands can be read in other ways, not least through a gender bias lens. He veers between alternately complimenting her on the rigor of the study and assailing her intellectual honesty, ultimately accusing her of perpetrating fraud. (It’s important to note that even if his “gotcha” is valid, it only pertains to one component of Sands’s multi-pronged analysis – and not the part that is getting talked about the most). What I find most interesting to consider, though, is the possibility that this is just a case of “we find what we expect to find.” After all, it’s perhaps not surprising that Garvey discovered things to be skeptical about in the study, considering that he admitted before reading it that he is “hostile” to the notion of sexism against female playwrights because he “often see[s] inferior plays produced by women (Sarah Ruhl, Lynn Nottage, Lydia R. Diamond) who seem to be favored by either the academy or the arbiters of political correctness.” Just as it’s not surprising that Jodi Schoenbrun Carter was so impressed with Sands’s work that she immediately volunteered to help her spread the word as widely as possible, when just two weeks before she had pledged to use her blog to discuss “the on-going struggle [she and her female] peers go through in both the commercial and nonprofit world.”

It’s hard for everyone to escape this bias, the ultimate bias — that our expectations influence our perceptions — and researchers are no exception. The only real way to fight it, as I learned in an ethics class this spring that was steeped in behavioral psychology research, is to ask oneself at every turn, “how could I be wrong?” It is only through naming our assumptions that we destroy their power to lead us astray.

  • isaac butler

    Hey Ian,

    I think there's a lot more criticism due Thomas Garvey for the "purely speculative" portion of his post. He takes a data set that he disputes, and a phone call he has no record of and extrapolates from that to calling someone (and her mentors) frauds on the basis of no evidence whatsoever because she did a study that says some stuff that he doesn't like. It's reckless character assassination of a recent college grad in the guise of counter-argument.

    It's also worth noting, as you do, that it's the second part of the study that people are focusing on and interested in, not the first or the third.

    None of this makes his challenges of the data unimportant. They are, and worth considering, but the rest of the post is really ugly and worthy of our condemnation.

  • Ian David Moss

    You're quite right. Garvey could easily have said what he wanted to say without resorting to nastiness.

  • Thomas Garvey

    Oh, for heaven's sake, Isaac, keep your bullshit to your own blog, please.

    Re: my phone call with Sands. My description of it is quite accurate – I scribbled notes frantically as Sands spoke, and every single line from her that I printed in my blog is verbatim. This is in keeping with standard journalistic practice, whatever Isaac may imagine (and if she feels she's been misrepresented, she can always email me a correction, or comment on my blog). I understand how the dialogue could make someone invested in Sands very "uncomfortable" – it was obvious then, just as it's obvious from the transcript, that not only was Sands deceiving the public but that she was quite aware herself of the deception. But what can I say? That's life. And that's Emily Glassberg Sands.

    As for the data set that I supposedly "dispute" – sorry again, Isaac, but if you read Sands's study carefully (that is, if you ever read it before shooting off your mouth) you'll discover that everything I say about her data she admits herself, in the fine print. That's how she covered her ass, as it were, with her auditors at Princeton: she's free from accusations of academic fraud, because she quietly admits that her Broadway analysis is essentially a simulation of the kind of analysis she would carry out if she actually had the appropriate data. The "truth gap" occurs between the study and the media. Sands elides subtle differences, silently allows people to assume that she had data she didn't have, and yes, even lies, I'm afraid, in the headings of one or two PowerPoint slides (I reproduced one of them). She did this, I speculate, because she was aware that generally she'd be talking to people like Isaac – biased toward her politically, and pretty much ignorant technically – instead of people like me (hostile to her politically, and technically somewhat savvy). In a word, Isaac and his ilk were her mark, and she played them – and continues to play them – brilliantly.

    Now on to you, Ian David. My being hostile to certain overly-praised female playwrights – and simply noting that the prevailing political winds of the theatre world are avidly and openly anti-sexist rather than sexist – does not mean that I am biased against honest statistical analysis, and it's kind of a smear on your part to pretend so. Or perhaps it's just a tired, Z-grade attempt at cultural relativism. Whatevah. But the fact that you state your imagined ideas about my "biases" – ("What I find most interesting to consider, though, is the possibility that this is just a case of "we find what we expect to find.") is both clichéd and slightly embarrassing. It's precisely that kind of thinking that Emily Glassberg Sands is depending on.

    Finally, to your rather discombobulated point that I "veer between alternately complimenting [Sands] on the rigor of the study and assailing her intellectual honesty, ultimately accusing her of perpetrating fraud." There are several parts to Sands's study, just as there will be several parts to my series on her. And one of those parts – her "script experiment" – is, indeed, brilliant; I've said so from the beginning, and therefore I've got to "veer" in my assessment when I'm discussing one part vs. another. I've also said that if Sands had only ended her analysis there, she would have really contributed something to the discussion.

    It's somehow telling, of course, that THIS part of Sands's study is considered "controversial" by so many ditzy identity-politics mavens (like Isaac). But ironically, this is what is actually fascinating about the study – that in her one solid piece of statistical analysis, with no doubts about her data or methodology, Sands finds not actual sexism, but a faith that sexism must exist. A finding which I think you and Isaac could be seen to exemplify. I'll be discussing that in future posts on Sands. So read till the end before you start throwing stones.

  • Ian David Moss

    Mr. Garvey,
    You can call me Ian. Since I've already pretty much stated everything I have to say about your original post, let me just repeat my earlier comment: you could easily have said what you needed to say without resorting to nastiness. You seem to be confusing the nature of speculative vs. assertive writing. Your writing about Sands's motives is assertive, even if you now admit that you were speculating. My writing about your motives is speculative, in that I very explicitly talk about the "possibility" that you wanted to find certain things in her study, so you looked for them. I don't know that this is the case, and freely admit as such, but I was able to use it as an opportunity to make a broader point which I felt and continue to feel is relevant to the discussion. After all, we don't have to be biased against honest statistical analysis in order to evaluate the meaning and import of that analysis differently.

    But hey, if you want to destroy any chances of people taking your hard work seriously, by all means, go on hyperactively calling people names in response to the slightest criticism of it.

  • Thomas Garvey

    You know, Ian, I suffer neither fools nor frauds gladly. If calling a spade a spade to you is "nasty," then so be it. But by ancient agreement, poseurs are a legitimate target of ridicule. Hence my tone toward Emily Glassberg Sands.

    And I think you misread my ironic use of the word "speculate" in my post. I wouldn't call anything I've written about Sands or her mentors speculative in the ordinary sense of the word. It is instead, as you yourself said, plausible – indeed, it is the common-sensical interpretation of what has occurred. People can spin the facts in some other direction, I suppose, but the facts – as admitted by the players themselves – are available all over the web. I alter none of those facts – hence, there is plenty of evidence, whatever the silly Mr. Butler may say, for my interpretation of events. The onus is rather on Mr. Butler to prove that my interpretation is unlikely or impossible. Good luck with that!

    And I do want to continue to counter your insistence that there's some validity to your idea that I "looked" for "certain things" in Sands's study due to "bias." Even your earlier quote from me is parsed to avoid its most pertinent content:

    "But I'm rather curious just how much Freakonomics-style analysis Sands can really apply to the case of female playwrights if they're indeed so under-represented on the stage. Where has she found a large enough sample to back up such claims as her assertion that "plays written by women sold almost one quarter more tickets per week than those by men, earning 18 percent higher grosses weekly"? Particularly given that she claims these numbers are based on Broadway plays, where Theresa Rebeck claims women are barely represented?"

    As you can see, my "bias" is essentially the following: I've got the taste and brains to see through skillful, but essentially second-rate, playwrights like Nottage and Rebeck. And from my background in statistics, I guessed immediately at the problems which, yes, plagued Emily Glassberg Sands's study. This isn't "bias" – it's worldliness.

    But I think your point about bias does, actually, have a valid target – yourself. Did you not assume that what you learned last spring in college was easily applicable to me? Yes, I think you did. So the best you can do is conjure a kind of orbiting pair of "biases" – yours and mine; that is, until you readily, rather than gingerly, admit the facts about Sands's study. And perhaps realize that what they taught you in college had its own "bias," too.

  • Ian David Moss

    I'm not especially interested in continuing this discussion, since you seem more invested in one-upmanship than having a calm and reasonable dialogue, so I'll just leave you with this thought.

    You said, "But I think your point about bias does, actually, have a valid target – yourself. Did you not assume that what you learned last spring in college was easily applicable to me? Yes, I think you did."

    Well, gosh golly gee, you sure got me there. In fact, you're so right that that's exactly what I said in the post.

    "It's hard for everyone to escape this bias, the ultimate bias — that our expectations influence our perceptions"

    Every decision we make comes with some degree of uncertainty. The question is not whether there is uncertainty or not, because there always is; the real question is the amount. Most of the time, we estimate this uncertainty in our minds without really thinking about it. And most of the time, we peg this uncertainty level as too low — especially, as it turns out, if we're men. Asking ourselves that question, "how could I be wrong," as I learned in graduate school, is the only reliable way (among a number of alternative methods that have been tested) to neutralize this tendency. If you like your statistical analysis honest, Mr. Taste and Brains, I suggest you do yourself a favor and check out the research literature on overconfidence bias. I think you'll be glad you did.