Holden is outraged. He believes strongly that metrics are critical. In response to Phil Cubeta telling him “Only a madman would dream of measuring the good, all good, on a single universal scale.” Holden responds:
Will you admit that there are good things about modernity? … Forget about a metric to measure "GRAND TOTAL GOOD ACCOMPLISHED" – how about one that measures "Number of indebted people who were able to become self-sustaining thanks to a loan?", and another that measures "Number of diarrhea-ridden children whose lives were saved by medication? … Measuring good is incredibly hard, but more viable if you keep in mind why the numbers are there and what their limitations are. It can be human, and more importantly it allows a form and scope of teamwork that would be impossible without it.
Albert Ruesga responds to Holden:
I think some metrics accomplish what you say for some projects. When a youth development program starts costing $25,000 per young person per year, might it not be better to sock that money away into college accounts for the participants? I don’t know. So many axes of value to compare — quality of instruction, intensity of intervention, paucity of other opportunities: how do we assign weights to these? We can never compare two youth development programs the way we compare two cans of soda: they differ in too many respects.
And Holden disagrees with my movie critic analogy:
The fundamental difference between charities and consumer products such as movies and restaurants is that the mere fact of being a charity’s customer doesn’t mean you know ANYTHING about it. If we want to separate the good charities from the bad, I simply see no way around doing difficult, costly, painful, formal evaluations.
I think that Holden is right, that as a customer/donor of a nonprofit, you can’t simply judge their effectiveness based on your experience. But that’s not what I’m suggesting. You can “experience” every aspect of a movie, simply by watching it. To judge a nonprofit, you need to do much more proactive analysis. Talk to the board, to staff members, the people served, and competitors in both the nonprofit and for profit space. Part of this qualitative analysis is going to include researching certain metrics relevant to the nonprofit in question. I think examining how cost effective a nonprofit behaves is critical, but the evaluation needs to qualitatively take into account the industry the nonprofit is operating in.
Laura Quinn, the founder of Idealware (you can read a background interview with her on Beth Kanter’s blog), left me a comment saying:
I have deeply conflicted thoughts about this. One the one hand I agree and would go much further than you have about administrative expense ratios. These are not only ineffective, they’re *damaging* to the sector…
On the other hand, if you can’t measure in a way the good you’re doing in the world, I would question what good you’re actually creating. This doesn’t mean sum it up into one tidy little figure – evaluation is a difficult science, but far under-used. Many in the nonprofit sector have a tendency to operate based on an instinct that they’re doing good, and these instincts can definitely be wrong…
My point is that it’s tempting to rely on a “gut check” for how useful something is, but guts can be wrong, and they’re certainly drawn to flashier programs rather than things that are just quietly working…
My training is as a Chartered Financial Analyst charterholder. Every day I analyze for profit companies as investment prospects for my clients. We examine an untold number of metrics, but we also learn the “story” of the company. Best Buy isn’t just a company with a 5.48% operating margin, it’s a company that believes deeply that selling consumer electronics is becoming a service business, not just a retail business. Costco isn’t just a company that took in $62.42 billion in revenue over the past four quarters, it is a company that searches high and low for the very best quality/value propositions to offer to their customers and believes that paying their employees much more than their competitors is critical to the company’s profitability.
The metrics are important, but you can’t find the best nonprofits (or for profits) by simply measuring metrics. You need to know the story, the core beliefs that are driving the company’s behavior. Just as it can be far too tempting to buy the “hottest” stocks, it can be tempting to assume that the nonprofit that throws a great charity ball or that has the cutest pictures of kids on its website is a good place to donate. This is Laura’s point when she warns against “gut checks”. But on the other hand, you can’t get lost in the metrics, you’ll never find a magic number or set of numbers that will tell you what you need to know.
3 Comments
Your point about movies is clearer now, and I agree with you in content – just not in emphasis. There are certainly many people who are tilted toward over-measurement, and it is worthwhile to caution against this, as you do. But MANY more people are tilted toward under-measurement. The status quo for 99.9% of donors is to go entirely by the story, with no metrics.
I agree that both the story and the metrics matter. But the latter are what desperately need more thought and focus, for this particular sector at this particular time.
Actually there are existing quantitative rating systems (although, as we have agreed, they use poor metrics). But there are no formal qualitative systems. I’m calling for a professional qualitative system similar to equity research. Not for individual donors to do their own evaluation.
These analysts would use metrics of course, but they would get to know the “story” as well.
Yes, existing metrics are terrible, because not enough thought (good thought incorporating emotion, not just mathematical thought) has been put into them. That’s exactly why we need more focus on metrics – we need to do a better job with them.
I think I now understand what you mean by a formal qualitative system. You’re saying that reviewers ought to publicly publish ratings, that are based on their subjective take on the whole picture rather than a formula. As you say, you should use metrics in forming ratings, but both are valuable.
FWIW, the current GiveWell site uses exactly the type of rating you describe (1-5 subjective ratings for “Strategies and Activities,” “Relevance of your dollar,” etc.)
As with movies, I assert that these numbers would be worthless without the ability to see the rationale behind them. We provide that as well.
I am not saying “problem solved.” Our scope and the quality of our information are a shadow of what they can and should be. And I’d love to see others – foundations, advisors, donors – taking the same approach.
(By the way, “professional” seems irrelevant – again, as with movies. What’s important is that the ratings be understandable and publicly available. I find IMDB movie reviews more useful than NYT ones …)