The bottom line on this discussion is that there will never a universal metric to measure impact, effectiveness, or whatever term one prefers. Can metrics be used improperly? Yes. Can they be manipulated? Yes. Can they be difficult to quantify and communicate? Yes. These possibilities do not outweigh the benefits of a well designed and implemented program of demonstrating a nonprofit’s value.

Our position in this confusing arena does not come from a philosophical or academic perspective: it comes from the trenches of helping nonprofits get the funding they deserve. Those nonprofits that make it easier for funders to connect the dots to their respective outcomes deserve to have capital flow to them. Those that choose not to go down this path do so at their own financial risk.

If the cost of a 1% increase in the ratio of high school graduates is NY $1M, and the cost of a 1% increase in live births in Uganda is $1M (both hypothetical measures of course), I think that donors would actually able to make an informed choice about how to spend their dollars – depending on their personal convictions. Without this type of universal measurement, it’s all just a shot in the dark, right?

Leyla Farah,
Cause+Effect – Public Relations with a Purpose

The first is that there are plenty of potentially relevant metrics with which one could back up one’s a claim for each work’s superiority: their longevity in years, the number of universities that include them in introductory freshmen humanities courses (as a proxy measure of their centrality to our cultural canon), a RottenTomatoes.com-style survey of critics. I can even imagine poor grad students counting allusions to them in last year’s bestsellers.

Relying solely on any one of these potentially valid measures, however, would obviously leave you wide open to criticism for the flaws of your methodology and the limits of the analysis. To construct a strong argument for your preferred choice, one could use both the metrics and qualitative measures. Same goes for nonprofits – the measures are neither perfect nor complete, but that is not the same as nonexistent.

I think the other point is the difficulty of comparing apples and oranges. Let me reframe the question as “Paradise Lost” work of literature vs. “Gone with the Wind” work of film. Both are widely-considered seminal works in their mediums. It’s not hard to imagine metrics, like those above, that could easily distinguish each as a leader within its respective medium. It is much harder, however, to compare them very convincingly across mediums. An author and a film buff might reach very different conclusions about which one matters more in today’s culture. Their distinctive values and tastes will influence that decision.

The same, I think, is true for nonprofits. Too universal a measure like “average cost of impact” might not be helpful for identifying whether a great afterschool program in New York or a great community health program in Uganda is better. The costs and the measures of impact are on different scales. But metrics certainly might help you identify each within its field as the seminal nonprofit. From there, one’s values and tastes might be expected to guide your choice.

But Phil, couldn’t you write up a paper comparing and contrasting Paradise Lost vs. Gone with the Wind and point to why you personally thought one was better than the other? Or at least the ways in which each one is different and the strengths and weaknesses of each? That’s what I’m asking for. I think the cop out is saying “Gone with the Wind is about such an important topic” (which is the same as a nonprofit citing the importance of their cause) or “I just know it is great, you have to read it yourself” (which is the same as many nonprofits just saying that they “know” they are doing a good job or telling you to come volunteer so you’ll get it). Evaluation does not require metrics but however it is done, it is critical.

This might sounds strange, but I don’t really believe we will ever find a simple set of standard measures. Even in the for-profit world, where everyone is producing an identical outcome (namely money), there are many approaches to analysis. You can’t say, Well Cisco’s “return on invested capital” is higher than GE’s, so Cisco is clearly better. It just doesn’t work like that.

I think the frame work of “average cost of impact” is correct, but it will not ever be something that can be accurately measured (but that’s OK, humans still measure many things that can’t be accurately measured, is War & Peace a better book than some trashy romance novel? Can you show me some stats that measure this or come close to proving it?).

I think this whole debate is key to the development of the sector. Glad to have you in the conversation Leyla.

Beyond the obvious – revenue, overhead, donor retention ratios, program growth, etc. – I’m interested in what you and the rest of the TP readers would like to see included in a standard set of metrics?

One item I’d offer: a measurement of “average cost of impact” – in other words, the organization’s total budget divided by the total number of people (or animals, or acres of land) it’s benefited within a specific time period. That metric would (1) force each organization to provide a definition of how it helps people (etc.) – and (2) force it to account for all the costs associated with providing that help.

Other thoughts?

Leyla Farah,
Cause+Effect – Public Relations with a Purpose