Philanthropy Evaluation: The Courtroom Approach

Steven Mayer is one of the people behind the Pathways to Progress website, dedicated to social justice philanthropy. Albert Ruesga wrote recently that “metrics based” philanthropy and “social justice” philanthropy are often viewed as two warring camps. But Albert suggested that in fact, “We fail to appreciate how closely united these two camps are in their rejection of philanthropy as usual.”

Today I want to highlight a recent essay by Steven Mayer that I think shows an approach to metrics that has the potential to bridge the divide between “metrics” and “social justice”. Because of the way Steven frames his approach to evaluation, I think he even presents a way to think about these issues that bridge the gap between the participants in the philanthrocapitalism debate.

Steve Mayer:

Our website JustPhilanthropy.org presents many productive avenues for pursuing social justice using the resources of philanthropy. Funders, nonprofits, and potential donors exploring these options frequently ask, “How can we evaluate these efforts?”

Needed: useful evaluation questions

“What is being achieved through this effort?” and “What kind of results are you getting?” are worthwhile questions that must be addressed to be fair to those who support this work. But demands for “measurable impact” and “outcome measures” are inappropriately placed on separate, local efforts; they apply more to the bigger picture, the picture indicated by disparities data. This is not to avoid the questions, but instead to find better ways of answering them. More satisfying data that inform next steps, stimulate innovation, engage participating stakeholders, and make better use of scarce philanthropic capital would come from asking for “evidence of progress” or even “early signs of impact.”

Think courtroom, not science

To appreciate these better questions, try this mental exercise: assume the program you support or operate has been accused of being trivial or ineffective, doing nothing to reduce disparities or improve social justice. What evidence could you provide in its defense? Think of a parade of witnesses testifying from their unique expertise, vantage point, experience, and vested interest. What “portfolio of evidence” could make a case good enough to persuade a jury of peers that this work, when considered in context, is useful and necessary for closing a key disparity?
No less rigorous or accountable

Asking for “evidence of progress” is by no means a diminished demand for rigor. Instead, it frames evaluation in more familiar and approachable terms. Data of all kinds can be considered — numbers, stories, graphs, pictures, records, opinions, artifacts, etc. There is no single “measure” that communicates effectiveness or truth, just as in a courtroom no single witness provides all the testimony. In a court, multiple lines of evidence are entered and judged on their merits, resulting in conclusions that stand tests of credibility and accountability.

I’ve seen lots of frameworks borrowed from different disciplines in an attempt to find a good way to evaluate the effectiveness of nonprofits and philanthropy. I even once suggested that we should turn to movie critics as a model of how evaluation should be performed. But I think that Steven’s court room framework is elegantly simple and captures the way I think of evaluation and impact analysis better than I have ever been able to describe.

The fact is, I think that the best way to evaluate the social sector is via a system similar to investment research on publicly traded stocks. But to most people, this is an alien system that they incorrectly believe is concerned only with quantitative evidence. In actuality, stock market analysis is much more like the idea that Steven presents, “Data of all kinds can be considered — numbers, stories, graphs,
pictures, records, opinions, artifacts, etc. There is no single
“measure” that communicates [the potential of an investment idea].” But most people do understand that in court rooms all sort of evidence is presented and evaluated as a composite whole.

Steven’s framework is brilliant. The one point I would make (I think Steven would agree although he doesn’t make this point explicitly) is that in the court room things musted be proved “beyond a reasonable doubt.” This makes sense because the ramifications for deciding incorrectly are very high. But in the nonprofit/philanthropy world, we simply need to get to a point where it can generally be agreed that some funding opportunities are better than others.

Bravo Steven!