Albert Ruesga, who blogs at White Courtesy Telephone, was a panel member at the recent Metrics Mania debate at the Bradley Center for Philanthropy & Civic Renewal. Albert posted the text of his comments today and they are outstanding.
At a time when we are seeing a growing backlashing against “philanthrocapitalism”, it is interesting to look at what is being grouped under that term. For many people, “metrics” and the push for more “evaluation” of philanthropy is an unwelcome element of a “business-like” approach to giving. I believe that evaluating nonprofits and philanthropy in general is necessary for a the Third Sector to become a high-performing, high-impact driver of social good. But as I wrote last week, I think that much “evaluation” takes a scientific approach to measurement that is borrowed from the hard sciences, while the lessons of the liberal arts (under which investing and financial markets should be categorized) are more appropriate.
Albert, I think, would agree. He writes:
Measurement and evaluation, when done properly, are not just a bit of value-added for philanthropic or nonprofit work, they’re absolutely essential. Only a fool would disagree with that proposition.
But here I mean not just the kinds of formal evaluations described by Gary Walker in his essay, but informal evaluation as well: the kinds of course corrections we naturally make when we embark on a project, take a false step, and adjust what we do accordingly. Evaluation is not and should not be the sole province of the highly compensated consultant. We evaluate all the time; our own eyes and ears notice things the most astute consultant will never notice; and we’ll often be our own worst critics.
Now here’s where the metrics schmetrics comes in, perhaps: More nonsense has been spoken and written about evaluation than about any other subject in philanthropy. The number of people practicing evaluation without a license and without a proper scientific and philosophical grounding in the subject is, in my view, a scandal. Worries about evaluation, engendered in part by logic models the length of whale intestines, have become the math anxiety of the philanthropic world…
…I want to make clear that I’m not in the least anti-evaluation. As I’ve written elsewhere, I’m concerned that we tend to seek a kind of scientific or moral certainty from a formal evaluation where none exists. The questions that funders most often bring to an evaluator—Was this program worth our $25,000 investment? Should we continue funding it?—are questions only they, the funders, can answer. Say we measure a 25 percent drop in the truancy rate for a hundred kids in some program, and a 25 percent increase in their test scores. Is that worth $25,000 to you? Each donor needs to answer that question for him- or herself. As donors we will never be absolved of our responsibility to use our good judgment.
“Evaluation… the math anxiety of the philanthropic world.” What a great line. Evaluating nonprofits and social good in general is not about math. When I advocate for a financial markets-type approach to evaluation, I am not calling for number crunching. I wrote about this concept in January and will repost my comments from then rather than repeating myself:
Economics is often called the “dismal science”. I know that many people think that finance is boring. But the vision of financial markets as nothing but numbers and spreadsheets does not capture the reality. Do investors buy stock in Apple because they spent hours and hours processing spreadsheet calculations? No. While at the end of the day, buyers of Apple stock believe that the return on capital being generated by the company will make for a profitable investment, the information they use to determine that are not just numbers. The way in which Apple has captured the imagination of the consumer, (an intangible piece of data that cannot be added to a spreadsheet) is by far the most valuable asset that Apple has and it is a major reason why investors have flocked to the stock.
Have you ever watched CNBC, the news channel of the financial markets? It is far from some kind of spreadsheet crunching lecture. Every day, investors or all types come on the show and make passionate arguments for why certain companies are good investments. While numbers and calculations underlie much of their thinking, it is the story, the human story of the companies they discuss that take center stage.
Warren Buffet is widely considered the best for-profit investor of his generation. Does he sit in a corner office reading a spreadsheet the way that Phil suggests? The quote below is from noted investor Whitney Tilson (Tilson is a huge fan of Buffet and a fellow columnist of mine at the Financial Times):
If the future were predictable with any degree of precision, then valuation would be easy. But the future is inherently unpredictable, so valuation is hard — and it’s ambiguous. Good thinking about valuation is less about plugging numbers into a spreadsheet than weighing many competing factors and determining probabilities. It’s neither art nor science — it’s roughly equal amounts of both.
The lack of precision around valuation makes a lot of people uncomfortable. To deal with this discomfort, some people wrap themselves in the security blanket of complex discounted cash flow analyses. My view of these things is best summarized by this brief exchange at the 1996 Berkshire Hathaway annual meeting:
Charlie Munger (Berkshire Hathaway’s vice chairman) said, “Warren talks about these discounted cash flows. I’ve never seen him do one.”
“It’s true,” replied Buffett. “If (the value of a company) doesn’t just scream out at you, it’s too close.”
Taking liberties with Tilson’s quote, I would argue that donors should not “wrap themselves in the security blanket of metrics” because “the lack of precision around measuring the impact that nonprofits achieve makes them uncomfortable.”
World-class investors do not sit in their office crunching spreadsheets all day. Neither should world-class donors. But the underlying logic of both should be that of achieving the highest return on investment.
Nothing proves the point that valuation is more art than science in financial markets than the Bear Stearns collapse. Until we figure out how that happened, I’m not going to line up in favor of a “market” approach for valuing nonprofit investments either.
But I do think Albert is on to something when he says the bigger question that donors have to answer is “is this worth $25,000 to you?” Sometimes factors such as the cost to society for not investing in programs that reduce truancy or raise test scores — to stay with Albert’s example — provide useful guidance on whether that $25,000 investment is warranted. Also, comparative data that shows the difference between one program’s results and another is also helpful in choosing which competing program merits investment, especially when the program that does a better job than the other costs more. The better results might justify the higher cost of the alternative model.
Focusing the question on the value of something “to the donor” is critical. This is the same system in place in financial markets where each person makes their own determination of value and then acts.
My point with financial markets is not that they value things perfectly, but that they allow capital to flow to the best companies and deny capital to weak companies. The Bear Sterns event is complicated, but it is also an example of how quickly financial markets can absolutely turn off access to capital to companies that fail to satisfy investors. Imagine if it was found that the Red Cross was operating in a way that was intolerable to investors and that by the end of the week donations had fallen to zero.
As I said, Bear Sterns is a very complicated event. In no way am I pointing to it as favorable outcome in financial markets. But it is an interesting example of the speed with which financial markets can turn on and off the flow of capital.
Disclosure: Nothing in this post should be construed as investment advice.
Thanks for the kind words. One of my favorite examples of counter-metrics investing comes from my time working at a community foundation in Boston. We funded a group that worked intensively with a small number of homeless women over the course of several years. There was no self-selection in this program: these were women who, because of the number and intensity of their problems, failed to qualify for the other programs available to people with addiction and mental health issues.
The cost per client for this program was on the high side, and only a small fraction of the clients (20%? 30%?) were able to make the transition to steady employment or permanent housing.
By the standards of traditional metrics, this program should not have attracted many donors. It nonetheless did. Donors could see with their own eyes the quality of the care delivered to people who otherwise might simply have fallen off the earth. That these women received some care, now — whatever the impacts might ultimately be — was important to donors.
What kinds of values are we measuring here?
Aspects of quality are very hard (though not always impossible) to measure. It’s hard to attach a number to the fulfillment of a social responsibility, to extending a hand to someone who might be least able to benefit from our help. Social networking theory enables us to calculate measures like closeness and flow in a group of advocates, say, but will it tell us when we’ve got all the right people lined up for all the right tasks for a given campaign?
I appreciate it that you advocate a middle path on the metrics issue. That’s the wisest course, in my view.
Great example Albert. It seems to me that we should measure that which we can and we should evaluate qualitatively that which we can not measure. But we should always seek to understand and support good execution, not just good causes.
It seems to me a recurring theme of the metrics mania debate is that it involves both art and science. If the “art” position is that non-profit value defies objective analysis, and the “science” is that non-profit worth can be reduced to equations of input and output, I expect I’m somewhere in the middle.
This is surprising, since in the interest of full disclosure, I should say that I am co-founder of socialmarkets.org, where donors “invest” in nonprofit projects based on their SROI (Social Return On Investment.) This sounds like pretty hard science, but there is actually quite a lot of room for art to soften the edges.
The stock market analogy already seen in this thread is spot on. Apple’s stock price is *informed* rather than *defined* by the financial science that slices and dices its cash flows. The beauty of a market is the marvelous job it does boiling down a large, complex set of valuation inputs into a single output called price. This number is useful on both an absolute scale and relative to other offerings in the market.
In a testament to the wisdom (or lunacy) of crowds, Apple’s stock price reflects the collective opinion of financial analysts, status-conscious teens and everyone in between. The potential to harness the same power to “price” The Red Cross or your local community foundation seems both possible and useful to me.
There are plenty of donors looking for a “best-bang-for-the-buck” (i.e. most SROI) approach to nonprofit investment, and right now there is not much useful data out there for them. The success of sites like Charity Navigator are a testament to the need for metrics, but they only tell potential donors about what nonprofits spend, rather than what they accomplish. Surely we can do better than that.
Those with a less scientific approach may not find metrics like SROI as compelling, but still potentially useful. Consider Albert Ruesga’s story of the high-risk, low-return homelessness project that he presents as an argument against metrics. This is where the difference between the sectors becomes significant.
For starters, unlike the for-profit model, there are often donors willing to invest in hard-luck nonprofit cases – as they ultimately did in Mr. Ruesga’s example. More importantly, since nonprofit metrics is still a new field, we can – and should – redefine the notion of *return* to more accurately capture the total social value being added. That would be the most constructive cross-product of art and science in this space: a more artistic approach to the science of metrics.
Jeff, you’ve made a really good point that I haven’t seen discussed before. Exchanges are very good at assigning value to non-tangible aspects of a company. Exchanges do a good job of valuing a company’s “brand” or their positive corporate culture. These are things that financial metrics have difficulty showing, but that clearly have real value. It hadn’t occurred to me that one of the advantages of an exchange is the explicit ability to assign value to non-tangible assets.
Jed Emerson is probably emailing me a paper he wrote 10 years ago outlining this aspect of exchanges as I write this!