A colleague of my made the argument to me yesterday that Forces for Good was flawed because the twelve organizations from which the Six Practices of High Impact Nonprofits was derived have not been proven to have achieved impact. This is true. The methodology used by Crutchfield and Grant did not employ double blind studies or even similarly rigorous analysis of the selected nonprofits’ programs effectiveness. Instead, the authors used peer surveys, expert interviews and the authors own in depth research. Why didn’t the Crutchfield and Grant simply use twelve proven high impact nonprofits as their research set?
Because no one can find them.
Philanthropy loves to talk about proven high impact nonprofits. We love to talk about “programs that work”. But the fact is, very, very few nonprofits have ever gone through extensive analysis that has proven that their programs have impact. Have you ever noticed that I talk about Nurse Family Partnership a lot? That’s because it is essentially in a class by itself. Other proven high impact nonprofits exist, but you’d have a hard time getting a consensus on twelve that Crutchfield and Grant could have used in their book.
Proven high impact nonprofits appear in lots of theories of effective philanthropy. But they aren’t seen very often in the wild.
So we have two options. We either spend our time running studies trying to prove whether nonprofit programs actually work or we follow the lead of Crutchfield and Grant and create a working model based on a set of plausible assumptions and get to work building the nonprofit organizations that we think are best positioned to create programs that will one day be proven to work.
Of course we still need to do evaluation. Of course, we need to look at claimed program effectiveness with a skeptical eye. Of course we need to demand that nonprofits constantly evaluate their programs in the most rigorous way to determine if they are working. Of course we need to fund outside evaluators to do the same.
But we also need to get on with the business of building a world class ecosystem of high performance nonprofits.
So I agree with my colleague’s critique. Forces for Good’s set of studied nonprofits have not been proven to be high impact organizations. The set of best practices distilled from those organizations might be incorrect if it turns out that some of the selected nonprofits’ programs were not particularly effective.
But guess what? Warren Buffett doesn’t run double blind studies to see if the companies he’s invested in have “proven” profit centers. We happen to know that lots of for-profit companies looked like they were doing just dandy until last year. Their profit centers turned out to be a sham, based on false assumptions and they blew up.
Warren Buffet was invested in some of those unproven companies.
He’s still the richest person in the world and he’s not changing the way he invests.
I wish we lived in a world populated with proven high impact organizations. I wish we lived in a world with proven stock trading systems that could guarantee a market beating rate of return.
But we don’t. Let’s get on with the business of philanthropy. Philanthropy’s job is providing capital to high performance nonprofits and refusing to fund low performing nonprofits. It is our job to invest in the best possible organizations and provide them the resources they need to grow and enhance their impact. It is a messy business. We’ll never know for sure if we’re backing the right organizations. But that’s the hand we’ve been dealt.
Even when we do “prove” that a program works, we’re not talking about proof of a mathematical or physical property. Proof in nonprofit effectiveness means that rigorous studies have been performed and that should give us a lot of confidence. But even a quick search of Google returns stories like this one on natural-selection studies:
Genetics in Japan have demonstrated that several statistical methods commonly used by biologists to detect natural selection at the molecular level tend to produce incorrect results. "Our finding means that hundreds of published studies on natural selection may have drawn incorrect conclusions."
We live in a world of uncertainty. The way forward is not to demand proof before moving. It is to getting moving this very second while at the same time constantly striving to check our assumptions. And when we find our beliefs were incorrect? Turn on a dime and barrel forward on the newest most probably course.