In his guest post, John MacIntosh of SeaChange Capital Partners made an important point:
The tools for evaluating for “impact” and “performance” come from different disciplines. “Impact” is a concept from social science where ideally we define the treatment, develop measures of impact (wages, employment rates, test scores, etc.), identify a comparison or control group, worry about selection and omitted variable biases, organize for large Ns, and hope for p-values above 2.0!
“Performance” is about leadership, culture, management information systems, accounting, unrestricted net assets, opportunities for donor engagement, and the like. Few people I have met are equally comfortable in these two domains; very, very few are masters of both.
I would argue that philanthropic institutions are currently geared towards thinking of themselves as impact researchers instead of performance investors. As I’ve tried to make clear, the goal is the same. Both disciplines are needed. A high performance organization that implements ineffective, poorly researched programs will fail to achieve impact. A poorly performing organization that tries to implement proven programs will fail to deliver them with fidelity and/or fail to grow.
But what would happen if funders thought of themselves primarily as performance investors and relied on a mix of internal, external and independent researchers to prove program effectiveness?
I think we’d finally be able to lay real claim to the idea that we are social investors. Calling a grant an “investment” would no longer be just the trendy way to describe what’s always been done. Grants truly would be investments because they would come in reaction to the capital needs of high performance organizations.
Remember the guest post by Kathleen Enright, the head of Grantmakers for Effective Organizations, during the Council on Foundations conference? It was titled The Need for Speed and reflected on the urgency of solving some of our most pressing problems:
This theme of speed and urgency reminded me of an observation that Ami Dar made when he spoke at the GEO/NFF Money Matters conference. He observed that whenever he’s in foundation offices, he never sees anyone walking quickly. The comment drew laughter from the crowd, but the point is an important one.
We need to start asking ourselves what it will take to infuse the kind of urgency in our own work. As it stands, our current modes of operating may get in the way of our ability to play an important role in solving our most pressing problems.
But speed isn’t always a good attribute. I certainly wouldn’t want someone researching the long term effect of multisystemic therapy on youth with behavioral issues to “rush” the study. I wouldn’t want to go into a social science research outfit and find people running down the halls. But investors do appropriately move at high speed. The timing of delivering capital is critical. If an organization needs money today, 9 months from now might not work.
Interestingly, for-profit capital providers work under this model. The vast majority of them focus on understanding companies. The definitely consider whether a company’s products and services “are effective,” but the best investors (like Warren Buffett) tend to assume that the company’s management knows what they are doing and trusts them to make the right decisions about whether to change their product mix or not.
These for-profit capital providers also depend on outside researchers at universities, independent consultants and firms with specialized expertise in areas like retail or biotechnology. The model does not downgrade the importance of academic research or systematic evaluation. It just asks the capital providers to play the role of investor instead of researcher and positions the role of researcher in the hands of other players in the system.
To be clear, there are also specialized for-profit investors who focus on only one niche market (oil and gas companies or media companies for instance). These groups do hire researchers and it is their industry expertise that sets them apart rather than their company evaluation process.
I’ve often heard people say that philanthropy is the research and development (R&D) of the social sector. But I wonder if that is wrong. Maybe we have a different role to play. Maybe we don’t have to invent anything. Maybe we just need to be social investors, providing capital to the most promising nonprofit organizations.
In a comment yesterday, reader Hildy Gottlieb worried that I was pushing a system “of rewards” where “funders decide from the top down what is best for a whole community".” But I actually think that my model aligns the interests of funders and grantees because suddenly rather than directing nonprofits to try different interventions and dangling money with the requirement it be used to the funders satisfaction, funders become deeply interested in providing capital in a way that best helps a nonprofit build a strong organization. In addition, we know from the for-profit market that while capital providers hold the power over early stage companies, it ends up being the investors who chase the highest performing companies and compete for the opportunity to provide them capital.
Nonprofits aren’t laboratories for funders to experiment in. The best are high performing social good generators. It is the job of funders to be the best at finding these organizations and helping them grow. We should take pride not in designing programs, but in being the smartest funder who always seems to find the best nonprofits before anyone else and whose portfolio of grantees thrive and change the world.
I have been following the thread of discussions about performance vs.impact and I have done a bit of research on all of the various ways organizations have proposed to evaluate these :measures”. Separately I have had some discussions with people involved with foundation and funders who are interested in having some sort of useful approach to comparing non-profits. I have been working on the concept of taking the approach that analysts use in the for-profit sector and coming up with a comparable framework that would apply in the non-profit sector. I believe that funders accept the fact that there are all sorts of theoretical defects in proposed ways of evaluating non-profits but are frustrated that the lack of perfection seems to stand in the way of a useful model that will provide good but not perfect results. That in fact is what happens in the for-profit sector. At present Charity Navigator and one or two others seem to be the only game in town.
I think that’s right Cullen, but there’s lots of forward activity on this. See New Philanthropy Capital and the Alliance for Effective Social Investing (a group I’m a part of).
Sean makes an interesting point when he says “maybe we don’t have to invent anything.” Since lots of evidence-based programs already exist then I would think a high performing org would look to these proven programs when setting out to achieve their mission. The high performing org would then implement the program with fidelity to the model. Along the way the high performing org may identify ways to improve the program and make appropriate adjustments.
We already have a pretty good idea about about what works (take a look at Child Trends). Maybe we don’t need to invent anything new – we just need to fund high-performing orgs that can execute.
I think that’s right Jeff. Of course, high performance orgs will have to keep inventing and researchers will keep having to study the effects of old and new interventions, but I think philanthropies big contribution can be intelligently funding high performing organizations and providing them the fuel they need to really move.
Great post, Sean. It really would be helpful if funders would identify high performing organizations and fund evidence-based programs at substantial levels over a number of years.
That way, impact would be likely for the clients, and the nonprofits would save resources that would otherwise go toward more fundraising. It seems like a waste to force organizations that are making a difference to always raise money because foundations want to fund different nonprofits every year.
If it works, keep supporting it! Why move on to something that is not known to work when so much need can be met with a well implemented model known to work…?
As both a fundraiser and a program evaluator, my biggest frustration with our sector isn’t whether foundations/agencies are doing one kind of evaluation or another . . . I find most are unaware of the need to do any sort of evaluation, do not read current research in the area in which they make grants/operate programs, and do not make serious attempts to understand best pracices. If a way could be found to 1) inform foundation and agency staff of current research and 2) use that research in any way to modify program offerings, that would be a huge step in the right direction.
Thanks for the note and it does in fact appear that the Alliance for Effective Social Investing may be on a viable track. My only question is whether any of this work addresses the grass-roots problem faced by a large number of foundations – which is to find a way to reliably, easily and efficiently rate organization eligible for grants such that they can make risk adjusted “portfolio” investments in a manner that works well in the for-profit sector. Achieving this would require a 1. A comprehensive model for scoring non-profit performance (including whether they have processes for evaluating impact) 2. A way for nonprofits to apply for and provide input to the scoring process and 3. An independent industry-wide “audit” process to selectively verify that the approach is working so that foundations subscribing to such a rating service would have confidence in the results. These sorts of ratings would go well beyond what Charity Navigator can provide at this point. What is also intersting to me as I have been looking at all the groups involve din this work is the extent to which people who have real operating and management experience in the for-profit sector don’t appear to be playing a role in the work.