Heather McLeod Grant, one of the authors of Forces for Good, has weighed in on the debate with a thoughtful comment that I present an abbreviated version of below. You can read the full comment here.
By Heather McLeod Grant
Sean,
Thanks for referencing our book, Forces for Good, as an example of both high-performing and high-impact nonprofits… First, a caveat – neither Leslie or I are “evaluation experts,” which is perhaps why we approached our methodology so pragmatically. We took both a “bottom-up” crowdsourcing approach (asking thousands of people in various fields to rate their peers and tell us who was high-impact), combined with a “top-down” approach, asking experts in these fields (including funders and academics) to help rank these organizations based on their deep knowledge of the field and various players. We also looked at whatever self-reported data we could find, and examined the evidence of “impact,” for each of the organizations we ended up writing about…Whether or not they are “high-performing” is debatable, depending on how you define the term: they all have had impact, but not all are perfectly managed organizations… The six practices we discovered really have to do with paths to scaling impact and achieving greater results (broadly defined), more than internal operational management or performance. I do think the sector could do more to emulate these nonprofits’ approach illustrated in the practice of “Adaptation”: using data (often real-time-feedback, not long term control group studies) as a method for constantly improving performance (and therefore impact), rather than seeing it merely as a tool for reporting to funders.
As we went through [the] process, we learned just how challenging this question of defining impact in the social sector truly is, for numerous reasons:
– One: There is no single metric like “shareholder value” in our field which is an agreed-upon proxy for impact (bear in mind even in the for-profit sector there are many other ratios/ indicators that investors look at to assess organizational performance)…
– Two: There’s the challenge of comparing apples to oranges: what you’re measuring as outcomes/impact in education (closing the achievement gap, graduation rates, etc.) is quite different from what you’d measure in the environmental field (reduced carbon emissions, protected species, etc.)…
– Three: Not all types of “impact” lend themselves to quantification in the first place; some types of social impact are intrinsically qualitative; e.g. you can measure the number of people who attend a symphony (an output), and overall membership rates over time, but how can you value the contribution of the arts to society? The impact of hearing well-performed music on individuals’ lives?
– Four: The time-scale it takes to achieve social change is often not easily measured in quarters or years, but in decades; so funders seeking “quick wins” will often shut down a program too soon; we know from evidence that programs like Head Start make a meaningful difference in kids lives, but often the real benefits don’t show up until much later in life…
– Fifth: Causality is often difficult to assign in the social sector. We know from experience that nonprofits are often working on multi-variable problems (e.g. children in poverty), and that their intervention might only address only one piece of the problem… So, while you can measure outputs and possibly outcomes, very few of these programs are solving the entirety of the problem – and many other programs are also working with these kids. Who then gets credit if an impoverished child who participated in various programs goes on to Harvard?…
– Sixth and last: One of the biggest ah-has from our research was that the best nonprofits often don’t focus merely on direct-impact, or measurable outcomes, but rather, they focus on larger systemic change– what we call “indirect impact” at the Monitor Institute. Over time, the highest-impact nonprofits start trying to address larger systems failures, and get to root causes or more sustainable long-term solutions than the band-aids that many social service programs provide…
Most nonprofits exist in the first place because of failure by markets or by government. And therefore, to be truly successful, they often move from providing services to actually seeking to influence whole systems. In this case, the “impact” they are having cannot be measured directly by their program’s output (number of teachers trained and placed in Teach for America’s case), or even their organization’s performance (being well-managed and highly data-driven)…Rather, Teach for America’s largest impact is arguably in the alumni network it has built, and what those alumni are now doing with their lives to transform public education in America. …Yet none of this would lend itself to conventional evaluation methods, and that’s my point…
…I’m not saying we shouldn’t evaluate: we should. (And I’m not an expert who can elucidate the pros and cons of various methods.) But we should do so thoughtfully, and weighing all of the challenges and complexities I’ve outlined above – recognizing that a one-size-fits-all approach won’t possibly work for the diversity of the social sector, and that we need a fairly broad and somewhat flexible definition of impact. I do advocate for moving away from a system where we evaluate nonprofits on arcane internal metrics like their “overhead ratios” which do nothing to illustrate the return on a donor’s investment, or the long term outcomes. We should focus much more on “impact”—even more than on performance (if the latter is overly focused on organizational perfection, rather than social change)…
But we shouldn’t become prisoner to these tools and methods either, or let the perfect become the enemy of the good. There’s a danger in becoming too obsessed with the data and tools, as well-intentioned as we might be; or using the wrong tool in the wrong context, and missing the larger picture. Sometimes philanthropy requires a leap of faith; and smart investing requires being thoughtful and weighing multiple factors, and understanding the context in which these nonprofits operate, and exercising judgment. It’s this that will ultimately lead to higher-impact nonprofits, and to a higher-performing social sector.