Performance Vs. Impact Debate Rekindled

Checklist Last summer, an epic debate took place on this blog as a number of readers and social sector leaders argued about whether donors should focus on supporting “high performing” or “high impact” nonprofits (see links here, here, here and here). The basic difference is that a high performing nonprofit is an organization that is run extremely well while a high impact nonprofit is one whose programs have been proven to work.

I took the position that donors should focus on high performing nonprofits, but argued that a “robust” definition of high performance was needed. Since then I’ve refined my robust definition of a high performing nonprofit to mean organizations that;

  • base their programs on research about what works,
  • actively collect information about the results of their programs,
  • systematically analyze this information,
  • adjust their activities in response to new information,
  • and operate with an absolute focus on producing results.

My central point in the debate was that while high impact nonprofits were great, they were few and far between and their proven impact only showed what had happened in the past. A high performing organization, as long as it executed on the five points above, was well positioned to deliver impact in the future.

Now, Holden Karnofsky of GiveWell, a participant in the first round of the debate, has offered up a new set of arguments on the issue. Holden writes:

“When I look at the three charities that I consider to be most “blue-chip” (our three highest-ranked charities), I don’t see a path of a “strong organization that eventually figured out what to do and whether it worked. ” Rather, I see organizations that stayed as small as possible – or didn’t exist – until they had strong evidence of impact for their basic approach. They built their choice of programming into their DNA, as much as they could, from day one.

The reigning consensus seems to treat “evidence of impact” as a late (or at least potentially late) step in the development of a nonprofit, but in fact it has been the first step for the strongest nonprofits I know of.

This makes intuitive sense to me as well.

  • Finding “approaches that work” is fundamentally a research challenge, and probably requires a completely different skill set from running an organization well.
  • Once an organization is “up and running,” it may become a very poor environment for a good impact evaluation. To me a good impact evaluation is one that has a real chance of demonstrating failure, and the stakes may simply be too high for an organization that has already built up significant funds, donors, clients, stories, staff, habits, etc.

The truth is that if an organization wants to become “high-impact,” there are already proven approaches for it to choose from; if it wants to investigate an approach that isn’t yet proven, it can (like VillageReach) stay at minimal size and essentially act as a “research project.”

I highlight Holden’s post because I think it reveals a frequent, incorrect assumption about high performing nonprofits. In his argument, Holden conflates “high performing” with the idea of a large nonprofit that is operating at some level of scale. But if you look at my five point definition of high performance, you can see that it describes an approach applicable to organizations of any size.

Critical to my definition is that a high performing organization base its programs on research about what works. So when Holden says that, “Finding approaches that work is fundamentally a research challenge, and probably requires a completely different skill set from running an organization well.” I have to say that I agree. It may be that large, high performing organizations can also invent new approaches to achieving social impact, but the process is more of a research and development activity.

To be clear, people on both sides of the debate care about achieving impact. Those who favor high performing nonprofits do so only because they believe that these types of organizations are most likely to achieve impact over time. If impact where a perfectly quantifiable and repeatable process, then I would agree with the high impact crowd. But unfortunately I think that achieving impact is an imperfect process. Just because a program has been proven to work in one set of circumstances does not mean that the same process will work elsewhere.

A high performing nonprofit is one that is able to implement well researched, proven approaches (whether the research was conducted internally or externally) with a high degree of fidelity to the original approach. But then also has the organizational capability to track their programs as they scale, determine what is working in their specific case, course correct as needed and constantly look to improve.

There is great research about what works across fields as diverse as business, psychology, weight loss and baseball. But finding an approach that works does not mean that anyone can easily implement it. If this were so, then every well read person would be a successful business person, with a healthy psychological profile, who was in great shape and played professional baseball as a hobby.

But truly successful individuals (and organizations are just collections of individuals) make it a habit to base their activities on evidence about what works, be mindful about what is working for them, reflect on what changes they might make and have an absolute focus on achieving results.

Ask any venture capitalist and they’ll tell you that they bet on people, not business plans. I think a focus on high performing nonprofits is simply an extension of this concept that emphasizes betting on organizations, not research. Great business people craft solid business plans and great organizations depend on credible research. But smart investors and smart donors should place their bets on the very best people and organizations and never fall for the seductively idea that research and planning can guarantee a positive result.


  1. Sean, I think there is an important debate going on that your post obscures/glosses over.

    The position you implicitly present as the “high-impact position” (when you say “Just because a program has been proven to work in one set of circumstances does not mean that the same process will work elsewhere”) is, I think, a straw man. I don’t think anyone says that program selection is sufficient for effectiveness, or that organizational quality is; I believe pretty much everyone agrees that both are ultimately necessary to achieve the “holy grail” of high impact.

    But there is a real debate over the order in which these qualities should be pursued.

    On one side, we have those who say that we should start with great organizations and gradually work our way towards appropriate program design. These people advocate funding organizations that have, for example, good management practices and good use of data, regardless of whether the programs they’re running have any evidence of impact behind them. Their definition of “high-performance” does not require appropriate program selection. Call this position “Organizational quality first.”

    On the other side, we have those who say that we should start with proven programs and try to build organizations around them. These people feel that the “incubation” phase of an organization is the truly crucial one, and advocate staying away from organizations that have reached a certain size/age without getting to evidence of impact, no matter how well-managed that organization is. Their definition of “high-performance” does require appropriate program selection. Call this position “Program selection first.”

    We are gravitating toward the “Program selection first” position. But I believe most people participating in this debate are in fact on the “Organizational quality first” side, in actions if not in words. As our post points out, the most recent draft of the Social Risk Assessment Protocol effectively gives twice the weight to a charity’s evaluation and adjustment practices that it gives to the charity’s choice of programs. (And the first draft didn’t even address this issue.) And we’ve both been at the meetings where a relatively mature, scaled, still-unproven organization is presented as “high-performance.”

    It is not entirely clear to me whether you are taking a side. Your earlier posts on this topic referenced program selection only as an eventual goal, not as a requirement, for high-performance organizations. But this post does put “base their program on research about what works” as a requirement for an organization to be considered high-performance. In my view, your definition therefore disagrees with the more popular definition, and would you put you in what I consider the “program selection first” camp.

    I think this is a very important debate because it comes down to what sort of organization should be funded.

  2. I like how you framed the issue as which one first. I believe that program selection is critical and require in my definition that a high performing nonprofit be one that bases its programs on research.

    Is that enough to bridge our gap or do you believe that an organization should prove that their implementation of a program is working as a first order requirement for funders?

  3. Jeff Mason says:

    Right on Sean! All high-impact orgs are also high-performing. Orgs with the high-performing characteristics that Sean listed above will be able to navigate their way to success.

    As mentioned, orgs that target outcomes where evidence-based models already exists should leverage these – no need for innovation. However, there are many populations and issues where such models don’t exist. In these circumstances it’s only the high-performing orgs that can effectively blaze a new trail to success because these orgs learn from their mistakes and continuously improve.

    We need all donors to base their decision to fund on whether or not an org has high-performing characteristics. These are the only orgs that stand any chance of INTENTIONALLY achieving there goals.

  4. Jeff, where I take issue is with this statement:

    there are many populations and issues where such models don’t exist. In these circumstances it’s only the high-performing orgs that can effectively blaze a new trail to success because these orgs learn from their mistakes and continuously improve.

    The “program selection first” approach would say that where a proven model doesn’t exist, funding should be aimed directly and exclusively at establishing such a model – i.e., conducting research, not supporting organizations. Rather than starting by building an organization, putting systems in place to track outcomes, and then eventually conducting a high-rigor study (like an RCT), a funder should essentially start with an RCT (there may be exploratory research first, but not the founding of a full-blown charity). This is my understanding of how the Nurse-Family Partnership did things.

    Sean, it sounds like your definition of “high-performance organization” is in agreement with ours, and, I believe, in disagreement with Jeff’s. I believe that the “research based” requirement, when applied rigorously, is itself sufficient to rule out an enormous number of organizations, and leaves us with very few organizations that can truly be considered “high-performance.”

  5. Very interesting discussion. I think that you, Holden, simple have a deeper level of trust that pure research can discover solutions than I do. I think that solutions to problems in physics, chemistry, etc, can be done this way. But I’m not so sure about social solutions (no pun intended Jeff).

    While for-profits are not a perfect analogy, we do know that most services are not designed in a lab, they are designed by an organization. I think that holds some important lessons for the social sector. I think that organizations operating on the ground may be best positioned to run programs which are essentially R&D. But I do think your point is very important and you may be correct.

    On my definition, I think I probably have a looser definition of “based on” when I talk about basing programs on research. I don’t mean only implementing a program which has been specifically tested, but building programs that are based on research. So if a new program is developed that provides after school art instruction because research shows that children who receive art eduction in school have higher graduation rates, I think this would qualify. Of course the more research, the better. And the more directly the program elements have been tested the better.

  6. Sean, as I argued in the post in question, it seems to me that the limited facts favor our view. “Research first” isn’t just an idea; it’s the story of the Nurse-Family Partnership, the only U.S. organization that anyone seems to feel truly comfortable labeling as “high-impact.” To varying degrees it is also the story of our top two international charities, and the ChildTrends website lists many more instances of proven programs without organizations.

    Of course, this sort of research doesn’t take place in a laboratory. It requires staff and processes similar in kind to what an organization would need. But everything is focused on the ultimate goal of gathering information: a design allowing causal inference is built in from day 1, academics are involved from day 1, and the resources marshaled are enough to gather information but no more.

    I don’t think any intuition or analogy makes it obvious whether this approach is superior to building an organization first, but the facts we’re aware of seem to point to it as the approach more likely to generate “high-impact” organizations.

    On the definition, if you consider the research case for an organization’s programs to be core to whether it’s “high-performance,” then we agree at the broadest level, though we may disagree over what constitutes a “good” research case.

  7. I think it’s helpful to return to the original point we were arguing: it may be that great charities are born, not made.

    Largely what we’re doing is questioning whether there is even a useful concept of “high-performance” that is distinct from “high-impact” – any such thing as an organization that is “on the way” to becoming a blue-chipper. The blue-chippers we know of basically jumped straight from “startup/exploratory” to “blue-chip,” with no intermediate “high-performance” phase at which it would have made sense for outside funders to come in.

  8. Jeff Mason says:

    In my opinion the difference between “high-performing” and “high-impact” is that a high-impact orgs have been proven through formal evaluation to be the cause of the social value generated. High-performing orgs show signs of impact but have not yet been through formal evaluation.

    I would classify Harlem Children’s Zone as a high-performing org. They have a specific goal, a well thought out strategy for achieving that goal, they have established key milestones/indicators that can be used to measure their progress towards their goals, they collect quality data that lets them know what appears to be working and what’s not working, and they use this data to make midcourse corrections and continuously improve. Orgs that do these things are more likely than others to eventually achieve proven impact. However, I don’t believe that HCZ has undergone a full formal evaluation. They have lots of data that are suggestive of impact.

  9. Jeff, in my view, “show signs of impact” is different from “have established key milestones/indicators that can be used to measure their progress towards their goals, they collect quality data that lets them know what appears to be working and what’s not working, and they use this data to make midcourse corrections and continuously improve.”

    “Signs of impact” is what GiveWell looks for. We don’t draw a bright line between “proven” and “showing signs”; we don’t think any organization is really “proven,” but we look for the organizations that seem to have the strongest evidence of impact.

    And from what we’ve seen, these organizations put together large parts of their case by utilizing independent research and exploratory/startup work. While we’re in favor of setting goals and having midcourse corrections, I’m not convinced that once an organization is out of the “startup” phase without basic evidence of impact, the process of setting goals and making midcourse corrections is likely to get it there. I see the startup phase and initial choice of program design as crucial.

  10. I think high impact should mean that a formal evaluation of the nonprofit’s programs have been conducted that demonstrate that they are having an impact. Preferably multiple evaluations.

    But an org can be determined to be high performing prior to any evaluations by examining the way in which they are running their organization (such as by designing their programs based on research about what works more generally).

    This is an important distinction because evaluations can only be done retrospectively, are expensive, and may be very difficult to conduct conclusively in many areas of the social sector.

    Holden, you make a good point about what should come first. I think it is a great practice for a nonprofit to gather as much proof as possible that their programs are working before they undertake significant growth. But I think your sample is a bit skewed because you are looking for high impact nonprofits and therefore it is not surprising that what you are finding are organizations which worked on proving their impact early on. In addition, your high hurdle for being convinced that an org is high impact means that you are limiting yourself to areas of the social sector where “proof” is easier to determine and therefore areas where conducting research is more cost-effective for the nonprofit.

    I think I agree with your thinking, but am more willing to view the sorts of nonprofits you are seeking as a subset of the sorts of nonprofits that donors should support rather than the best or only kind.

  11. Sean, the main justification that has been raised for supporting high-performance organizations (by yourself and others, including Jeff directly above) is that they are the ones most likely to become high-impact.

    In assessing this claim, it seems fair to focus on the history of high-impact organizations, as I have done. I don’t think my sample is “skewed” for this purpose – I think the set of actual “high-impact” orgs is exactly the right sample.

    There may be other reasons to support “high-performance” organizations, other than the idea that they’ll eventually become “high-impact.” But these reasons need to be spelled out better than they have been.

    My personal take on what to fund is as follows:

    -Fund only organizations whose program selection makes good use of available evidence – *or* projects that are focused on creating new high-quality evidence (i.e., studies).

    -If you consider “program selection makes good use of available evidence” to be a requirement for “high-performing,” then I endorse funding “high-performing orgs.” I think this is different from the definition that has been used to date, but may represent an evolution rather than overturning of the concept.

    -I think the definition of “high-impact” you have given only makes sense in the context of a very particular set of organizations (U.S. human services focused on long-term outcomes) and becomes unreasonable when applied to other sectors. So the idea of “high-impact” vs. “high-performance” under these definitions isn’t crucial for me.