Professional philanthropy, like all professions, has built a special language to describe its work. This sort of language can be used to more precisely discuss issues of importance to a field or it can be jargon that obscures meaning and serves to identify professionals to each other while excluding “outsiders”.
Most donors, regardless of the vocabulary they use, want their donations to produce results.What characterizes “results” may be very different to various donors. Sometimes the desire to see results can lead donors to seek indicators, like low overhead expense ratios, that are actually bad proxies for results. Sometimes the “result” a donor seeks might simply be public recognition. But believing that donors do not seek results is akin to believing that they would be just as happy throwing their money in the trash.
So I’d like to explain the three words that are used to describe the results of nonprofits; Outputs, Outcomes and Impact. Instead of being confusing jargon, outputs, outcomes and impact actual mimic very common sense approaches to achieving results in everyday personal efforts such as working out to improve your health (more on this later):
- Outputs: These are the activities done by the nonprofit. The meals served by a soup kitchen are outputs.
- Outcomes: These are the observed effects of the outputs on the beneficiaries of the nonprofit. The degree to which the meals served by the soup kitchen reduce hunger in the population served by the soup kitchen.
- Impact: This is the degree to which the outcomes observed by a nonprofit are attributable to its activities. The impact of the soup kitchen is the degree to which a reduction of hunger in the population they serve is attributable to its efforts. While a soup kitchen might serve a lot of meals and correctly observe that hunger is subsequently less prevalent in the population it serves, the reduction in hunger might simply be attributable to an improving economy, or a new school lunch program or some other activities that are not part of the soup kitchen’s efforts.
While outputs, outcomes and impact might sound like jargon, they are an extremely useful vocabulary for discussing the results of a nonprofit. They help illustrate the tradeoff between the difficulty of obtaining knowledge and the value of the knowledge.
- Outputs: Relatively easy to count. Are often selected based only on a theory about what is helpful. If the outputs counted do not lead to anything meaningful, the “results” are meaningless.
- Outcomes: More difficult to measure. Do measure the observed effects of the nonprofit’s activities. However, the observed outcomes may not actually be due to the nonprofit’s activities. If so, the “results” are meaningless.
- Impact: Very difficult to measure. Requires some form of analysis which attempts to hold static the effects of other influences. This is the gold standard because the results are proven.
A quick example to make all of this personal, practical and relevant:
If you are trying to get in shape, you may well try to lose weight.
- Output: The amount of calories you consume minus the amount of calories you burn.
- Outcome: Your observed weight.
- Impact: The degree to which your level of health is improved by your weight loss.
Outputs are obviously a good place to start. You can’t legitimately argue that you are trying to lose weight if you pay no attention to the amount you eat and workout.
Outcomes are better. You can directly observe your weight and know the degree to which your diet and exercise plan appears to be having an effect.
Impact is best. You may be restricting calories, working out and observing a reduction in your weight. But only a rigorous evaluation can eliminate the potential effects of outside influences. Maybe your weigh loss is due to a serious undiagnosed medical issue or some other influence that has nothing to do with your efforts.
I can hear the urgent comment of a large group of readers already. “The outputs, outcomes and impact of our efforts are far more difficult than tracking what you eat and how much you weigh!” That’s 100% correct. No one is saying that measuring results is easy.
But here’s the fantastic secret of tracking outputs, outcomes and impact. Rigorous studies have proven that simply attempting to track activities results in impact across many different fields. Just tracking calories consumed, results in weight loss that is attributable to the tracking itself, not the specifics of the diet and exercise regime followed.
This is why it is so important that nonprofits that want to actual make a difference at least attempt to track their activities in some way. While the process can be difficult and only becomes more difficult as you move from counting inputs to proving impact, just the effort to begin the process will increase the measurable or immeasurable impact you are having.
You can read Part II of this post here.
23 Comments
Great article Sean! Outcomes and impact can be so confusing for nonprofit folks. Thanks for making it clearer!
Sandy Rees
Fundraising Coach
Outcomes may be difficult to measure, and impact even harder, but you are SO right that these are critical to the success of nonprofit work. Donors jumped on those indicators that charity watchdogs created because it’s hard as a donor to know how to measure the impact of what you give. But I agree that low overhead expense ratios miss the point. That has nothing to do with impact.
And it’s not just about attracting and keeping donors, it’s about answering those critical questions that so often get lost in the business of daily life at a nonprofit: do we need to exist? Are we fulfilling our mission? Are we making a difference? And yes, it can can be kind of scary to ask those questions, just like it can be scary to get on a scale.
I agree with Alexandra. As the ED of a small literary nonprofit, we’ve thought about measuring our outcomes since we started. We have hired two different people to do evaluations of our programs. We do work at measuring the immediate impact but unfortunately it’s very hard to measure the impact when it could happen long after the participant was part of the program. I wish there were a way to do that that didn’t require a lot of money and a stable communities. Our students often leave the area where we first meet them and are not trackable. We have some ideas and are working on the problem.
Thanks Sandy and Alexandra. Glad you liked the post.
Thanks for continuing to clarify this oft-confusing topic!
For purposes of operational and constituent management, we also break down the outputs area in various types of metrics, which leads to a slightly different path on the output-outcome-impact continuum. Each increasing level takes a more capable system and process, but should yield additional value.
Basic Outputs – Direct tracking metrics that can be exported from individual systems. (ex: Total # of Clients)
Calculated Metrics – Ratios and other calculated manipulations of the Basic Outputs, often between systems. (ex: Cost per Client)
Advanced Analytics – Operational value and effectiveness measurements and analysis, connecting outputs with strategic plans. (ex: Effectiveness of Marketing Campaigns in Obtaining Clients)
Cross-Domain Analytics – Coalescing data from across a portfolio – either across operational areas within one organization or across multiple organizations – into meaningful groups, allowing for full spectrum analysis. (ex: Total Client Services across Multiple Programs, Total Children Served by Age)
Hey Sean,
Thanks for this excellent post. I tend to wrongly conflate outcomes and impact, erring toward the definition you give of impact but calling it outcomes. This is a useful theoretical distinction. Indeed, outcomes is perhaps a point along the road toward impact but I would argue that the flow you give of outputs, outcomes, then impact, is not quite as linear (worst to best) as you state.
Outputs and impact are more pure metrics in that they both isolate an effect of the intervention. The output obviously tells us very little (such as in your example a client receiving food aid) and the impact telling us a lot (the degree to which hunger decreased, nutrition increased, etc., as a result of that aid). While it might seem that outcomes are superior to outputs, there is serious risk that outcomes are misleading, as you get to a bit.
Most damaging, the outcomes metric can mask harms from an output. You allude to a scenario where there are multiple factors at play toward a positive end of an intervention, but an intervention can actually have a negative effect, which might be offset by other environmental factors, not only masking the harm of the intervention but wrapping it in a cloak of success.
My point then, is not to say that outcomes are not important, indeed they are a useful bridge between outputs and impact, at the very least providing a quick and dirty method of calculating improvement or regression amongst a served population. However, I do think it is a mistake to say that outcomes are categorically better than outputs when outcomes have such great potential for mis-analysis.
Excellent post Sean! At first I was like “huh? what’s this pic doing on Sean’s post? Is it a mistake? Has he gone over to the dark side with his visual marketing???” But then I read your post and saw the clever connection. It is super important to be clear about outputs, outcomes and impact and what they are and what they are not. And I totally agree that nonprofits attempts to track these put them well on their way toward the goal of understanding the difference their work is making.
As to the weight loss point, I agree from personal experience. I’d love to say I’ve been tracking, as I have in the past, how many potato chips, french frieds, chocolate chip cupcakes, and NY Super Fudge Chunk ice cream scoops I’m consuming and how much I exercise as a means toward curtailing my bad eating habits (and bad health outcomes) but maybe I’ll have to start again!!!
How’s that for some public accountability for results? 🙂
David, thanks for sharing your process. Certainly Outputs, Outcomes and Impact could be broken down into more and more segments.
David,
I would argue that Outcomes are objectively more important (better) than Outputs, but admit freely that they are harder to measure. I guess like any powerful tool, they can become dangerous if used incorrectly (miscalculated).
Hmm. Full disclosure, I just swapped out the photo, slightly edited the title of this post and added a sentence to more obviously connect the photo to the content. I selected the photo after finishing the post and so ended up picking one that was relevant to the very last content. Not very smart graphic design.
Regarding public accountability. It is interesting to note that a hot trend in fitness is to publicly post your work out, food eaten or weight via twitter, Facebook, etc. Talk about motivation to stay on track!
Sean – Agree that the clarity you provide to each of these terms is useful. Hope you will do a follow-up post, however that further expands on a comment you have buried in the comments: ” …like any powerful tool, they can become dangerous if used incorrectly (miscalculated).” Unfortunately too many organizations waste precious time and resources measuring outputs and perhaps, outcomes, never getting to impact. They ‘measure’ because they have been told to measure and it becomes an activity in and of itself.
So thank you for some very useful clarity, but an encouragement to say more in the future about the importance of understanding WHY an orgis is measuring and how any measuring will be constructively used.
Excellent point Bonnie. I’ll consider the topic for a future post.
I definitely agree that outcomes have more value than outputs, and likewise for impact. Obviously you can count things till you’re blue in the face, but if they aren’t producing outcomes – meaning they aren’t furthering your Theory of Change and your mission – then they’re not providing any value.
However, I don’t think you can track outcomes very well if you can’t even track your outputs, which we see all over the place (Melinda can probably comment here!). Whether from lack of systems, knowledge, skills, focus, or simply the environment, tracking outputs can be a significant challenge for many organizations. And it’s incredibly difficult to get to outcomes if you don’t know how many blankets you gave out.
In response to Bonnie, there’s the concept of what gets measured gets paid attention to. Sometimes just helping an organization figure out how to measure a few things, whether related to their mission execution or internal operations, can have a significant effect on how the organization gets managed. But yes, of course it has to be the “right” things, and the results have to be used constructively.
To Bonnie and David’s points about the dangers of measuring outputs, outcomes, and impact: Yes, there are dangers involved with the amount of time and effort put toward the measurement activities overwhelming the benefits of the program being measured. And funders are often the culprit in demanding sophisticated evaluations for rather paltry sums of funds – certainly the measurement should be commensurate with the funds provided (and funders should be underwriting the costs of measurement for sure!)
Other dangers include being drowned in data that one cannot possibly ever sift through in a meaningful way; spending lots of time and resources to measure things that aren’t important; asking the wrong questions to begin with; the list goes on and on.
Two important questions to consider in any measurement activity:
1) Why are you collecting data?
2) What will you do with the data once you have them?
PS: Thanks Sean for swapping out the picture.
Hi Sean,
A good Wizard of Oz reference is never wasted on me so in response to you tweet “Outputs, Outcomes & Impact, Oh My!”, I offer up that a nonprofit needs a brain, a heart, and courage. A brain to track the outputs quantitatively, a heart to know why you want those outputs and what to measure that really matters; and courage to draw the correlations and links to the bigger change that you are driving toward. Courage to try and courage to stop or change things that are not working. And the courage to collaborate with others to achieve impact on a much larger scale.
Check out this post on 2 nonprofits that have collaborated and co-located for greater impact: http://bit.ly/9jI22c
I also draw your readers’ attention to the series of impact discussions on Social Edge:
http://www.socialedge.org/discussions/success-metrics
Especially the one on the Power of Impact Measurement led by the talented Lakshmi Karan (@lkaran):
http://www.socialedge.org/discussions/success-metrics/archive/2010/03/12/the-power-of-impact-measurement
Cheers,
Jill
Thanks Jill. Those are all great links!
Nice post Sean, and timely, as I delivered a “introduction to social impact measurement” to our Devon School for Social Entrepreneurs yesterday.
You are right to say that understanding the jargon is important (my slight amendment is that the outputs are the NUMBERS of activities completed / delivered), and what you lay out above is helpful. I would also recommend the New Economics Foundation’s work in this area, particularly at http://www.proveandimprove.org
I would only add a couple of things: that it is also crucial for organisations / social entrepreneurs to understand the story or narrative behind what they do and how they are seeking to make change. This is crucial for knowing what outcomes they are seeking to measure (and therefore what indicators to choose to know that change is happening), and to communicate what they do to others.
[see the storyboard mapping exercise on website above for this]
This is also part of evaluation being seen not just as ‘reporting’ to donors, philanthropists and investors of all types, but also an important part of improving an organisation’s work: we need to prove that what we are doing is making a difference, but we also need to improve those services wherever possible, learning from what we’ve done so far.
Cheers!
Nick, you’ve gotten right at the “why” behind outputs, outcomes and impact. I completely agree.
Nick T hit it RIGHT ON THE MONEY. That is reason #1 why outcomes and impact system shold matter, i.e. so the non-profits can improve their effectiveness and mission success
While the distinction among output, outcomes, and impact is certainly valuable, the more interesting part of this post is the comment section, which makes it very clear that measuring impact, while most desirable, is also most difficult.
With the new paradigm for nonprofit/social enterprise funding — private/public partnerships and and investment model — nonprofits must find a way to accurately and convincingly measure impact.
Since we’re at the beginning of the measuring-impact journey, and our tools and techniques haven’t been perfected, making any attempt impresses. Sharing what you’re learning about measurement impresses even more. Working with others to develop standards impresses the most.
Great post, Sean.
To your point about professionals using jargon that excludes outsiders, it can be enlightening to read how regular people describe the “outputs, outcomes and impact” of the nonprofits that touch their lives. These are the stories we collect in the form of stakeholder reviews at GreatNonprofits.org.
One young woman, for example, described the impact the Kristin Brooks Hope Center, in Washington, D.C., had on her life: “One night I called the hopeline hoping for alternative release than hanging myself. I talked to a lady and she told me I was a good person and I needed to believe in myself. Six years later, I’m still here.” — Jenny93
Couple of thoughts I’d like to add……
First off, a nonprofit should be collecting output, outcome, and impact data for a single reason: to improve the services provided to its clients/constituents.
Second, I think that nonprofits need to be honest with themselves about what level of influence they can truly have. Not all nonprofits can provide services that lead to impact, and that is ok. For example, a shelter that just provides meals for people really is only dealing in outputs. But that is still a good service. One of the major problems I’ve observed in doing this work is that nonprofits THINK they can lead to impact, but in reality their scope or size is too small to achieve anything beyond changes in outcomes.
Finally, as someone formally trained in evaluation, I actually think measuring outputs, outcomes, and impact is fairly easy and straightforward. The truly difficult part is getting nonprofits to identify the specific things they want to track.
Example: a nonprofit says that it wants to measure ‘academic performance’ for its population. But what does that mean? Do they mean improved school attendance, improved grades, improved standardized test scores, grade-to-grade promotion, high school graduation, increased belief that a student can attend college? All of these could be thought to make up the larger concept of academic performance.
Thanks Isaac (for those that don’t know, Isaac is head of evaluation for a nonprofit that is regularly recognized as a leader in measuring their performance),
In case you missed it, my follow up post to this one was on the “why” of evaluation in which I agreed with your point that the reason is all about improving services.
One your second point, I fully recognize your formal training in evaluation, so this is a question rather than a challenge. But shouldn’t every nonprofit strive for impact, even if it is not a “root cause” solution. For instance, a shelter providing meals better be actually making life better for the people they serve. You can easily create outputs, just shovel out means, but if you are handing out meals in Beverly Hills (for instance) you might be having zero impact.
I’d like to think that impact can be achieved without a nonprofit laying claim on solving a root cause. Don’t you agree?
On your last point, I think the reason that impact is difficult to measure is because of the “what do you mean” issue you raise. But it isn’t just a matter of figuring out what you mean. Every example you give of “academic performance” is simply a proxy. It actually really is difficult to measure certain conceptual ideas like “healthier communities”, “successful people”, and “happier families”. But I agree that it is not out of reach to settle on the most useful proxies and then measure your progress to those goals.
A discussion we had recently at SDSVP really made us think about the different types of impact, and the different requirements to measure those alternative facets. Basically, impacts on the beneficiaries vs impacts on society. For example, take a program providing transitional housing/education for homeless to get them permanently off the street. The regular way of assessing impact seems to be things like measuring how many people they successfully got and kept off the street, and possibly even looking at homeless rates overall. But what about the discussion of the value to the local region of not having that homeless person on the street? What was the change to local businesses by that person not being there, for example? That is a much tougher assessment, that can’t be measured solely with internal case management data, but is a very valuable measurement of societal impact – and is more directly in the control of a single organization than things like the overall homeless rate.