In case you missed it, my follow up post to this one was on the “why” of evaluation in which I agreed with your point that the reason is all about improving services.

One your second point, I fully recognize your formal training in evaluation, so this is a question rather than a challenge. But shouldn’t every nonprofit strive for impact, even if it is not a “root cause” solution. For instance, a shelter providing meals better be actually making life better for the people they serve. You can easily create outputs, just shovel out means, but if you are handing out meals in Beverly Hills (for instance) you might be having zero impact.

I’d like to think that impact can be achieved without a nonprofit laying claim on solving a root cause. Don’t you agree?

On your last point, I think the reason that impact is difficult to measure is because of the “what do you mean” issue you raise. But it isn’t just a matter of figuring out what you mean. Every example you give of “academic performance” is simply a proxy. It actually really is difficult to measure certain conceptual ideas like “healthier communities”, “successful people”, and “happier families”. But I agree that it is not out of reach to settle on the most useful proxies and then measure your progress to those goals.

First off, a nonprofit should be collecting output, outcome, and impact data for a single reason: to improve the services provided to its clients/constituents.

Second, I think that nonprofits need to be honest with themselves about what level of influence they can truly have. Not all nonprofits can provide services that lead to impact, and that is ok. For example, a shelter that just provides meals for people really is only dealing in outputs. But that is still a good service. One of the major problems I’ve observed in doing this work is that nonprofits THINK they can lead to impact, but in reality their scope or size is too small to achieve anything beyond changes in outcomes.

Finally, as someone formally trained in evaluation, I actually think measuring outputs, outcomes, and impact is fairly easy and straightforward. The truly difficult part is getting nonprofits to identify the specific things they want to track.

Example: a nonprofit says that it wants to measure ‘academic performance’ for its population. But what does that mean? Do they mean improved school attendance, improved grades, improved standardized test scores, grade-to-grade promotion, high school graduation, increased belief that a student can attend college? All of these could be thought to make up the larger concept of academic performance.

To your point about professionals using jargon that excludes outsiders, it can be enlightening to read how regular people describe the “outputs, outcomes and impact” of the nonprofits that touch their lives. These are the stories we collect in the form of stakeholder reviews at GreatNonprofits.org.

One young woman, for example, described the impact the Kristin Brooks Hope Center, in Washington, D.C., had on her life: “One night I called the hopeline hoping for alternative release than hanging myself. I talked to a lady and she told me I was a good person and I needed to believe in myself. Six years later, I’m still here.” — Jenny93

With the new paradigm for nonprofit/social enterprise funding — private/public partnerships and and investment model — nonprofits must find a way to accurately and convincingly measure impact.

Since we’re at the beginning of the measuring-impact journey, and our tools and techniques haven’t been perfected, making any attempt impresses. Sharing what you’re learning about measurement impresses even more. Working with others to develop standards impresses the most.

You are right to say that understanding the jargon is important (my slight amendment is that the outputs are the NUMBERS of activities completed / delivered), and what you lay out above is helpful. I would also recommend the New Economics Foundation’s work in this area, particularly at http://www.proveandimprove.org

I would only add a couple of things: that it is also crucial for organisations / social entrepreneurs to understand the story or narrative behind what they do and how they are seeking to make change. This is crucial for knowing what outcomes they are seeking to measure (and therefore what indicators to choose to know that change is happening), and to communicate what they do to others.
[see the storyboard mapping exercise on website above for this]

This is also part of evaluation being seen not just as ‘reporting’ to donors, philanthropists and investors of all types, but also an important part of improving an organisation’s work: we need to prove that what we are doing is making a difference, but we also need to improve those services wherever possible, learning from what we’ve done so far.

Cheers!

A good Wizard of Oz reference is never wasted on me so in response to you tweet “Outputs, Outcomes & Impact, Oh My!”, I offer up that a nonprofit needs a brain, a heart, and courage. A brain to track the outputs quantitatively, a heart to know why you want those outputs and what to measure that really matters; and courage to draw the correlations and links to the bigger change that you are driving toward. Courage to try and courage to stop or change things that are not working. And the courage to collaborate with others to achieve impact on a much larger scale.

Check out this post on 2 nonprofits that have collaborated and co-located for greater impact: http://bit.ly/9jI22c

I also draw your readers’ attention to the series of impact discussions on Social Edge:
http://www.socialedge.org/discussions/success-metrics

Especially the one on the Power of Impact Measurement led by the talented Lakshmi Karan (@lkaran):
http://www.socialedge.org/discussions/success-metrics/archive/2010/03/12/the-power-of-impact-measurement

Cheers,
Jill

Other dangers include being drowned in data that one cannot possibly ever sift through in a meaningful way; spending lots of time and resources to measure things that aren’t important; asking the wrong questions to begin with; the list goes on and on.

Two important questions to consider in any measurement activity:

1) Why are you collecting data?
2) What will you do with the data once you have them?

PS: Thanks Sean for swapping out the picture.

However, I don’t think you can track outcomes very well if you can’t even track your outputs, which we see all over the place (Melinda can probably comment here!). Whether from lack of systems, knowledge, skills, focus, or simply the environment, tracking outputs can be a significant challenge for many organizations. And it’s incredibly difficult to get to outcomes if you don’t know how many blankets you gave out.

In response to Bonnie, there’s the concept of what gets measured gets paid attention to. Sometimes just helping an organization figure out how to measure a few things, whether related to their mission execution or internal operations, can have a significant effect on how the organization gets managed. But yes, of course it has to be the “right” things, and the results have to be used constructively.

So thank you for some very useful clarity, but an encouragement to say more in the future about the importance of understanding WHY an orgis is measuring and how any measuring will be constructively used.