In 2010, the New Yorker published an article titled The Truth Wears Off, that looked at the existence of the “Decline Effect”, the seemingly inevitable way that when scientific studies are repeated over and over, they tend to follow a path of diminishingly positive results.

In the article, Jonah Lehrer wrote:

“The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.”

Lehrer’s article incited a flood of angry letters and emails claiming that he was undermining scientific research and drumming up a controversy that doesn’t exist. However, my reading of the article (and Lehrer’s responses to his critics) suggests a much more modest claim is at the heart of his article.

Human knowledge is an evolving concept.

For all the perceived precision of a large study “proving” that something is true, the fact remains that over time our understanding of facts and truths change.

Lehrer explains a number of reasons behind what is know as the “Decline Effect”. Taken together, much of the issue has to do with human cognitive biases and behavioral issues in the way we process information. For instance, Lehrer points to the way that scientific journals seem to greatly prefer to publish studies that prove something to be true, so scientists have a significant incentive for their studies to find these results.

But even if you peel away all of the messiness of the human practice of scientific study, you are still left with the idea that seeking truth is a process not an event.

Lehrer writes:

“The decline effect is actually a decline of illusion. While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that.

[The Decline Effect is so troubling] Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true.”

Now the silly way to react to the decline effect is to turn our backs on science and decide that if it cannot present us with the unquestionable truth, then it doesn’t work (this is the message that some of Lehrer’s critics through he was pushing). The more useful way to react is simply to understand that the concepts of “truth” and “fact” are far less rigid and concise than we tend to treat them. The search for truth, for “proven programs” will not end some day when we finally, finally, finally discover the real truth.

As Ralph Waldo Emerson said, “Life is a journey, not a destination.”

What this means for nonprofits and funders who want to direct their resources towards programs that actually work is that doing so will always be a continuous process. There will never be a a final, definitive study that tells us the “truth” of the best way to eradicate poverty, to end obesity, to give every individual the opportunities they deserve.

But that doesn’t mean we should lower our ambitions nor reject the scientific process. Instead, I think that Ted Cadsby, writing in the Harvard Business Review had it right when he argued in favor of adopting a mindset of “provisional truth”:

“Provisional truth requires that we think of our explanations as hypotheses — always subject to replacement based on new information or alternative ways of structuring existing information. Provisional truth means challenging our interpretations with disconfirming evidence and alternative perspectives. Provisional truth does not preclude drawing conclusions or taking action; but it demands that we be skeptical about our first reasonable explanations in the realm of complex problems. It keeps us humble and mentally flexible, constantly asking ourselves if we’ve really got everything figured out and responding, "Probably not."

But of course the scientists among you will recognize that the skepticism embedded in the idea of “provisional truth” is in fact a core aspect of the scientific process. The Decline Effect doesn’t discredit the process of scientific inquiry. Instead it simply lays waste to the fetishism of the scientific process that deludes people into thinking that we can at last completely understand and control our world once we discover “the truth”.

Sometimes a comic says it all.

“From the donors’ perspective we need something more than metrics and annual reports that cements the organization’s commitment to social good. Nonprofits have typically relied on (been given a pass by) their corporate structure and the public accountability of their boards. I’d say both of these are not strong enough. Social businesses, particularly those that meet the standards of a B Corporation, are beginning to document and commit themselves contractually to social good. This is a step toward mission insurance, but the B Corporation* standards are written mostly to protect business owners, then the investors, then the customers. We need something that will work for downstream investors or donors.

And there is nothing available to ensure customers – be they the borrowers from a microfinance organization, the teen employed by a job training company, or the working mom seeking a new credential to better her job prospects – that the services they are getting are being designed and delivered with a measurable, enforceable commitment to bettering lives and communities.”

Lucy’s proposal was made in the context of her thinking about how we might insure that social enterprises, both for-profit and nonprofit, actually achieve their social mission. But I was struck by the way that the concept illustrates so perfectly the current paradigm shift in philanthropy.

Whereas philanthropy has historically been about the gift as a moral act by the donor, today more and more people think about philanthropy in the context of the results achieved by the gift. This historical view is embedded in the legal obligations of nonprofits. Nonprofits actually have no legal requirement to achieve results. Instead they need only have a socially beneficial mission and ensure that no benefit incur to the donors that support them.

The focus on results and the emergence of for-profit social enterprises turn this understanding of philanthropy on its head. What we need now is a legal understanding of social businesses that requires them to achieve social beneficial results while recognizing that having financial benefit accrue to supporters may in fact increase and accelerate social impact.

In a comment on Lucy’s post I wrote:

“We know there are documented examples of nonprofit programs creating harm (for instance it is well documented that the DARE program to "keep kids off drugs" has no positive impact and some evidence suggests that it increases drug use).

In the world of consumers, this would be stopped. If a drug designed to decrease the drug cravings of addicts was found to have no effect and possibly to increase cravings, it would be recalled immediately.

While I think it might be hard to "insure" that a social mission is met (just like it would be hard to "insure" that a for-profit became and stayed profitable), it seems very doable to "insure" to some degree of conviction that social programs meet minimum standards. The meeting of this hurdle would then flow upstream and greatly increase the chance of mission success.”

Today, the tax protection that nonprofits enjoy is dependent only on them having a social mission and ensuring that no financial benefits flow to donors. And yet we wonder why “evidence-based” social activity, performance management and the measurement of results is so rare.

While the challenges of measuring results are real and should not be underestimated, imagine the positive transformation that would occur if tax protection was dependent on the achievement of socially positive results and whether investor/donors benefitted from the organizations activities took a backseat from a regulatory standpoint.

Which entity should receive tax benefits? A nonprofit whose programs exacerbate the problems of the people they intend to help, or an organization – either nonprofit or for-profit – that actually helps people?

Lucy’s posts were great because out of the need to communicate nonprofit due diligence best practices to a 10-year-old, Lucy manage to distill everything down to three simple questions:

1. What does the organization do?
2. How do they do it?
3. How do they know if they are making a difference?

I’ve written up my own list of five simple questions to ask, but they are a good deal more technical. Sometimes I think the “smart giving” movement gets lost in our own nuanced debates and forgets how powerful it can be to reframe our discussions so they make sense to anyone (in this case, a motivated 10-year-old ended up being the perfect foil for Lucy).

Here’s why I think Lucy’s questions are so powerful and are the core of what more complex evaluation processes are trying to get at.

What does the organizations do?

This seems to be such an obvious question that it doesn’t need to be asked. But visit the Red Cross’s website and try to explain what they do. Or check out the American Cancer Society, which many people think does cancer research, and then realize that only 17% of their program expenses go to research.

Understanding what an organizations actually does should always be the first step to building conviction in a decision to support them. A good answer to this question can’t just discuss the organization’s goals or focus area, but should describe the programs or approaches taken by the nonprofit in pursuit of those goals.

How do they do it?

The question above might be answered above for a college access nonprofit that the organization provides assistance to under-privileged high school students in the process of applying for college. But lots of nonprofits do that sort of work, so how does the nonprofit in question provide the service? How does their service differ from similar organizations? How do they fund their activities?

How do they know if they are making a difference?

Any high performing nonprofit is going to have some process in place for trying to get a handle on whether they are having success in their programs. The answer to this question doesn’t need to come in the form of a spreadsheet. It might be completely qualitative. But regardless of how it is answered, a solid nonprofit should be able to speak convincingly about their own efforts to know if they are making a difference.

In some ways, that’s about all you need. If before you make a donation you are able to fully describe what an organization does, how they do it and how they know if they’re making a difference, you are well on your way to knowing that your money is actually going to make a difference.

I’m sure some people will argue that this is too simple of a process. My own advice on this matter requires that a nonprofit base their programs on evidence about what works or, if the program is experimental, make clear that the program is a research effort. But we can go on adding qualifications and additional due diligence forever.

In a world where most people do little to no research before donating, it seems to me that a big positive shift would occur if donors began to spend 15 minutes figuring out the answers to Lucy’s questions before making a donation.

By Ian Thorpe

In the organization where I work, like in many other development organizations, there has been a lot of push over the past few years on “evidence-based” policies and programmes. So when I tell people I work on Knowledge Management, they often imagine that I’m working on strengthening academic research, or on building massive all-encompassing databases full of peer-reviewed scientific knowledge.

Although I am working on some databases – this isn’t what I actually do most of my time – nor despite what some of my professional colleagues think – is this what I think we should be doing.

Development is a complex business, if it weren’t we would have gotten further along in solving the world’s problems before now. One common reason cited as to why we haven’t done better is that we don’t have enough data, and we don’t have enough evidence.

A number of remedies are commonly proposed to help address this:

1. Collect more statistical data – more surveys, more administrative data collection. More recently we have started to say that we need more real time data collection.
2. More research – more academic studies, more randomized controlled tests, more papers published, papers published more quickly.
3. More evaluation – we need to more systematically evaluate more of our programmes to understood what worked and what didn’t and what lessons we can learn. We need to use better evaluation techniques.
4. More, bigger and more open databases – we often acknowledge that a lot of research has already been done or data collected, but that it is not easily available as it is stuck behind pay walls, fragmented and not well disseminated or easily searchable. To address this we strive to make big well-organized mega-databases that are the preeminent knowledge sources on their particular topic, and advocate for more free access to data and research.

Guess what – I actually agree that all these things are worthwhile. I mean how couldn’t I? BUT – too many people seem to believe that if we keep collecting more and more data, do more and more research and evaluations and make more and more comprehensive databases, then we will have everything we need to do evidence based development work. Basically, if we look hard enough, the truth is out there…

There are a couple of reasons why I don’t agree with this:

• There are limits to how much evidence you can collect
• There are other important dimensions to knowledge that are actionable, yet tend to get overlooked when we take too strong a focus on “evidence”

Firstly the limits of what knowledge you can collect. In developing county contexts in particular, it can be difficult and expensive to come by high quality, timely and relevant data. Existing data collection systems are often weak, and while they can be developed there are still limits in terms of accessibility of marginalized populations and cost of developing surveys to them to such an extent that they can provide the data needed to answer many of the development policy concerns we have.

Similarly for research, there are a large number of potential questions that we would like to address, but availability of data, costs, time and limitations in the research methods themselves mean that there are a lot of questions that can’t be answered in a sufficiently timely manner for the development of policies and development programmes.

Evaluation also is limited in that it can be very costly, yet only tell you part of what you need to know in terms of whether a programme was effective and why.

One particular challenge for any knowledge related work is that of generalizability. To what extent can the results of a study of evaluation be generalizable to other contexts and other timeframes and how much do they tell you about what you should do and what will work in the future.

Another important limitation of “evidence” is that even when it exists and is fairly clear (which for the reasons stated above frequently isn’t the case), it often isn’t sufficient to motivate policy makers, politicians, families etc. to take action. Any findings or recommendations also need to be contextualized to  the local culture and to the power relations of the situation where you are trying to use the evidence. People often choose to interpret evidence in a way which supports their current beliefs, are not necessarily going to use peer-review in a reputable journal as their benchmark on whether to trust the source, and may not accept advice they don’t like that they perceive may weaken their current influence or power.

None of this means that data, research and evaluation aren’t needed. But it does mean that they are not enough. So what’s missing?

An important aspect of knowledge transfer and change is personal relationships. Most people don’t have time or the skills to examine all the available evidence first hand. This means they rely on the opinion of others whom they trust. Similarly standard methods for collecting, storing  and disseminating research often have little impact with people being too busy to seek out the evidence they need, or to even develop the skills to do so. Again people frequently ask others rather than access the evidence directly themselves.

Also there is a whole range of knowledge that isn’t captured by research, that of personal experience. Often you can understand a situation, and describe it to share it with others, but you can’t back it up with scientific research (a trivial example is that I’m pretty sure I know the quickest way to walk to the station in the morning- but I have neither measured it nor timed it).  It might be that it would be too expensive and difficult to prove it through research, or that by the time you know the answer, it would already be too late. Some knowledge is in the form of skills or even instinct which doesn’t easily lend itself to being formally captured at all. This type of knowledge is known in the business as tacit knowledge. Here is a handy diagramme that explains the difference between the two (link to original).

So, in order to take advantage of the part of knowledge that lies below the surface (the part which isn’t “evidence” in the formal sense), then you need to take other approaches. These can involve using tools to try to capture some of what is currently hidden to make it more shareable (tools such as after action reports, end of assignments reports, self-reflection exercises, lessons learned, story telling etc.) and through approaches that make it easier for people with shared knowledge interests to find each other, trust each other, share with each other and collaborate (through approaches such as knowledge fairs, communities of practice, social networking, cocreation).

In fact I find that the most interesting, and promising work I do in the area of knowledge management is not about evidence at all – but is about the social dimension to knowledge. What I need to do is make a better case for this with my colleagues  – but then I’m sure they are going to ask me to show them the evidence!

For all the talk about overhead expense ratios being a bad way to decide if a charity is any good, there aren’t many tools available that help donors do better evaluations. There are organizations like Philanthropedia, GiveWell, Root Cause, New Philanthropy Capital and GreatNonprofits, which offer information, but as a group they only cover a tiny sliver of organizations. Charity Navigator is overhauling their rating system in order to move beyond overhead expense ratios, but even they only rate 5,500 of the literally million-plus nonprofits in the country.

So what to do if you are wondering if your local after school tutoring program or a nearby homeless shelter is worthy of support? Last month, I published a piece in the Chronicle of Philanthropy that offered five questions that donors can ask nonprofits, but I intentionally designed the questions to apply to a huge range of organizations.

A great new tool has now been launched by GiveWell. Designed as a Do-It-Yourself Evaluation kit for the every day donor, GiveWell offers cause specific questions that donors can pose to nonprofits they are considering supporting.

Rather than trying to design some sort of quantitative system of analysis, GiveWell rightly recognizes that the average donor can learn a great deal about a nonprofit by asking a series of important, but relatively simply questions.

An example:

Issue Area: K-12 Education

GiveWell’s Overview of Issue Area

Example Organizations: Children’s Scholarship Fund, KIPP, Teach for America

Questions:

• What do you do to improve K-12 education? What is your relationship with the school? Do you work within it or outside it?
• What academic literature on education – particularly randomized controlled trials – exists on the type of intervention you are conducting?
• Who is targeted by your activities? What are the requirements for participation? In the case of over-subscription, how do you determine who gets in?
• Have you done a randomized controlled trial of your program? If not, are you planning to? If not, why not?
• Have you collected any systematic data on student satisfaction / retention? How happy are students with the program? What do they wish were improved? How often do they drop out and for what reasons do they drop out?
• To what extent do you stay in touch with students? Have you systematically collected (and can you share the reports on) information regarding any outcomes, particularly test scores & graduation rates?
• Have you tried to assess the impact of your program on later life outcomes, compared to how participants would have done without the program?
• How much has been spent on this program? How many students have been served?
• How would your activities change if you had more revenue than expected? Less? Would more revenue translate directly into more students served, and up to what point?

GiveWell is generally focused on assessing the evidence of impact: the strength of information about program results and the degree to which they prove that the program is working. I tend to be more focused on assessing organizational performance: the strength of the information about how the nonprofit operates and the degree to which that information indicates that they are performing at a high level. What I like about GiveWell’s questions is that they can be used to at least begin to explore both dynamics.

Personally, I think that unless a donor has any specific expertise in the issue area, it is less important that they try to determine if the answers to these questions prove that the nonprofit’s programs work and more important that they assess their overall interaction with the nonprofit around these question. Did the nonprofit seem prepared to address the questions or did they feel like they would need to do special work to find your answers? When they answered negatively to questions about the availability of evidence, did they attempt to convince you that those reports were too expense/not important, or did they share an enthusiasm for obtaining that sort of evidence over time?

The current GiveWell tool kit includes issue specific questions to ask organizations across a range of almost 100 different areas both domestically and internationally.

You can explore the new offering here.

While the took kit is designed for donors, I think the questions are highly useful for nonprofits as well. Organizations within any of the issue areas covered by GiveWell should consider sharing the questions with their management team and board and discussing the extent to which they feel they can answer them. Ideally, board members and management team members should all be able to answer these questions on their own, since most of them get at core dynamics at the nonprofit rather then seeking statistical details.

Probing Questions All Donors Should Ask Before Making a Significant Gift

By Sean Stannard-Stockton | Chronicle of Philanthropy

One of the holy grails of nonprofit evaluation is to be able to compare nonprofits with different missions. Concepts like “social return on investment” strive to measure how much “good” an organization is creating, regardless of whether it is a soup kitchen or a job-training program. Given the difficulties of comparing the results of different types of organizations, it makes more sense for potential donors to ask a specific set of questions of all organizations.

Successful programs often look quite different from one another. However, high-performing organizations, those that have the ability to carry out successful programs, have similar characteristics. These organizations base their programs on research about what works, actively collect information about the results of their programs, systematically analyze this information, adjust their activities in response to new information, and focus all their energy on producing results.

To figure out whether nonprofits meet those standards, donors can ask specific questions that will help them decide whether a group is worth supporting. Each answer must be interpreted in the context of the nonprofit’s operations.

For instance, it would be unreasonable to expect a small grass-roots organization to present extensive evidence supporting its answers. However, no matter how big or small the group, donors can still assess whether an organization has the proclivity to become a high-performing group and whether it has put in place all that it can to move toward that goal. Even most large organizations do not possess all of the attributes of a high-performing organization. It is important that donors not let the perfect be the enemy of the good.

With that caveat in mind, here are the questions that donors should ask when they are considering a significant gift.

On what research or evidence did the organization design its programs? Whether you are evaluating a local after-school tutoring program or a global disaster-relief program, a high-performing nonprofit should be able to speak about the evidence and research that shaped its programs. While many organizations have not conducted extensive evaluations of their programs’ results, all programs should at least be based on knowledge about what works. When a nonprofit is exploring an unproven approach, it is critical that the program is treated as a research project to test an idea—and that donors are told that.

What information does the nonprofit collect about the results of its programs?For-profit organizations can track their revenue and expenses to determine exactly how much profit they are producing. Nonprofits need to track not only their financial transactions but also the social results that their programs achieve. The relevant information will vary at different organizations. However, all high-performing nonprofits should be making a consistent effort to collect the information that they believe is most relevant to measuring progress toward their goals.

How does the organization systematically analyze the information it collects? It is not enough simply to collect information; the whole point of gathering data is to better understand a situation. Whether information analysis takes the form of sophisticated statistical analysis or simply regularly scheduled reviews and discussion among the board and staff members, high-performing nonprofits should diligently attempt to understand the meaning of the information they collect.

How has the nonprofit adjusted its activities in response to new information?Unfortunately, knowing what should be done is not enough to spur every organization to action, just as knowing that eating less and exercising more are the keys to losing weight does not guarantee a successful diet. Nonprofits must be ready and able to adjust their activities as needed. High-performing nonprofits should be able to discuss specific instances in which they responded to new information by stopping or significantly altering their activities.

Does the organization have an absolute focus on producing results? In the business world, the intended result—a profit—is also the fuel that sustains the organization. In the nonprofit world, program results don’t usually pay the rent. That means that nonprofits are at risk of giving higher priority to fund raising and other revenue-generating activities than producing program results. Producing the revenue needed to run an organization is critical, but it is a means to an end. Sustaining an organization is useful only to the extent that it enables the delivery of program results. High-performing nonprofits should be able to speak convincingly to their absolute focus on results.

Plenty of nonprofit groups can show financial success. But donors need to ask the probing questions that will make sure that dollars flow only to organizations that turn financial resources into program results.

Sean Stannard-Stockton is chief executive of Tactical Philanthropy Advisors, in Burlingame, Calif., and author of the Tactical Philanthropy blog. He is a regular columnist for The Chronicle of Philanthropy.

The phrase “next Nurse-Family Partnership” was a reference to the nonprofit Nurse-Family Partnership, which is widely viewed as being one of the best examples of a high impact nonprofit with strong evidence.

So today I’m happy to feature a guest post from Peggy Hill, Nurse-Family Partnership’s Chief Strategic Relations Officer, in which she recommends how we can give rise to more nonprofits with strong evidence of impact. Who better to discuss how to give rise to the next Nurse-Family Partnership than the NFP itself?

By Peggy Hill

The Nurse-Family Partnership® stands as a widely recognized example of careful program development, scientific testing, and extensive, disciplined implementation in community settings. We are now confident that transformational changes of public health importance are possible when this program of nurse home visits to first-time mothers living in poverty is conducted properly. That is why we often hear the philanthropic call to invest in the “next NFP.”

Nurse-Family Partnership has a leadership position in evidence-based practice, and is eager to share lessons learned. We also continue to face new challenges. In fact, some of these new challenges are even more difficult as we enter into the arenas of policy-making, finance, inter-governmental relations, and complex public-private partnerships. We continue to need philanthropic support to go to significant scale with effectiveness.

What can philanthropy and government do to increase the odds of success in moving evidence-based programs into broader practice? We have learned three things that are powerful and could be focal points.

1. Invest in preparing a program’s host agencies and personnel to implement research-based programs properly. Different evidence-based programs have different requirements, and their practitioners need specific skills to be competent. New pilots practice with flight simulators before they are made responsible for multi-million dollar equipment and hundreds of lives. Let’s assume it’s no easier when expecting people to fix the most intransigent ills in our society.

New host agencies need to acquire accurate knowledge about how to implement a program successfully; they need sufficient funding well-suited to the program’s design; and they need guidance to recruit capable staff and help them become expert. And all programs need a data system to track performance so staff can figure it out quickly when things go wrong. (And things will go wrong.) Invest in these critical efforts to lay strong foundations for program operation.

2. Set policy in a way that fosters practice excellence and good outcomes. Rules and money shape behavior. Rules can be good. They lend consistency and can set standards that define success. They can also force application to standards that are over-broad or irrelevant to what particular program models need to do to produce desired outcomes. What makes the most sense is for policy language to specify the outcomes that funders want to achieve, and require grantees to specify the nature of the intervention they want to use, the rationale and evidence for that intervention’s effectiveness, and precisely how they will hold themselves accountable for implementing that intervention well and achieving intended results.

3. Design evaluation of evidence-based programs based on specific knowledge of what each program requires to accomplish its outcomes. Be realistic about the limitations of any evaluation. Evaluation design is important because the results are used to inform policy, practice and future funding. And what gets measured tends to be what gets done. Attend to indicators of good program implementation before you expect good outcomes. Measure what is most important for each program’s success, not what is easiest, or a least-common-denominator across very diverse programs, because the resulting data may not be relevant or sufficient to inform decisions. Be realistic about what a small evaluation budget can buy in terms of methods and rock-solid conclusions.

The road ahead…

H.L. Mencken was quoted as saying, “For every complex problem there is a solution that is simple, neat, and wrong.” Dr. Del Elliott and his team at the Center for the Study and Prevention of Violence at the University of Colorado reviewed more than 800 studies of violence prevention programs. They found 11 programs that worked, and 19 more that looked promising. The painful reality is that many social programs simply don’t make a big difference, for a whole host of reasons. Those that do make a difference tend to have clear, powerful core principles made operational through well-articulated intervention strategies; and equally well-crafted implementation supports in practice, policy and financing. When elegantly integrated at scale in the hands of diverse, competent practitioners, they can produce the impacts we all want to see.

Yesterday I wrote about the importance of nonprofits tracking their outputs, outcomes and impact to the best of their ability. Reader Nick Temple of the School for Social Entrepreneurs commented:

“[Nonprofits should see outputs, outcomes and impact]… not just as ‘reporting’ to donors, philanthropists and investors of all types, but also an important part of improving an organisation’s work: we need to prove that what we are doing is making a difference, but we also need to improve those services wherever possible, learning from what we’ve done so far.”

This point of view is one that I agree with completely and one that I think needs to be more broadly accepted for nonprofits to fully embrace tracking their results.

In our current system, funders typically decide what sorts of reports they want to see from nonprofits. Nonprofits then have to create these reports for each funder. These might be elaborate, custom reports for major foundations or it might require spending time responding to the idiosyncratic requests of an individual major donor.

I believe that nonprofits are best positioned to know what information is most important for them to track. The most relevant information is that which reveals how well a nonprofit is performing and the degree to which its programs are making a difference. This information should be mission critical for the nonprofit, even if no funders are asking for reports. It is also the information that funders should be most interested in receiving.

This means that high performing nonprofits should be tracking their outputs, outcomes and impact for their own purposes and funders should be seeking to support these sorts of high performing nonprofits. The reporting burden then becomes much lighter on the nonprofit and specialized reporting might only be available for very large donors.

For too long, the conversation around measuring and tracking results has been framed as a sort of finger wagging campaign by the funding side of the table who often waste their grantees time by requesting information that is of little relevance and which ends up sitting on the shelf at the foundation.

But nonprofits perpetuate this dysfunction by not bothering to track mission critical outputs, outcomes and impact.

This is why my definition of a high performing nonprofit focuses on how they interact with the data they track. While nonprofits may often view the process of collecting this data as a waste of time, that is only because the dysfunctional grantee/funder relationship drives nonprofits to report on information with little value to their work. But if a nonprofits leads the creation of their data tracking process, the result is a better run organization, that has more impact and which can far more easily fulfill the understandable reporting needs of funders.

High performing nonprofits…

• …base their programs on research about what works.
• …actively collect information about the results of their programs.
• …systematically analyze this information.
• …adjust their activities in response to new information.
• …operate with an absolute focus on producing results.

Most donors, regardless of the vocabulary they use, want their donations to produce results.What characterizes “results” may be very different to various donors. Sometimes the desire to see results can lead donors to seek indicators, like low overhead expense ratios, that are actually bad proxies for results. Sometimes the “result” a donor seeks might simply be public recognition. But believing that donors do not seek results is akin to believing that they would be just as happy throwing their money in the trash.

So I’d like to explain the three words that are used to describe the results of nonprofits; Outputs, Outcomes and Impact. Instead of being confusing jargon, outputs, outcomes and impact actual mimic very common sense approaches to achieving results in everyday personal efforts such as working out to improve your health (more on this later):

• Outputs: These are the activities done by the nonprofit. The meals served by a soup kitchen are outputs.
• Outcomes: These are the observed effects of the outputs on the beneficiaries of the nonprofit. The degree to which the meals served by the soup kitchen reduce hunger in the population served by the soup kitchen.
• Impact: This is the degree to which the outcomes observed by a nonprofit are attributable to its activities. The impact of the soup kitchen is the degree to which a reduction of hunger in the population they serve is attributable to its efforts. While a soup kitchen might serve a lot of meals and correctly observe that hunger is subsequently less prevalent in the population it serves, the reduction in hunger might simply be attributable to an improving economy, or a new school lunch program or some other activities that are not part of the soup kitchen’s efforts.

While outputs, outcomes and impact might sound like jargon, they are an extremely useful vocabulary for discussing the results of a nonprofit. They help illustrate the tradeoff between the difficulty of obtaining knowledge and the value of the knowledge.

• Outputs: Relatively easy to count. Are often selected based only on a theory about what is helpful. If the outputs counted do not lead to anything meaningful, the “results” are meaningless.
• Outcomes: More difficult to measure. Do measure the observed effects of the nonprofit’s activities. However, the observed outcomes may not actually be due to the nonprofit’s activities. If so, the “results” are meaningless.
• Impact: Very difficult to measure. Requires some form of analysis which attempts to hold static the effects of other influences. This is the gold standard because the results are proven.

A quick example to make all of this personal, practical and relevant:

If you are trying to get in shape, you may well try to lose weight.

• Output: The amount of calories you consume minus the amount of calories you burn.
• Outcome: Your observed weight.
• Impact: The degree to which your level of health is improved by your weight loss.

Outputs are obviously a good place to start. You can’t legitimately argue that you are trying to lose weight if you pay no attention to the amount you eat and workout.

Outcomes are better. You can directly observe your weight and know the degree to which your diet and exercise plan appears to be having an effect.

Impact is best. You may be restricting calories, working out and observing a reduction in your weight. But only a rigorous evaluation can eliminate the potential effects of outside influences. Maybe your weigh loss is due to a serious undiagnosed medical issue or some other influence that has nothing to do with your efforts.

I can hear the urgent comment of a large group of readers already. “The outputs, outcomes and impact of our efforts are far more difficult than tracking what you eat and how much you weigh!” That’s 100% correct. No one is saying that measuring results is easy.

But here’s the fantastic secret of tracking outputs, outcomes and impact. Rigorous studies have proven that simply attempting to track activities results in impact across many different fields. Just tracking calories consumed, results in weight loss that is attributable to the tracking itself, not the specifics of the diet and exercise regime followed.

This is why it is so important that nonprofits that want to actual make a difference at least attempt to track their activities in some way. While the process can be difficult and only becomes more difficult as you move from counting inputs to proving impact, just the effort to begin the process will increase the measurable or immeasurable impact you are having.

You can read Part II of this post here.

What if instead we tried to build a more effective philanthropy that was designed for real life, human donors?

The first step of course would be to actual ask donors what they wanted. So today I’m glad to be able to point to the newly released Money For Good report from Hope Consulting. The report was underwritten by the Hewlett and Rockefeller Foundations as well as a number of other well known organizations.

The Money for Good report tries to answer three questions, 1) how can we increase charitable giving from individuals, 2) how can we increase donations to the highest performing nonprofits and 3) how can we realize the potential of impact investing.

The report opens with this quote:

“It is our nature to see the world based on our own context, experiences, and points of view. People in all walks of life struggle with this bias every day. How can a new product fail when you and your cohort believed that it was a great idea? The need to understand the world as it is – not as we wish it were – has caused primary market research to become a multi-billion dollar industry.

The motivation behind the Money for Good project was to seek the ‘voice of the customer’ for charitable giving and impact investing.”

The report is in depth and I encourage you to read the whole thing. But today I want to focus on a particular chart that explores the opportunity to “improve the quality of giving,” which is the critical question motivating the movement for a more effective philanthropy.

The percentages refer to the amount of donors who currently do the things mentioned in the dark blue boxes. So if we want donors to give based on relative performance, to make donations that are as effective as possible, we need to help donors move from the left to the right on this chart.

The first, promising sign is that 85% of donors do care about the performance of the nonprofits they support. But only 32% of donors do any research at all to determine if the nonprofits they support are actually any good. This finding is similar to the survey that kickstarted the recent posts here and on other blogs about whether donors care.

The second major gap is how to encourage donors to use quality information. Charity Navigator proved over the last decade that donors can be encouraged to research nonprofits. The problem of course is that the overhead expense ratio metric they popularized (the Money For Good report confirms that overhead expense is the #1 most important piece of information for donors who do research) is very low quality in determining effectiveness. But Charity Navigator should be noted for their hugely successful campaign to close the Care vs Act Gap. Today Charity Navigator is working to implement a major overhaul of their rating system that they hope will increase the quality of their ratings (disclosure: I’m on the advisory board for this effort).

The Money For Good report argues that closing the “Care vs. Act” Gap and the “Quality Information” Gap so that donors go from just caring about the effectiveness of nonprofits to actually doing high quality research, means:

1. Creating many initiatives that address these issues simultaneously.
2. Providing simple information donors will use.
3. Pushing information to donors.
4. Building board awareness around select key messages.

The final goal, closing the Good vs Best Gap, or helping donors support the best nonprofits not just any that are doing a good job, is in some ways a wholly different goal. According to the Money for Good report, only 15% of donors even state that they care about supporting organizations that are better than others at achieving their goals.

To me, this means that we have two separate issues at stake,

1. We need to help the 85% of donors who already care about supporting effective nonprofits act on this interest by creating high quality information that is designed in ways that donors actual want to use it.
2. We need to work to increase the percentage of donors who care about maximizing the effectiveness of their giving by supporting the best nonprofits.

The first issue is one of meeting a pre-existing demand. The second is about creating demand. Both can be done. Both sorts of issues are tackled all the time in various markets.

But the one thing we can’t do is simply build philanthropic products and services for imaginary “rational” actors who are seeking massive amounts of data and metrics, and love to run Excel spreadsheets before every donations. These donors are as imaginary as Vulcans.

I took the position that donors should focus on high performing nonprofits, but argued that a “robust” definition of high performance was needed. Since then I’ve refined my robust definition of a high performing nonprofit to mean organizations that;

• base their programs on research about what works,
• actively collect information about the results of their programs,
• systematically analyze this information,
• adjust their activities in response to new information,
• and operate with an absolute focus on producing results.

My central point in the debate was that while high impact nonprofits were great, they were few and far between and their proven impact only showed what had happened in the past. A high performing organization, as long as it executed on the five points above, was well positioned to deliver impact in the future.

Now, Holden Karnofsky of GiveWell, a participant in the first round of the debate, has offered up a new set of arguments on the issue. Holden writes:

“When I look at the three charities that I consider to be most “blue-chip” (our three highest-ranked charities), I don’t see a path of a “strong organization that eventually figured out what to do and whether it worked. ” Rather, I see organizations that stayed as small as possible – or didn’t exist – until they had strong evidence of impact for their basic approach. They built their choice of programming into their DNA, as much as they could, from day one.

The reigning consensus seems to treat “evidence of impact” as a late (or at least potentially late) step in the development of a nonprofit, but in fact it has been the first step for the strongest nonprofits I know of.

This makes intuitive sense to me as well.

• Finding “approaches that work” is fundamentally a research challenge, and probably requires a completely different skill set from running an organization well.
• Once an organization is “up and running,” it may become a very poor environment for a good impact evaluation. To me a good impact evaluation is one that has a real chance of demonstrating failure, and the stakes may simply be too high for an organization that has already built up significant funds, donors, clients, stories, staff, habits, etc.

The truth is that if an organization wants to become “high-impact,” there are already proven approaches for it to choose from; if it wants to investigate an approach that isn’t yet proven, it can (like VillageReach) stay at minimal size and essentially act as a “research project.”

I highlight Holden’s post because I think it reveals a frequent, incorrect assumption about high performing nonprofits. In his argument, Holden conflates “high performing” with the idea of a large nonprofit that is operating at some level of scale. But if you look at my five point definition of high performance, you can see that it describes an approach applicable to organizations of any size.

Critical to my definition is that a high performing organization base its programs on research about what works. So when Holden says that, “Finding approaches that work is fundamentally a research challenge, and probably requires a completely different skill set from running an organization well.” I have to say that I agree. It may be that large, high performing organizations can also invent new approaches to achieving social impact, but the process is more of a research and development activity.

To be clear, people on both sides of the debate care about achieving impact. Those who favor high performing nonprofits do so only because they believe that these types of organizations are most likely to achieve impact over time. If impact where a perfectly quantifiable and repeatable process, then I would agree with the high impact crowd. But unfortunately I think that achieving impact is an imperfect process. Just because a program has been proven to work in one set of circumstances does not mean that the same process will work elsewhere.

A high performing nonprofit is one that is able to implement well researched, proven approaches (whether the research was conducted internally or externally) with a high degree of fidelity to the original approach. But then also has the organizational capability to track their programs as they scale, determine what is working in their specific case, course correct as needed and constantly look to improve.

There is great research about what works across fields as diverse as business, psychology, weight loss and baseball. But finding an approach that works does not mean that anyone can easily implement it. If this were so, then every well read person would be a successful business person, with a healthy psychological profile, who was in great shape and played professional baseball as a hobby.

But truly successful individuals (and organizations are just collections of individuals) make it a habit to base their activities on evidence about what works, be mindful about what is working for them, reflect on what changes they might make and have an absolute focus on achieving results.

Ask any venture capitalist and they’ll tell you that they bet on people, not business plans. I think a focus on high performing nonprofits is simply an extension of this concept that emphasizes betting on organizations, not research. Great business people craft solid business plans and great organizations depend on credible research. But smart investors and smart donors should place their bets on the very best people and organizations and never fall for the seductively idea that research and planning can guarantee a positive result.

I couldn’t agree more! However, that only furthers my point. My point is that even in a field where collecting evidence and determining what really works is relatively easier than doing so in philanthropy, the rise of evidence-based decision making is only a recent phenomenon.

The point of my post is that philanthropy’s limited embrace of evidence-based grantmaking is not unusual. The history of evidence-based decision making in medicine gives me hope because it suggests that philanthropy is on the right trajectory towards more fully incorporating evidence into our decision making process. The history of medicine shows that we still have a long road ahead, but we are on the right track.

However, I certainly did not mean to imply that philanthropy would adopt the same standards of evidence as the medical community. Each field needs to establish its own best practices of establishing what works.

When thought about this way, you can see why I care about holistic approaches to understanding the social sector. While statistical analysis or stakeholder reviews or expert reviews or financial analysis all offer a view into the truth, each are limited in their ability to understand the full truth. So it was with great interest that I read an article in the Foundation Review about the Knight Foundation’s use of investigative journalists to do impact assessment.

Starting in 2006, the Knight Foundation hired investigative journalists to write “clear, honest, factual and transparent quarterly reports” that attempted to “draw out something close to the truth” about the results of the Foundation’s programs. The journalists were given full access to internal Knight Foundation documents and employees, and free range to pursue the story of what really happened through interviews with grantees, community leaders and anyone else of the reporters’ choosing.

The Foundation Review article, authored by the former vice president of communications for the Foundation, concludes that the resulting reports were “more direct, even critical, than any prior Knight Foundation attempt to evaluate and assess,” and that they “produced deeper looks into the intent and outcome of major initiatives.”

Great journalism helps the community it serves to discover the truth about the reality of the world in which they live. Great journalism recognizes that the truth is not simply the facts, but is dependent on the context in which the facts exist. Importantly, journalism uses narrative to communicate its findings.Stories are not distractions from the truth, they are the vehicle through which humans have always come to understand the truth.

The role of investigative journalists in helping us understand financial markets and politics is well understood. Books like Too Big to Fail, by New York Times reporter Andrew Sorkin set the standard for helping us understand “what really happened” during the financial crisis, just as All The President’s Men by Woodward & Bernstein got at the truth of “what really happened” during Watergate. Importantly, these books and the newspaper articles by their authors are understandable by non-professionals and in fact are extremely popular with laypeople.

Too often when we think about “impact assessment” we think of statistical analysis and dense research reports. These techniques have huge importance, the are often the foundation on which great journalism draws, but we must not fail prey to the idea that research reports and statistical analysis represent “the truth”. The truth of reality is a multifaceted mosaic that all humans seek to understand. We don’t do ourselves any favors when we discuss philanthropic impact as if it is an arcane science through which only specially trained professionals can determine what works.

Philanthropic impact is nothing more than the extent to which philanthropy actually makes a difference. The field of impact assessment is nothing less than the search for The Truth.

At the heart of the portal is a McKinsey report that examines best practices in social impact assessment (disclosure: I reviewed the report prior to release and participated in a feedback group for McKinsey). The report is well done and I found the matrix on page 11 illuminating in the way it describes appropriate objectives based on the type of social intervention and the maturation stage of the solution.

The new learning portal also features a number of video interviews with Jed Emerson of Blended Value, Tris Lumley of New Philanthropy Capital, Brian Trelstad of Acumen Fund and myself.

In this video, the four of us discuss what social impact assessment is and why it is important.

In this video, McKinsey’s team interviews me about my view that a mass movement towards effective philanthropy is underway.

McKinsey’s Learning for Social Impact portal and report are good additions to the field and I encourage you to explore the site.