By Holden Karnofsky

Imagine that, as a donor, you’re considering the following pitch:

“For 12 years our campaign has delivered a proven preventive health measure. For the $23 million we’ve spent cumulatively, we’ve directly and demonstrably saved several million lives.”

If those claims checked out, and you found these accomplishments to resonate strongly with your values, would there be any reasons not to support this campaign with your donation? If you got this pitch today, there would be a big reason not to support it: the above describes the smallpox eradication campaign, which ended in 1979 as it became clear that it had achieved total success, with no smallpox cases left to prevent.

Donating today to smallpox eradication would clearly be silly, but it’s just an extreme version of what we believe many donors are doing: giving to causes and programs they believe in, without checking whether those programs need – and can use – more money.

In trying to find the best charities to recommend to donors, we’ve found that this question of how (and whether) a charity is going to use additional revenue (as opposed to the funds it’s already spent or budgeted) is one of the hardest questions to answer.

We call it the problem of room for more funding.

There are several reasons that a charity may not use additional funds the same way it used past funds, or the same way as the donor hopes.

• A successful program can rely on many factors besides money, such as skilled labor, political support, and (as in the case of smallpox) appropriate target populations whose problems are suited to the solutions being offered. For example, we have argued that the surgery charity Smile Train appears to have a shortage of skilled surgeons, not a shortage of funds, for its core program.
• The programs donors want to fund don’t necessarily match the programs charities want to carry out. Thus, a charity may focus on one program in solicitations, when its intent is to use donations for another program or simply add them to reserves. We have seen charity representatives make explicit statements to this effect.
• Charities may be able to execute different activities with an additional $20 million vs. an additional $1,000 – the total amount of additional funding they’re getting matters. For example, see our review of the Schistosomiasis Control Initiative (SCI).

Understanding how your $100, or $10,000, affects a charity with a multimillion dollar budget is a challenge we’ve been struggling with for years. One answer we don’t think works is restricting/earmarking your donation. A charity can formally honor an earmark while effectively using your donation to fund other activities.

However, it does seem possible to answer this question using a very different method: requesting scenario analysis that asks how a charity’s activities would change at different levels of total unrestricted funding. This approach allows us to check back later and see the extent to which actual activities were in line with actual funding (and, if there is a discrepancy, to have a conversation about it).

Applying this approach has led us to much more concrete, and sometimes surprising, picture of charities’ room for more funding. For example, earlier this year we found that KIPP, a celebrated and GiveWell-recommended charity, has no short-term funding needs at the national level – but KIPP Houston has urgent needs.

We believe that this sort of scenario analysis can be practically, and relatively cheaply, produced by just about any charity. However, in practice we have found it very hard to get this sort of information because there doesn’t seem to be anyone else asking for it. We think that one of the most constructive things a broad-based charity accreditation service could do would be to push large numbers of charities to generate and share "room for more funding" analysis. For our part, we do this analysis for any charity that is a contender to become one of GiveWell’s top charities.

Dealing with "room for more funding" as a major grantmaker
One of the reasons that "room for more funding" is a relatively neglected topic is because it doesn’t seem to be as much of an issue for major funders. It may be difficult for an individual to understand the impact of their $100 or $10,000 – but a foundation able to commit $1 million up front can design and fund a project all on its own. However, we’ve lately been exploring (via GiveWell Labs) a giving style more similar to major grantmakers’, and at this early stage, we’re still finding the problem of "room for more funding" to be relevant. In brief,

• We prefer to fund projects that are primarily designed and proposed by charities, rather than imposing our own strategy.
• But with some of the stronger potential projects, we find ourselves wondering whether another funder would step in if we didn’t. By funding a strong project, are we causing it to happen, or merely saving another funder money that they’ll spend on other projects following their own priorities?

We have a lot of work to do in terms of understanding this issue, but it seems to us that "trying to fund what other funders won’t" can lead to some strange situations (for example, imagine two funders who each suspect the other of being interested in a project, and thus both hold out trying to see whether the other will fund it). Arguably the best philanthropist is not the one who funds the best projects – it’s the one who funds the best projects that wouldn’t get funded otherwise, and who therefore looks to a naïve outsider like a merely mediocre philanthropist.

We feel that the issue of room for more funding is severely under-discussed and under-appreciated. If and when we can raise the issue’s profile, we expect to see a lot of progress on solutions for gaining clarity into a group’s room for more funding: helping people find the best next thing (instead of the best past thing) to fund.

By David Bonbright

When we study the existing approaches and tools used to understand the results of organizations that seek to create social value, we can see that something new and exciting is slowly being born.

For the first time, a performance measurement approach is gaining traction that takes seriously those who are meant to benefit from their work. This approach, known as constituency voice or beneficiary feedback, uses techniques familiar to us from customer satisfaction surveys, combining them with lesser known, but tremendously important social accountability methods such as community scorecards and deliberative democracy.

But even as constituency voice methodology proves its value, it has to break through the cost barrier. By definition, the people who tend be the least heard are those who lack access to technology, and even literacy, and therefore won’t be easy to reach through the Internet. To engage them one needs to conduct survey interviews, either face-to-face or by telephone. To give respondents the confidence that their feedback will not be used against them, it needs to be collected by people who are trusted to be independent. In the commercial world, this means independent professionals of the type that the customer satisfaction firms employ. And this requires the most coveted type of funding for grant-funded organizations – unrestricted income. Few nonprofits have enough of this kind of money.

For the past two years Keystone Accountability has been developing and testing a very low cost model that utilizes volunteers to collect constituency feedback from those who are meant to benefit from social programs. The work so far has utilized undergraduates engaged in service learning programs. There are three major benefits to university service learning for feedback collection. First, universities can hold and sustain arm’s length, high trust, ongoing relationships with civil society organizations. Second, they have the capability in their faculties to provide statistical and survey expertise to support the volunteers. Third, they have a self-renewing supply of volunteers.

It is clear from Keystone’s pilot work with undergraduates from two American universities – St. John’s University and Georgetown – that undergraduate volunteers can undertake meaningful feedback data gathering.

In Keystone’s pilots, the student volunteers worked with the charities and their constituents to develop short questionnaires. They then conducted interviews with random samples of the charities’ constituents. Finally, with the help of their professors, they analyzed the data and presented it to their “clients”.

The students rated the experience of feedback data gathering higher than the usual volunteering activities as they enjoyed the constituent interviews and could see that they were making a real contribution to the organizations. In one case, a charity discovered that it needed to change one of its basic services – and it subsequently did so. Another discovered a demand for a new service that it also later met through a new program. In all cases, the feedback generated new insights and opportunities for the charities to improve.

There is much to learn as we take these fledgling efforts forward, but this just might be the beginning of a repurposing of volunteering to contribute to one of the most difficult problems in philanthropy and social investing – enabling everyone in the ecosystem to have in real-time empirically rigorous data on constituent satisfaction.

By Valerie Threlfall

The power of feedback is almost universally acknowledged. Every day, millions of people turn to reviews written by end-users in Yelp, Trip Advisor, GreatNonprofits, and Global Giving to inform their purchasing and philanthropic decisions. The end-user’s perspective on their experience is seen as an asset just about everywhere—except in the world of foundations.

Consider a recent report published by the Center for Effective Philanthropy (CEP). The State of Foundation Performance: A Survey of Foundation CEOs, updates work CEP completed a decade ago, looking at the types of data foundations use to assess their performance. While the report shows that foundations are increasingly looking to a range of indicators, notably surveys of grantees, only about a quarter of funders gather feedback from beneficiaries in any way. This, despite analysis showing that it may be among the most valuable types of feedback. CEOs of foundations who collect beneficiary feedback rate themselves as having a better understanding of the progress their foundation is making against its strategies and a more accurate understanding of the impact the foundation is having on the communities and fields in which it works.

As head of CEP’s YouthTruth initiative, which gathers comparative feedback from students – the ultimate beneficiaries of education reform efforts, I have seen the power beneficiary feedback can have in informing local school change efforts. Consider this recent post about The California Endowment’s YouthTruth experience. So the question I ask myself is this: How can we better understand why foundations have not seriously embraced beneficiary feedback, and how can we encourage its use?

To elevate beneficiary perspectives, I am convinced that at least three things need to happen.

· We need to believe as a field that beneficiaries’ perspectives matter.

· Foundation leaders need to understand the connection between beneficiaries’ perspectives and good strategy – to acknowledge that this data has the potential to inform funders’ theory of change.

· And we need easy-to-use tools to collect and make sense of the feedback as we acquire it.

The third area is where we have made the most progress, as we are increasingly well-positioned with new tools for efficiently gathering and processing beneficiary feedback. Now we need to push ourselves on these first two points.

With regard to the first, we need to address the biases that convince some in the field that beneficiary feedback is less valuable or objective than the opinions of say, grantees or donors.

Daniel Stid describes the situation this way on Bridgespan’s blog:

“The working assumption of the suppliers [of nonprofit human services] and their funders alike is that they know what is best for the individuals and families they are serving… reducing [beneficiaries] to passive recipients… And why bother [gathering feedback] when your beneficiaries can’t really take their business elsewhere?”

With regard to the second point, can foundations expect to learn anything that will enable them to make different decisions from beneficiary feedback? Speaking from my own experience, we don’t know for sure, but we are getting close.

YouthTruth was started by CEP, with support from the Bill & Melinda Gates Foundation to test this very notion. To date, YouthTruth has gathered perspectives from more than 71,000 students from 164 schools nationwide. With this dataset, we have been able to examine the degree to which there are systematic differences in student perspectives based on the type of school they attend, as one example.

Moreover, we have the ability to identify differences in student perceptions — for instance, about their futures — across groups implementing a common strategy, and provide insights about the relative effectiveness of various efforts. This to me seems like helpful and actionable data that can inform funders’ decision-making. While this is one isolated example, we need many more experiments to push our thinking about the benefits and limits of beneficiary feedback.

I would argue that since beneficiaries are ultimately the people we are all trying to serve, we need to take dramatic steps to take their perspectives into account.

We need to ask each other the tough questions about why this feedback is not being more widely used and challenge ourselves to determine the potential of beneficiary feedback once and for all.

In 2010, the New Yorker published an article titled The Truth Wears Off, that looked at the existence of the “Decline Effect”, the seemingly inevitable way that when scientific studies are repeated over and over, they tend to follow a path of diminishingly positive results.

In the article, Jonah Lehrer wrote:

“The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.”

Lehrer’s article incited a flood of angry letters and emails claiming that he was undermining scientific research and drumming up a controversy that doesn’t exist. However, my reading of the article (and Lehrer’s responses to his critics) suggests a much more modest claim is at the heart of his article.

Human knowledge is an evolving concept.

For all the perceived precision of a large study “proving” that something is true, the fact remains that over time our understanding of facts and truths change.

Lehrer explains a number of reasons behind what is know as the “Decline Effect”. Taken together, much of the issue has to do with human cognitive biases and behavioral issues in the way we process information. For instance, Lehrer points to the way that scientific journals seem to greatly prefer to publish studies that prove something to be true, so scientists have a significant incentive for their studies to find these results.

But even if you peel away all of the messiness of the human practice of scientific study, you are still left with the idea that seeking truth is a process not an event.

Lehrer writes:

“The decline effect is actually a decline of illusion. While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that.

[The Decline Effect is so troubling] Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true.”

Now the silly way to react to the decline effect is to turn our backs on science and decide that if it cannot present us with the unquestionable truth, then it doesn’t work (this is the message that some of Lehrer’s critics through he was pushing). The more useful way to react is simply to understand that the concepts of “truth” and “fact” are far less rigid and concise than we tend to treat them. The search for truth, for “proven programs” will not end some day when we finally, finally, finally discover the real truth.

As Ralph Waldo Emerson said, “Life is a journey, not a destination.”

What this means for nonprofits and funders who want to direct their resources towards programs that actually work is that doing so will always be a continuous process. There will never be a a final, definitive study that tells us the “truth” of the best way to eradicate poverty, to end obesity, to give every individual the opportunities they deserve.

But that doesn’t mean we should lower our ambitions nor reject the scientific process. Instead, I think that Ted Cadsby, writing in the Harvard Business Review had it right when he argued in favor of adopting a mindset of “provisional truth”:

“Provisional truth requires that we think of our explanations as hypotheses — always subject to replacement based on new information or alternative ways of structuring existing information. Provisional truth means challenging our interpretations with disconfirming evidence and alternative perspectives. Provisional truth does not preclude drawing conclusions or taking action; but it demands that we be skeptical about our first reasonable explanations in the realm of complex problems. It keeps us humble and mentally flexible, constantly asking ourselves if we’ve really got everything figured out and responding, "Probably not."

But of course the scientists among you will recognize that the skepticism embedded in the idea of “provisional truth” is in fact a core aspect of the scientific process. The Decline Effect doesn’t discredit the process of scientific inquiry. Instead it simply lays waste to the fetishism of the scientific process that deludes people into thinking that we can at last completely understand and control our world once we discover “the truth”.

But can five questions answered with a couple of pages of text really help donors understand which organizations are effective?

I think they can.

Last year, I created my own list of “five questions every nonprofit should be able to answer” with the same intentions of Charting Impact; to help funders understand which organizations are effective. In developing my questions and speaking about them with nonprofits and donors, I’ve come to the conclusion that with these types of questions, it is how the nonprofit answers them, not the specifics of the answers that matter.

The fact is, there is no short, simple answer that can prove effectiveness. This is true in the for-profit sector as well. Even in a field where the goal (profit) is easily measureable, there are no reliable, simple data that can answer the question of whether an organization is (and will continue to be) effective. But the Charting Impact questions don’t seek to get nonprofits to provide any specific data. The elegancy of their approach is that the questions give each nonprofit the freedom to answer in the way that is best suited to their particular situation while requiring that they respond in a manner that reveals the organization’s strategic strengths and understanding of the environment in which they operate.’

The questions are:

1. What is your organization aiming to accomplish?
2. What are your strategies for making this happen?
3. What are your organization’s capabilities for doing this?
4. How will your organization know if you are making progress?
5. What have and haven’t you accomplished so far?

By focusing at the level of goal setting and accomplishment, the questions are applicable to any organization trying to achieve any goal.

Given that the projects backers — GuideStar, BBB Wise Giving and Independent Sector — have such reach in the nonprofit sector, I think it is feasible that Charting Impact reports could become standard reports available on most all nonprofit websites for organizations included in the GuideStar database. Encouragingly, unlike so many external review processes that only add to the burden of tasks that nonprofits must accomplish, the Charting Impact report should be relatively simple for effective organizations to complete. Any organization that finds it difficult to complete will find it a highly useful process to go through in their journey towards becoming more effective.

Most importantly, I think the Charting Impact report helps undermine the fantasy that someday we’ll have a simple, quantitative rating system that will answer the effectiveness question. While the report is only a starting point in a donor’s mission to discover which nonprofits are effective, it does start the journey off in the right direction by asking the most important questions in a standard format while allowing them to be answered in a free form format that recognizes the vast differences between different issue areas and levels of organizational development.

PS: For those nonprofits that think foundations should be subject to the same level of accountability that they require of nonprofits, note that the Charting Impact questions can easily be turned around and asked of funders. In fact, the Hewlett Foundation has already published their report.

There has been a long-standing debate among nonprofits and those that benefit from nonprofits’ goods and services about how to measure nonprofit effectiveness. It seems every organization has a different method for evaluating its progress, and yet we all know that communicating successes – and failures – are of the most vital importance to stakeholders and donors. Not only is there greater demand for transparency and accountability in the new giving era, but there is a greater demand for data on which to base decisions. At the end of the day, people want to know how their contributions are being used, and they want to find that information easily.

The Money for Good I research from Hope Consulting showed us that people base their giving decisions on a nonprofit’s legitimacy, impact, efficiency and reputation, and yet very few conduct extensive research to determine find that impact data. And so nonprofits are plagued with the issue of needing to demonstrate their impact quickly and concisely in order to paint an accurate and complete picture of their nonprofit to the public.

With this in mind, GuideStar joined two leaders in the nonprofit sector — BBB Wise Giving Alliance and Independent Sector — to harness the collective intelligence of more than 200 respected nonprofit and philanthropic leaders on the subject. We came up with five simple questions that, if answered thoroughly and to the best of a nonprofit’s knowledge, will produce a powerful tool to help nonprofits communicate more effectively with their stakeholders. This initiative is aptly named Charting Impact.

Charting Impact isn’t just another good way to share an organization’s impact, but is rather a new, common format that all organizations can use, regardless of type, size, or mission. Charting Impact allows the sharing of assessment information in a concise, standardized way, which ultimately enables new collaborations and resources to be directed to effective organizations.

Led by an advisory group of trusted leaders on nonprofit effectiveness, we designed and extensively tested the initiative in order to:

• Encourage people to invest their money, time, and attention in effective organizations.
• Help your organization highlight the difference you make.
• Position your organization to work with and learn from other organizations.
• Help your organization sharpen your approaches to making a difference.

If done correctly, a “Charting Impact Report” should consist of an organization’s responses to the five questions, with three external reviews that help validate the self-reported information. Once an organization has used the online interface to complete its report, its responses will produce a document with a unique URL that will be shared on the Charting Impact website, on the nonprofit’s GuideStar profile, on the reports of charities participating in BBB Wise Giving Alliance evaluations, and – in the future – with other websites and information sources about nonprofits. Participants will receive guidance about promoting their Charting Impact Report once they publish their report.

We encourage organizations to answer these questions and share their report on their own website and through their own media channels. We believe the program can go a long way in both helping nonprofits demonstrate their effectiveness and helping donors get the clear and concise information that they require about a nonprofit. 

What do you think of Charting Impact? We want to hear your feedback in the comments here or at ceoblog@guidestar.org.

By Colby Dailey

I believe that philanthropists’ willingness to pay for social returns positions them to play a unique, catalyzing role in scaling the impact investing industry. In a recent paper published by the Federal Reserve Bank of San Francisco, my co-author Ben Thornley and I laid out a new framework for understanding impact investors.  Four aspects of that framework are helpful for thinking about what role we, as philanthropists, can have in scaling the industry:

1. Impact investors are best described as having a "willingness to pay for social returns" rather than, as they are more typically described, as having financial-first or impact-first motivations;

2. Investors themselves are the best drivers for improving performance measurement as they alone know their actual preferences for social returns;

3. By innovating broadly, investors can drive improvement in performance measurement; and

4. Social performance measurement is a key to drawing additional investment into the impact investment universe.

Rather than thinking about investors in the typical impact-motivated/financially-motivated categorization, we can describe them as having a "willingness to pay for social returns".  I would argue that nearly all investors have some willingness to pay for social returns, whether it be the value they place on the return, such as meeting their mission objectives, or the price they pay, such as potentially forfeiting market rate returns.  We see this at work any time an investor chooses not to invest in a company whose activities she disagrees with for social reasons; and the growth of social responsible investing demonstrates that such willingness to pay is a prevalent, mainstream investor behavior.

Thinking about impact investors in this way enables us to see the impact investment industry as a whole, having a diverse core of investors, rather than as a polarized industry locating investors with competing motivations at each end.  In this framework, philanthropic investors, by definition, are the most willing to pay for social returns as they seek little to no financial return from their investments. 

Impact performance tools and practices are the way by which investors can truly understand and express their own preferences for social return, i.e. their willingness to pay for it. By extension, the better investors understand the social return or the impact of their investment, the more able they will be to make informed – and more – investments.  Thus more effective tools, practices and reporting of returns are keys to drawing investment into the industry.  Because investors alone know their actual preferences for social return, they are the best drivers of improvements to performance measurement.  In this vein, because donors prefer all or almost all social return, they can be a very powerful group of impact investors – they can bring substantial investment capital to the industry and drive and support efforts to better measure impact and report social return.

As diverse investors innovate broadly in social performance tools and practices, social performance measurement will become more effective and potentially more standardized.  We are already seeing some convergence as investors in certain sectors – such as the environment or education, for example – use similar metrics to show impact.  As investors improve performance measurement, they will be more able to disclose their impact, thus providing valuable information to the field, giving more investors what they need to gauge their own preferences for social return and invest accordingly. As donors begin to make more impact investments and disclose their presumably large social returns in those sectors, they will inform other investors’ preferences for social return.

Admittedly the challenge of measuring impact is daunting.  However, I would posit that the philanthropic sector is uniquely poised to scale the impact investing industry and lead the field forward.  Not limited by preference for financial return, donors can deploy capital as impact investors targeting a wider range of impact investments than other groups of impact investors.  Moreover, by demanding, leading and supporting innovation in the social performance measurement tools and practices that disclose the large social returns of their investments, donors who make impact investments provide valuable information to the field, giving more investors what they need to invest in the efforts that change the world.

By Bob Ottenhoff

Over the years, GuideStar has taken on the monumental task of providing transparency into the nonprofit sector, with care to maintain a sense of neutrality, providing tools and data users can access to guide their decisions.

Now, we come to yet another crossroads in our history. Late last year we adopted a new strategic plan with an ambitious long term vision that embraces a world where:

• Information that supports quality decision-making flows through the sector at low cost for providers and users,
• Nonprofits know their performance, have tools to improve their performance, and are able to share their data freely and widely with donors at low cost
• Donors are positioned to target their giving to the organizations that best fit their giving preferences – impact or passion-driven
• Best fit nonprofits benefit from easier assessment of performance with higher donations from purposeful donors
• The marketplace shifts toward purposeful giving (vs. responsive), and the cost of fundraising and grantmaking decline

GuideStar’s acquisition of Philanthropedia and Social Actions combines entrepreneurial talent, new technologies and adds deeper, wider and better data to our information stream. We are excited about being able to achieve efficiencies and scale faster by engaging the talent and platforms already created by both these entities.

For some time, we have been looking for a way to utilize the in-depth analysis conducted by private foundations. Philanthropedia uses a sophisticated process and network to tap into the “wisdom of experts” to analyze the effectiveness of nonprofits working in a particular cause area. We currently capture some of this information and display it via our TakeAction platform. This year they have added nine more cause-areas, have approximately 200 organizations in the top tier recommended category, 1,400+ organizations with reviews from several experts and a new “start-up” category with profiles and an expert comment on why these 25 or so start-ups have high potential. Their 2011 goals include adding ten additional cause areas.

We plan to use GuideStar’s data and processing assets to enhance Philanthropedia’s research platform and to scale Philanthropedia’s mission. Their crowd-sourcing of experts, coupled with the Great Nonprofit reviews (“crowd-sourcing of users”) we have invested in at GuideStar over the last two years, will give us a comprehensive compilation of third-party nonprofit reviews. Philanthropedia is led by co-founder Deyan Vitanov, with a fulltime staff of two and a part-time staff of two overseas product development engineers. We’re looking to apply their entrepreneurial spirit and energy into major roles in building GuideStar Labs, developing GuideStar’s TakeAction as our online laboratory and strengthening our data sources and content partnerships.

As for Social Actions, we see GuideStar’s future tied to what Peter Dietz said in his blog recently: “The stewards of databases are no longer just asked to open up their datasets, but to make them available in such a way that they link with other data repositories by design.” This linkage is key to integrating data in a way that makes sense to different user groups. We have engaged Peter and Christine Egger to help us take their Social Actions concept to the next level. We built TakeAction with open source technology, for all the same reasons Social Actions built their platform on this same commonly accepted, scalable and rapid application technology. As a first step, we will integrate Social Actions into our TakeAction micro site and use it as part of our GuideStar Lab for testing out new tools and services.

Social Actions also provides a new layer of social innovators to our growing data network. We envision leveraging the online communities and resources that Social Actions brings, by weaving in their action-oriented resources into a user’s decision-making process. We believe development and adoption of a standard protocol that will simplify cross-platform communication among transaction providers will have huge strategic implications to the sector. We’ll also use Social Actions Open Action XML API and semantic web technology to expand our data sharing services.

Powered by these new acquisitions, we’ll be one step closer to turning data into information and knowledge and making GuideStar information widely distributed through a range of partners to all purposeful donors.

You can see this dynamic working if you think about whether you would rather be given $100 today or be given it in one year. The difference in the value of the present and the future is called the “discount rate”. In all likelihood you would probably prefer to get $100 in one year rather than $10 today. But there is some point at which you become indifferent. If you are indifferent between receiving $90 today or $100 in a year, then you are using a 10% discount rate (approximately).

However, for some time I’ve been wondering about how this fundamental tool for financial decision making works when it comes to philanthropic decision making.

Which would you rather have, a crime free society and clean environment today or in the future? Clearly, sooner is better. But what if it is either/or? Anyone with a child is going to tell you they’d rather their children live in that better world than that they receive the benefit of it. This preference for the future is hardwired into our species as becomes evident any time a parent puts their children’s needs ahead of their own. It is so hardwired into our neurological makeup that adult humans will often choose to put themselves in physical danger if needed to protect the physical health of any child, let alone their own.

Why does this matter? Because positive discount rates erode the value of investments. If I offer you an investment which promises you a 10% return on your money, you’ll be indifferent if you have a discount rate of 10%. The lower your discount rate, the more attractive any given investment is. The higher the rate, the higher the prospective returns need to be to attract your attention.

But if humans have a preference for social good to occur in the future or to benefit our children rather than us, then it means we have a negative discount rate. It means, we’d rather $100 of social value occur in the future rather then $110 of social value occur today.

If this is true, it means that social returns on investment are radically higher than we might otherwise suppose because, unlike financial investments, we actual prefer that social impact accrue to our children and their children.

But this concept can’t be entirely correct. Many forms of social good are not discrete events, but rather conditions within which life occurs. Wiping out polio, as Bill Gates is urging the world to join him in focusing on, is not a discrete event. If it is wiped out now, more lives will be saved and less suffering will occur than if it is wiped out later.

This means that maximizing the value of philanthropy rests on three elements 1) we should seek to create long lasting positive conditions rather than just discrete, socially good events (which is one way to express a preference for correcting the root cause of a problem rather than treating the symptoms), because doing so will create social value that accrues to more people, 2) we should seek to achieve those conditions as quickly as possible because doing so will allow the benefits to accrue to the maximum number of people but, 3) when forced to make a trade off between activities that create benefits in the future vs. benefits today, we should prioritize the future given the hardwired negative discount rates that we use to value the future vs. the present when it comes to social impact.

There are probably a number of ways to reconcile these three elements. There certainly are some creative tensions between them and so different people may choose different approaches. But for me, they suggest that as a general rule, philanthropy should focus on creating impact that:

• Is lasting rather than fleeting (working on causes rather than symptoms), but;
• Is implemented as quickly as possible, but;
• Prioritizes the needs of the future over the present.

In other words, donors and foundations should: Act quickly to create lasting solutions that prioritize the needs of the future.

To my way of thinking, the best way to get that done is for donors to make rapid grantmaking decisions in support of nonprofit organizations that seek to tackle the underlying causes of social problems, using grants which build the long term sustainability of their grantees while recognizing that truly effective interventions take time to achieve success.

“From the donors’ perspective we need something more than metrics and annual reports that cements the organization’s commitment to social good. Nonprofits have typically relied on (been given a pass by) their corporate structure and the public accountability of their boards. I’d say both of these are not strong enough. Social businesses, particularly those that meet the standards of a B Corporation, are beginning to document and commit themselves contractually to social good. This is a step toward mission insurance, but the B Corporation* standards are written mostly to protect business owners, then the investors, then the customers. We need something that will work for downstream investors or donors.

And there is nothing available to ensure customers – be they the borrowers from a microfinance organization, the teen employed by a job training company, or the working mom seeking a new credential to better her job prospects – that the services they are getting are being designed and delivered with a measurable, enforceable commitment to bettering lives and communities.”

Lucy’s proposal was made in the context of her thinking about how we might insure that social enterprises, both for-profit and nonprofit, actually achieve their social mission. But I was struck by the way that the concept illustrates so perfectly the current paradigm shift in philanthropy.

Whereas philanthropy has historically been about the gift as a moral act by the donor, today more and more people think about philanthropy in the context of the results achieved by the gift. This historical view is embedded in the legal obligations of nonprofits. Nonprofits actually have no legal requirement to achieve results. Instead they need only have a socially beneficial mission and ensure that no benefit incur to the donors that support them.

The focus on results and the emergence of for-profit social enterprises turn this understanding of philanthropy on its head. What we need now is a legal understanding of social businesses that requires them to achieve social beneficial results while recognizing that having financial benefit accrue to supporters may in fact increase and accelerate social impact.

In a comment on Lucy’s post I wrote:

“We know there are documented examples of nonprofit programs creating harm (for instance it is well documented that the DARE program to "keep kids off drugs" has no positive impact and some evidence suggests that it increases drug use).

In the world of consumers, this would be stopped. If a drug designed to decrease the drug cravings of addicts was found to have no effect and possibly to increase cravings, it would be recalled immediately.

While I think it might be hard to "insure" that a social mission is met (just like it would be hard to "insure" that a for-profit became and stayed profitable), it seems very doable to "insure" to some degree of conviction that social programs meet minimum standards. The meeting of this hurdle would then flow upstream and greatly increase the chance of mission success.”

Today, the tax protection that nonprofits enjoy is dependent only on them having a social mission and ensuring that no financial benefits flow to donors. And yet we wonder why “evidence-based” social activity, performance management and the measurement of results is so rare.

While the challenges of measuring results are real and should not be underestimated, imagine the positive transformation that would occur if tax protection was dependent on the achievement of socially positive results and whether investor/donors benefitted from the organizations activities took a backseat from a regulatory standpoint.

Which entity should receive tax benefits? A nonprofit whose programs exacerbate the problems of the people they intend to help, or an organization – either nonprofit or for-profit – that actually helps people?

“Thanks for the provocative commentary on the report Sean. We couldn’t agree with you more about the importance and power of leverage. In the coming years, we believe that the most effective funders will increasingly be able to leverage outsized impact and trigger resources that are far larger than their own, whether it’s by influencing other funders (institutional or individual), catalyzing government support, mobilizing public will and opinion, or stimulating markets to sustainably provide services…

But I wonder, though, whether the main distinction you are really highlighting is less about “internal” versus “external” leverage (many of the examples of Acting Bigger from our report—especially in the section on “Leveraging Others’ Resources”—are primarily about influencing the flow of external resources, after all), as much as it’s about “direct” and “indirect” impact.

And in that respect, your point remains a powerful one. In the end, the greatest impact of the Women Moving Millions campaign, for example, may not be in exactly how many dollars the initial gift ends up leveraging directly, but instead in how the effort raises awareness about women’s and girl’s issues and how it shifts the behaviors and attitudes of generations of women who are empowered and encouraged to give in the years to come.”

Gabriel makes a good point in questioning the internal/external vocabulary I used. My point in the post (and in the presentation I gave on Friday to the Social Venture Partners conference) was to suggest that the impact of a funder’s actions may be felt much more largely away from the targeted area intended by the funder.

In the post I wrote:

“I believe that in many cases, external leverage is FAR more powerful than internal leverage. But internal leverage is far more measurable. We can see this dynamic at work in Warren Buffett’s gift to the the Gates Foundation. The internal leverage (Buffett’s utilization of the Gates Foundation’s resources to distribute his giving) is significant. But as I’ve argued many times, the big impact of Buffett’s gift is the external leverage generated through the richest people in America modeling a changing attitude to philanthropy.”

Gabriel suggests that direct/indirect might be better than internal/external. I hesitate to use this vocabulary, because I think the “external”/”indirect” impact of Buffett’s gift, for example, is actually very direct. His actions are directly influencing the way Americans think about philanthropy. It may be difficult to measure and it may be difficult to control, but it is very real.

But Gabriel is right that internal/external isn’t very good either. Clearly a funder can leverage internal sources of leverage (non-monetary resources) or external sources (other fund flows). These actions can be directed towards precise, targeted activities or towards influencing larger spheres.

Regardless of what we call it, an important take away for me is that a real tension can exist between measuring impact and maximizing impact. The relationship between the two is neither completely correlated or negatively correlated. Often measuring impact can be a key tool for maximizing it. Yet I think it is important for philanthropy to grapple with the idea that those things which can be measured are not always the most important things. As we strive towards measuring impact, it is critical that we don’t end up thinking too small. Achieving measureable objectives while ignoring opportunities to create much larger impact simply because the larger impact isn’t measureable would be a tragedy of epic proportions.

In their excellent report What’s Next for Philanthropy, the Monitor Institute argues that philanthropy needs to “act bigger”, which I find to be a far more evocative phrase than simple seeking leverage.

The report offers five keys to acting bigger:

• UNDERSTAND THE CONTEXT. Philanthropists are just one part of a larger ecosystem of actors, and in almost all cases they will need to engage many different stakeholders if they hope to address today’s pressing social and environmental challenges.
• PICK THE RIGHT TOOL(S) FOR THE JOB. Funders have a wide range of assets—money, knowledge, networks, expertise, and influence— that can be deliberately applied to create social change.
• ALIGN INDEPENDENT ACTION. Coordinating resources toward common goals no longer must mean developing a consensus based collaborative. Funders are forging new ways of working together, from learning groups to strategic alignment networks, that fit different purposes and circumstances, allowing individual funders to aggregate and amass resources of all kinds and effectively “punch above their weight.”
• ACTIVATE NETWORKS. Although the individual grant is the typical unit of analysis for most foundations, the success of any grant or organization is rarely sufficient to move the needle on a complex problem. Advances in network theory and practice now allow funders to be much more deliberate about supporting and participating in networks and in thinking about how the collective impact of a coordinated portfolio of grants can produce more significant change.
• LEVERAGE OTHERS’ RESOURCES. In addition to finding new ways of working together toward collective goals, some funders are also exploring how they can use their independent resources as levers to catalyze much larger streams of funding and activity from other sources.

I think these are all good recommendations.But also I think that they are focusing on what might be called “internal leverage”. I wonder if there might be significantly more power in what might be called “external leverage”.

Internal leverage are steps taken to increase the targeted impact of a grantmakers own grants. One example would be the practice of venture philanthropy, where the funder provides significant non-monetary support to their grantees.

External leverage are steps taken to increase system wide impact through influencing others. One example might be GiveWell, which publishes detailed analysis of nonprofits and recommends a select few organizations with the hope that other donors/funders will support these organizations.

A critical distinction between the two approaches is the degree to which the results of the leverage can be measured. Because internal leverage hopes to enhance the impact of targeted activities, a donor can strive to understand how their efforts to use leverage improved results beyond what would have happened if they had simply made a grant. While measuring results of any kind is often difficult, internal leverage simply tries to create more/bigger results of the same kind that are sought by a non-leveraged grant.

External leverage on the other hand adds fuels to all sorts of fires in ways that might be unknowable. GiveWell, for instance, tries hard to track how much money is influenced by their work. For example, they ask donors to tell their recommended nonprofits if they make a grant based on their research. But not only is it certain that GiveWell is influencing grants that they never hear about, GiveWell is also creating impact through helping change the way people think about giving.

I believe that in many cases, external leverage is FAR more powerful than internal leverage. But internal leverage is far more measurable. We can see this dynamic at work in Warren Buffett’s gift to the the Gates Foundation. The internal leverage (Buffett’s utilization of the Gates Foundation’s resources to distribute his giving) is significant. But as I’ve argued many times, the big impact of Buffett’s gift is the external leverage generated through the richest people in America modeling a changing attitude to philanthropy.

External leverage is fundamentally underappreciated by philanthropy. We see this dynamic at work in the minuscule amount of foundation grants made to support improving the philanthropy of individuals (even though individuals give 6.6 times the amount that foundations give each year). While grants made to support the philanthropy of individuals are highly leveraged and can help move significant sums of philanthropic capital, the impact is seen throughout the system rather than within a target defined by a single grantmaker.

The Monitor report makes another very important point. It argues that philanthropy must get comfortable with the “creative tensions” that exist between different types of valid goals.

If external leverage truly is more powerful than internal leverage, then philanthropy might need to get comfortable with the creative tension between measuring impact and maximizing impact.

Probing Questions All Donors Should Ask Before Making a Significant Gift

By Sean Stannard-Stockton | Chronicle of Philanthropy

One of the holy grails of nonprofit evaluation is to be able to compare nonprofits with different missions. Concepts like “social return on investment” strive to measure how much “good” an organization is creating, regardless of whether it is a soup kitchen or a job-training program. Given the difficulties of comparing the results of different types of organizations, it makes more sense for potential donors to ask a specific set of questions of all organizations.

Successful programs often look quite different from one another. However, high-performing organizations, those that have the ability to carry out successful programs, have similar characteristics. These organizations base their programs on research about what works, actively collect information about the results of their programs, systematically analyze this information, adjust their activities in response to new information, and focus all their energy on producing results.

To figure out whether nonprofits meet those standards, donors can ask specific questions that will help them decide whether a group is worth supporting. Each answer must be interpreted in the context of the nonprofit’s operations.

For instance, it would be unreasonable to expect a small grass-roots organization to present extensive evidence supporting its answers. However, no matter how big or small the group, donors can still assess whether an organization has the proclivity to become a high-performing group and whether it has put in place all that it can to move toward that goal. Even most large organizations do not possess all of the attributes of a high-performing organization. It is important that donors not let the perfect be the enemy of the good.

With that caveat in mind, here are the questions that donors should ask when they are considering a significant gift.

On what research or evidence did the organization design its programs? Whether you are evaluating a local after-school tutoring program or a global disaster-relief program, a high-performing nonprofit should be able to speak about the evidence and research that shaped its programs. While many organizations have not conducted extensive evaluations of their programs’ results, all programs should at least be based on knowledge about what works. When a nonprofit is exploring an unproven approach, it is critical that the program is treated as a research project to test an idea—and that donors are told that.

What information does the nonprofit collect about the results of its programs?For-profit organizations can track their revenue and expenses to determine exactly how much profit they are producing. Nonprofits need to track not only their financial transactions but also the social results that their programs achieve. The relevant information will vary at different organizations. However, all high-performing nonprofits should be making a consistent effort to collect the information that they believe is most relevant to measuring progress toward their goals.

How does the organization systematically analyze the information it collects? It is not enough simply to collect information; the whole point of gathering data is to better understand a situation. Whether information analysis takes the form of sophisticated statistical analysis or simply regularly scheduled reviews and discussion among the board and staff members, high-performing nonprofits should diligently attempt to understand the meaning of the information they collect.

How has the nonprofit adjusted its activities in response to new information?Unfortunately, knowing what should be done is not enough to spur every organization to action, just as knowing that eating less and exercising more are the keys to losing weight does not guarantee a successful diet. Nonprofits must be ready and able to adjust their activities as needed. High-performing nonprofits should be able to discuss specific instances in which they responded to new information by stopping or significantly altering their activities.

Does the organization have an absolute focus on producing results? In the business world, the intended result—a profit—is also the fuel that sustains the organization. In the nonprofit world, program results don’t usually pay the rent. That means that nonprofits are at risk of giving higher priority to fund raising and other revenue-generating activities than producing program results. Producing the revenue needed to run an organization is critical, but it is a means to an end. Sustaining an organization is useful only to the extent that it enables the delivery of program results. High-performing nonprofits should be able to speak convincingly to their absolute focus on results.

Plenty of nonprofit groups can show financial success. But donors need to ask the probing questions that will make sure that dollars flow only to organizations that turn financial resources into program results.

Sean Stannard-Stockton is chief executive of Tactical Philanthropy Advisors, in Burlingame, Calif., and author of the Tactical Philanthropy blog. He is a regular columnist for The Chronicle of Philanthropy.

The phrase “next Nurse-Family Partnership” was a reference to the nonprofit Nurse-Family Partnership, which is widely viewed as being one of the best examples of a high impact nonprofit with strong evidence.

So today I’m happy to feature a guest post from Peggy Hill, Nurse-Family Partnership’s Chief Strategic Relations Officer, in which she recommends how we can give rise to more nonprofits with strong evidence of impact. Who better to discuss how to give rise to the next Nurse-Family Partnership than the NFP itself?

By Peggy Hill

The Nurse-Family Partnership® stands as a widely recognized example of careful program development, scientific testing, and extensive, disciplined implementation in community settings. We are now confident that transformational changes of public health importance are possible when this program of nurse home visits to first-time mothers living in poverty is conducted properly. That is why we often hear the philanthropic call to invest in the “next NFP.”

Nurse-Family Partnership has a leadership position in evidence-based practice, and is eager to share lessons learned. We also continue to face new challenges. In fact, some of these new challenges are even more difficult as we enter into the arenas of policy-making, finance, inter-governmental relations, and complex public-private partnerships. We continue to need philanthropic support to go to significant scale with effectiveness.

What can philanthropy and government do to increase the odds of success in moving evidence-based programs into broader practice? We have learned three things that are powerful and could be focal points.

1. Invest in preparing a program’s host agencies and personnel to implement research-based programs properly. Different evidence-based programs have different requirements, and their practitioners need specific skills to be competent. New pilots practice with flight simulators before they are made responsible for multi-million dollar equipment and hundreds of lives. Let’s assume it’s no easier when expecting people to fix the most intransigent ills in our society.

New host agencies need to acquire accurate knowledge about how to implement a program successfully; they need sufficient funding well-suited to the program’s design; and they need guidance to recruit capable staff and help them become expert. And all programs need a data system to track performance so staff can figure it out quickly when things go wrong. (And things will go wrong.) Invest in these critical efforts to lay strong foundations for program operation.

2. Set policy in a way that fosters practice excellence and good outcomes. Rules and money shape behavior. Rules can be good. They lend consistency and can set standards that define success. They can also force application to standards that are over-broad or irrelevant to what particular program models need to do to produce desired outcomes. What makes the most sense is for policy language to specify the outcomes that funders want to achieve, and require grantees to specify the nature of the intervention they want to use, the rationale and evidence for that intervention’s effectiveness, and precisely how they will hold themselves accountable for implementing that intervention well and achieving intended results.

3. Design evaluation of evidence-based programs based on specific knowledge of what each program requires to accomplish its outcomes. Be realistic about the limitations of any evaluation. Evaluation design is important because the results are used to inform policy, practice and future funding. And what gets measured tends to be what gets done. Attend to indicators of good program implementation before you expect good outcomes. Measure what is most important for each program’s success, not what is easiest, or a least-common-denominator across very diverse programs, because the resulting data may not be relevant or sufficient to inform decisions. Be realistic about what a small evaluation budget can buy in terms of methods and rock-solid conclusions.

The road ahead…

H.L. Mencken was quoted as saying, “For every complex problem there is a solution that is simple, neat, and wrong.” Dr. Del Elliott and his team at the Center for the Study and Prevention of Violence at the University of Colorado reviewed more than 800 studies of violence prevention programs. They found 11 programs that worked, and 19 more that looked promising. The painful reality is that many social programs simply don’t make a big difference, for a whole host of reasons. Those that do make a difference tend to have clear, powerful core principles made operational through well-articulated intervention strategies; and equally well-crafted implementation supports in practice, policy and financing. When elegantly integrated at scale in the hands of diverse, competent practitioners, they can produce the impacts we all want to see.

One of the comments on the first post (which generated a lively discussion in the comments section) came from Isaac Castillo, head of evaluation at Latin American Youth Center. LAYC is known for their diligent efforts to track their performance (they even consult for other nonprofits) and was recently announced as one of the few pre-selected subgrantees of the Social Innovation Fund. Isaac wrote:

“I actually think measuring outputs, outcomes, and impact is fairly easy and straightforward. The truly difficult part is getting nonprofits to identify the specific things they want to track.”

This of course flies in the face of a lot of thinking about measuring results in the social sector. While I do think that measuring results conclusively can be difficult, it is possible to take a “fairly easy and straightforward” approach to results measurement.

So today I want to rerun a piece on measuring impact from the Mulago Foundation. When I first published this piece, Paul Brest, head of the Hewlett Foundation left a comment saying, “This is excellent. Should be bottled and distributed.” The piece popped up recently when it randomly re-circulated via Twitter.

The Mulago Foundation: how we think about impact

We measure impact because it’s the only way to know whether our money is doing any good. In fact, we don’t invest in organizations that don’t measure impact – they’re flying blind and we would be too. Those organizations that do measure impact perform better and evolve faster, and discussions around measuring impact almost always lead to new ideas about effectiveness and efficiency.

Everyone’s got their own definition of impact and here’s ours: Impact is a change in the state of the world brought about by an intervention. It’s the final result of behaviors (outcomes) that are generated by activities (outputs) that are driven by resources (inputs).

We’re a small shop, so we needed to develop an approach with enough rigor to be believable, but simple enough to be doable. When we work with organizations, we use these five steps to determine impact and calculate bang for the donor buck:

1. Figure out what you’re trying to accomplish: the real mission.

You can’t think about impact until you know what you’re setting out accomplish. Most mission statements don’t help that much. We re-formulate the mission in a phrase of ~8 words or less that includes 1) a target population (or setting), 2) a verb, and 3) an ultimate outcome that implies something to measure – like this:

• getting African one-acre farmers out of poverty
• preventing HIV infection in Brazil

If we can’t we can’t get to this kind of concise statement, we don’t go any further- either because they don’t really know what they’re trying to do or because we simply wouldn’t be able to know if they’re doing it.

2. Pick the right indicator

Try this: figure out the single best indicator that would demonstrate mission accomplished. Ignore the howls of protest, it’s a really useful exercise. Here’s some examples relating to the missions shown above:

• Change in farm income
• Decrease in HIV infection rates

Sometimes that best indicator is doable, and that’s great. Other times you might need to capture it with a carefully chosen – and minimal – combination of indicators. When there is a behavior with a well-documented connection to impact – like children sleeping under mosquito nets – you can measure that and use it as a proxy for impact. Projects that can’t at least identify a behavior to measure are too gauzy for us to consider. Notice that while things like “awareness” or “empowerment” might be critical to the process that drives behaviors, we’re interested in measuring the change that results from that behavior.

We don’t pretend that this method captures all of the useful impacts and accomplishments of a given organization and their intervention. What it does do for us as philanthropic investors is answer the most critical question: did they fulfill the mission?

3. Get real numbers

You need to 1) show a change and 2) have confidence that it’s real. This means that

• You got a baseline and measured again at the right interval, and
• You sampled enough of the right people (or trees, or whatever) in the right way.

There are two parts to figuring this out: the logical side and the technical side. With an adequate knowledge of the setting, you can do a lot by just eyeballing the evaluation plan – looking carefully at the methods to be used to see if they make sense. Most bad schemes have an obvious flaw on close examination: they didn’t get good baseline data, they’re asking the dads when they ought to ask the moms, they’re asking in a culturally inappropriate way. The technical part has mostly to do with sample size, and a competent statistician can easily help you figure what is adequate.

4. Make the case for attribution

If you have real numbers that show impact, you need to make the case that it was your efforts that caused the change. This is the hardest part of measuring impact, because it asks you to be able to say what would have happened without you. When real numbers show there has been a change, a useful thing to ask is “what else could possibly explain the impact we observed?”

There are three levels – in ascending order of cost and complexity – of demonstrating attribution:

1. Narrative attribution: You’ve got before-and-after data showing a change and airtight story that shows that it is very unlikely that the change was from something else. This approach is vastly overused, but it can be valid when the change is big, tightly coupled with the intervention, involves few variables (factors that might have influenced the change), and you’ve got a deep knowledge of the setting.
2. Matched controls: At the outset of your work, you identified settings or populations similar enough to ones you work with to serve as valid comparisons. This works when there aren’t too many other variables, you can find good matches, and you can watch the process closely enough to know that significant unforeseen factors didn’t arise during the intervention period. This is rarely perfect; it’s often good enough.
3. Randomized controlled trials: RCT’s are the gold standard in most cases and are needed when the stakes are high and there are too many variables to be able to confidently say that your comparison groups are similar enough to show attribution.

5. Calculate bang-for-the-buck

Now that you know you’ve got real impact, you need to know what it cost. You can always generate impact by spending a ton of money, but it won’t give good value for the philanthropic dollar and it won’t be scalable (and it probably won’t last). Stick with the key impact you’ve chosen; don’t get sucked into the current trend of trying to monetize every social impact you can think of.

The easiest – and arguably most valid – way to calculate bang-for-the-buck is to divide the total donor money spent by the total impact. In organizations that do more than one kind of project, it is often possible to split out what they spent for their various impacts. Remember that start-ups are expensive and don’t worry so much about their current figures, but do see if their projections for steady-state operations make sense and assume (as we learned the hard way) that they are usually at the way-optimistic end of the scale.

In the end, though, the key to figuring out real impact is an honest, curious, and constructive skepticism. A healthy dose of skepticism – not cynicism – is a gift to doers, funders and the social sector as a whole.