For all the talk about overhead expense ratios being a bad way to decide if a charity is any good, there aren’t many tools available that help donors do better evaluations. There are organizations like Philanthropedia, GiveWell, Root Cause, New Philanthropy Capital and GreatNonprofits, which offer information, but as a group they only cover a tiny sliver of organizations. Charity Navigator is overhauling their rating system in order to move beyond overhead expense ratios, but even they only rate 5,500 of the literally million-plus nonprofits in the country.
So what to do if you are wondering if your local after school tutoring program or a nearby homeless shelter is worthy of support? Last month, I published a piece in the Chronicle of Philanthropy that offered five questions that donors can ask nonprofits, but I intentionally designed the questions to apply to a huge range of organizations.
A great new tool has now been launched by GiveWell. Designed as a Do-It-Yourself Evaluation kit for the every day donor, GiveWell offers cause specific questions that donors can pose to nonprofits they are considering supporting.
Rather than trying to design some sort of quantitative system of analysis, GiveWell rightly recognizes that the average donor can learn a great deal about a nonprofit by asking a series of important, but relatively simply questions.
An example:
Issue Area: K-12 Education
GiveWell’s Overview of Issue Area
Example Organizations: Children’s Scholarship Fund, KIPP, Teach for America
Questions:
- What do you do to improve K-12 education? What is your relationship with the school? Do you work within it or outside it?
- What academic literature on education – particularly randomized controlled trials – exists on the type of intervention you are conducting?
- Who is targeted by your activities? What are the requirements for participation? In the case of over-subscription, how do you determine who gets in?
- Have you done a randomized controlled trial of your program? If not, are you planning to? If not, why not?
- Have you collected any systematic data on student satisfaction / retention? How happy are students with the program? What do they wish were improved? How often do they drop out and for what reasons do they drop out?
- To what extent do you stay in touch with students? Have you systematically collected (and can you share the reports on) information regarding any outcomes, particularly test scores & graduation rates?
- Have you tried to assess the impact of your program on later life outcomes, compared to how participants would have done without the program?
- How much has been spent on this program? How many students have been served?
- How would your activities change if you had more revenue than expected? Less? Would more revenue translate directly into more students served, and up to what point?
GiveWell is generally focused on assessing the evidence of impact: the strength of information about program results and the degree to which they prove that the program is working. I tend to be more focused on assessing organizational performance: the strength of the information about how the nonprofit operates and the degree to which that information indicates that they are performing at a high level. What I like about GiveWell’s questions is that they can be used to at least begin to explore both dynamics.
Personally, I think that unless a donor has any specific expertise in the issue area, it is less important that they try to determine if the answers to these questions prove that the nonprofit’s programs work and more important that they assess their overall interaction with the nonprofit around these question. Did the nonprofit seem prepared to address the questions or did they feel like they would need to do special work to find your answers? When they answered negatively to questions about the availability of evidence, did they attempt to convince you that those reports were too expense/not important, or did they share an enthusiasm for obtaining that sort of evidence over time?
The current GiveWell tool kit includes issue specific questions to ask organizations across a range of almost 100 different areas both domestically and internationally.
You can explore the new offering here.
While the took kit is designed for donors, I think the questions are highly useful for nonprofits as well. Organizations within any of the issue areas covered by GiveWell should consider sharing the questions with their management team and board and discussing the extent to which they feel they can answer them. Ideally, board members and management team members should all be able to answer these questions on their own, since most of them get at core dynamics at the nonprofit rather then seeking statistical details.
7 Comments
I’m ok with all of these questions, except the fourth one: RCTs.
The real reason that an organization such as mine (Latin American Youth Center) hasn’t done more RCTs is because they are expensive. Very expensive. The current RCT we are doing now has a total cost of $580,000. And we are doing it on the cheap.
Most RCTs will cost at least $750,000, assuming you actually want to get a large enough sample size to have credible data.
Only a very small percentage (probably less than 0.5%) of nonprofits will have the finances to do an RCT in their history. And asking this question puts a continuing focus on what many evaluators now consider to be a flawed way to think about evaluating social service programs (particularly ones in multi-service or holistic settings).
That being said, we at LAYC can easily answer all of these questions for our education programs, and for our employment programs, and for our residential housing programs, etc. For us (and many other high-performing non-profits that I know of), it is not a question of program design, capacity, or expertise. It is simply a matter of a lack of funding to undertake an expensive research process.
So if any donor out there is reading this, and really wants to give money to a non-profit, but is concerned about the lack of an RCT, here is a suggestion:
Write a check for $750,000 over 3-4 years to the nonprofit so they can do the RCT.
Put your money where your expectations are. Otherwise, lower you expectations to match what you are willing to fund for evaluation.
Isaac, GiveWell will have to answer this themselves, but I liked that they included the question “If not, why not?” Your answer is a great response! It might not be such a great response for KIPP or Teach for America.
Oh, I know it is GiveWell’s thing, and not yours. I just wanted to point out the flawed thinking (in my opinion) of the RCT question.
We at LAYC could quite literally start at least two more RCTs tomorrow if we were able to obtain the funds. The problem is that no one is willing to pay for an RCT (because they are SO expensive).
And I agree, my answer works for a nonprofit like LAYC with a $15 million annual budget. It doesnt’ work quite as well for an organization that is 10x our size, or that has a massive endowment.
Isaac, our answer is similar to Sean’s.
We purposefully didn’t include a scoring guide with these questions. What answers are appropriate depends on the nature of the nonprofit and its relationship to the donor.
Personally, if I were giving to a U.S. equality of opportunity charity and had no up-close knowledge of different nonprofits, I’d rather give to a more established one with an RCT than a similar one without an RCT. However, if I had some pre-existing reason to be honing in on LAYC (perhaps a personal connection), I’d ask the questions we listed, and I think your answer to the question on RCTs would be a good one.
I’m an outsider to the philanthropy-charity world–I run an adult education program for a community-action agency in the Northeast, serving mainly low-income immigrants.
But I like these questions–a sort of informal and accessible gauge to help potential donors target their giving.
Further, as somebody who’s looking to grow his education program–and who needs funds to do that–I think this battery of questions would be worthwhile for me to consider as I develop my growth plan.
So for me to apply this right now to the work I do, it’s more of an internal forecasting-planning tool than an external evaluation. What other questions should I be asking of myself and my organization?
Thanks for the comment, WritingOutLoud. I wrote my own version of this sort of approach that you might be interested in. You will find it here.
Thanks, Sean. I checked out the other post and will incorporate those ideas into our growth plan–even as it’s in its embryonic stage! I particularly like the point that nonprofits must not only collect and analyze information, but also RESPOND to it.