Rebooting Nonprofit Evaluation Debate

A lively debate about nonprofit evaluation and metrics has been raging in response to my request for input on my meeting later this week with Google.org. However, the conversation has splintered into a debate over whether a systematic, “metric” driven process of scientific measurement is needed, or whether the frame of scientific measurement is “an epistemologically impoverished frame” through which to understand nonprofit evaluation.

I personally believe evaluating nonprofits is mostly about evaluating their output (the social good they produce). Since it is difficult (impossible?) to quantify this output, I think the focus on metrics as a framework for evaluation is misplaced. Metrics can be used, but they should be designed on a case-by-case basis for each situation. That being said, I think the conversation has fallen into the trap of being constrained by historical frames of reference.

I want to have a different conversation.

I’m interested in what information is available to donors who want to evaluate a nonprofit and which of this information is useful. Google.com is mostly a resource that points to information; they don’t tend to create a lot of their own content. So if we imagine a future version of the nonprofit data inside of Google Finance, I don’t imagine it will be some new metric that we design. Instead, it will point to existing information on the web. When I first wrote about nonprofit info in Google Finance, I said I hoped they would not display Charity Navigator ratings (although I would support them noting if a nonprofit had a zero or one star rating since I do believe that a Charity Navigator rating at this level is a significant red flag)

So the conversation I want to have is what information do readers think that donors should consider when evaluating a nonprofit? Then secondly, where or how can this information be captured online so that it can be displayed in Google Finance?

Open Invitation to Foundation Employees

I realize that if you work at a foundation, you may not want to jump into a conversation that involves telling another foundation what to do. However, the conversation we’re having here is really important and would not be complete without the input of the army of program officers (ie. Nonprofit evaluators) that read this blog. So please consider commenting anonymously (just let us know you’re a program officer) or comment publicly and realize that we’re having a broad conversation about nonprofit evaluation that goes beyond Google.org and Google Finance

Open Invitation to Nonprofit Employees

A conversation about nonprofit evaluation would not be complete without the input of the nonprofits being evaluated. What information do you, as nonprofits, what donors looking at when they evaluate you? It could be that someday the Google Finance website about your organization becomes the top ranked search result on google for your nonprofit. What do you want on that page?

70 Comments

  1. Phil Steinmeyer says:

    It’s currently difficult to get at information that charities provide on 990s.

    You can sometimes find them at a charity’s own web site, but going to the site of each charity you’re interested in and hunting for this info is cumbersome.

    There are, I think, other ways to find this info online, but none of them that I know of are particularly easy. (If anyone *does* know of an easy way to get this info, please chime in.)

    Anyways, 990 info seems to fit Google’s paradigm – indexing/collecting/pointing to existing info, rather than creating new info. Yes, I know that there are problems with 990s, but they’re better than nothing, IMO.

  2. Philippe Bradley says:

    I can’t see how the Google Finance approach to presenting this data can work, given how heterogeneous the data is.

    Some people want to donate to an organisation that’s very efficient at reuniting refugee families, others, to organisations that are effective at driving cancer research.

    Unlike stock prices and fundamentals, there is so much diversity to the important data you want access to when evaluating NGOs, and none of that data is (at the moment) being output by these organisations as XML (with all the necessary metadata), so Google Finance is going to have a horrible time just bringing the important info onto their pages, let alone deciding how to present these different metrics, LET ALONE allowing sorting and searching by metric.

    It just doesn’t seem like the right approach to presenting useful information about nonprofits. Something along the lines of Metacritic.com is much more appropriate in this day and age, but as you point out, Google doesn’t like curating information using humans, not algorithms (though with Google Knol, maybe that attitude is slowly changing).

  3. Phil, 990 data is clearly important. But it only addresses the cost side of the equation, it says nothing about the social good being produced. It is like an income statement for a for-profit company that leaves out revenue. Certainly the cost data is useful (and easy for Google or others to aggregate and display), but it I think it is a bit like saying that when you design a new car, “you must have wheels”. Well, yes… but what else?

    Guidestar makes this information available already. Maybe Google should buy Guidestar?

  4. Philippe, why couldn’t Google display Great Nonprofit (MetaCritic for nonprofits) rankings? (Perla Ni, who’s commenting in this thread is founder of Great Nonprofits).

    Google Knol does sound like a promising part of this equation. Let’s not constrain this conversation too much by Google’s historical choices. Also, let’s not assume that every nonprofit needs to be listed in the Google database. Currently, the Hoover’s data they’re using only includes larger nonprofits. In the for-profit markets we only expect data to be available for public companies. Most for-profit companies are private and it is almost impossible to find much info on them. Maybe nonprofits of a certain size should be included? Or maybe nonprofits should be allowed to elect to be “public”? In this case then they would be expected to provide certain data just as public companies are required to.

  5. Phil Steinmeyer says:

    Sean, believe me, I understand the limitations of 990 type data.

    But you’re asking what can Google contribute to the field with minimal human intervention (i.e. primarily using algorithms). While 990s are not good for all purposes, they are good for *some* purposes – there is useful information to be gleaned from them, IMO.

    Basically, I agree with Philippe that the problem of measuring charities doesn’t lend itself particularly well to the type of solution that Google usually excels at.

  6. Teri Behrens says:

    I’ve been a lurker on this blog and this conversation is enticing me out in the open.

    As much as I’d like to rejoin the evaluation discussion, I’m going to frame the question as, “How can Google help donors choose organizations to support?”

    Seems to me the first step is to help donors themselves be clear about what it is they want to support — what they are trying to accomplish with their philanthropic dollars. Are they looking for organizations that are going to contribute to large-scale systems change — like reducing racism — where their dollars may be only a fraction of the budget? Do they want to have a deep impact in a particular geographic community (like the Ninth Ward in NO) or with a particular population group (third grade children)? Or do they want to be able to see the direct impact of their own dollars on individuals — like funding a Habitat for Humanity house? Phil B. talked about the need for NPO’s to be clear about goals; ditto for funders. If someone doing a search starts with putting some paramaters around what they want to accomplish (scale and locus of impact), Google could help identify who is on that playing field. This should be (and might actually be) readily retrievable from the nonprofits’ own websites.

    Next, how could Google help donors choose among those doing the work they care about? 990’s are out there and Lucy listed many other sources of data. I also think that most investors put their money where they do because they have confidence in the people involved — whether it be for-profit or non-profit. How about links to biographies of key personnel, like the CEO and board members? Most organizations list these names on their websites and it would be simple to link to other information on the web about these individuals.

    I also think it would be helpful to link to any media coverage of the nonprofits.

    Some (not many yet, but some) organizations are starting to put their logic models / program theories on their websites, either explicitly in the form of a diagram or in narrative explaining how and why they do their work. Pulling this, where it is there, should be doable and useful.

  7. Erich Riesenberg says:

    It is good that your comments have matured away from the persistent bashing of Charity Navigator. Is this due in part to the unraveling of Givewell, which you have so favorably compared to Charity Navigator?

    I will be fascinated if anyone can come up something to be displayed on Google that is not related to the 990. Do you have any ideas? You did not mention any.

    By the way, are you even aware of the changes being made to the 990? It will contain a trove of new qualitative information. Did you ever engage in that debate? I feel grateful for groups such as Charity Navigator for pushing the issue for years, and bringing about results.

    Cheers!

  8. Edith Asibey says:

    Re. Google.com is mostly a resource that points to information; they don’t tend to create a lot of their own content.
    To expand on Sean and Teri’s comments, I think Google could provide great value by offering a live snapshot of web content available on nonprofits. Google could aggregate, organize and summarizing existing information: the good news as well as the bad news. Such a snapshot should help anyone–I don’t see why Google should tend to donors only–who wants to learn more about a given organization. It could also help compare organizations working on the same issues and/or the same region.

    Some suggestions (by no means a complete list):
    – Org’s website and/or blog
    – Thematic focus [who else works on this issue?]
    – Geographic focus [who else works in this county?]
    – List and bios of key personnel [where else are they mentioned?]
    – Financial info publicly available (990s, etc)
    – Current and past funders [which other orgs do they support?]
    – Evaluations publicly available of the org and/or its programs
    – Media coverage
    – International and in-country media coverage for orgs working overseas (VERY important and often neglected point, if you ask me)
    – Blog mentions
    – Online videos
    – Presence in Second Life, Facebook and other communities

    A caveat is that a list like this would likely favor orgs that are Internet-savvy and who have prominent spokespeople. Yet it could help collect a diversity of sources, opinions and data about any given organization. For instance, if an org has recently received negative coverage, this coverage is not likely to be found on its website (with notable exceptions).

  9. Teri,
    Making nonprofits searchable by the criteria you mention is a great idea. It would be interesting to see this evolve as a social tagging concept where donors could label a nonprofit based on causes, geography, etc.

    I also like the idea of somehow providing more complete (objective) bios than the “official” bios supplied by the nonprofit. It seems this is something that could be done via the Wiki format that I suggested.

    The media coverage element is already in the current Google Finance page (both news reports and blog posts).

    If you look at the Google Finance for-profit pages, you find quantitative info (which in the nonprofits’ case would come from the 990). You also find links to more qualitative info. Maybe it is simply too early for an aggregater like Google to add value. Maybe we need more qualitative research being provided first.

    Teri, if Google provided a platform for nonprofit evaluation to be made available to the public, do you think that large foundations would find it attractive to make their existing evaluations available through this platform?

  10. Edith, you make a good point that everything we’ve suggested tends to favor internet-savvy orgs. That is because we’re assuming the nonprofit would be the one supplying/generating the content. It seems to me that with public for-profit companies (the other entities in the Google Finance database) you have legally required information disclosed by the for-profits (under SEC rules) and third party generated analysis.

    I’m beginning to think that until we have third party generated evaluation and/or more extensive disclosure information, that the only real option is a more free-form wiki format rather the data driven model that Google Finance is designed to accommodate.

  11. Erich,
    I’ll be commenting more completely on GiveWell in a full post shortly. I strongly favor GiveWell style analysis over Charity Navigator. That being said, I swore off bashing CN when Trent Stamp announced his retirement. I haven’t been offering a lot of my own ideas because I’m trying to solicit input from the field. Although I have provided a number ideas over the past couple weeks of blogging on this topic.

  12. Erich Riesenberg says:

    Sean, I seriously doubt you have done much study of Givewell’s evaluations, your comments are so very superficial. If Givewell actually has such a wonderful evaluation system, why would that not be the obvious basis for any sort of Google data? Because it is an unwieldly yet superfical spreadsheet data dump?

    I don’t think you or any of the cacophony of nonprofit bloggers have improved the discussion of nonprofit effectiveness at all. Essentially, you all come back to the “life saved” or “meal distributed” or “surgery performed” per dollar spent metric, but even that is only moderately helpful. Deciding whether a nonprofit is creating long term change is largely qualitative and requires personal experience. It fascinates me it took Givewell a year to conclude it costs less to save a life in Africa than to provide employment assistance in one of the most expensive cities in the world, but I guess that is where common sense comes into play.

    Your inability to decide whether expense ratios are worthless, helpful sometimes, or should not even be made available should give anyone pause as to the depth of your understanding of the evaulation of nonprofits.

    Great 2008!
    Erich

  13. Edith Asibey says:

    Sean,
    I do think that Google could greatly help philanthropies and nonprofits become more prepared for and accustomed with providing info and data to shed light into their goals, strategies and results. The data wouldn’t be provided only by the nonprofits, but would be collected through multiple online sources (media, blogs, etc.). And as an incentive, organizations who share their third-party evaluations would have more ‘complete’ profiles than the ones who don’t.

  14. Erich Riesenberg says:

    Oh, by the way, I never did understand why you decided to stop bashing Charity Navigator when Trent Stamp retired.

    Is it because Trent Stamp had been hired by the Eisner Foundation, a $150 million dollar foundation, and you realized that if Eisner trusts the man who built Charity Navigator, it might look foolish / naive / ignorant for you to contine bashing it?

    Cheers!
    Erich

  15. Edith,
    If Google could figure out a good way to “permission” nonprofits to upload information in a way that was secure, maybe the database could be a way for donors and nonprofits to communicate about mission, effectiveness, etc, in a transparent way that would provide the resulting information to everyone. That would seem to reduce the cost to the nonprofit since once they answered a question, the answer would be available in the future.

    And of course they could also link to third party data.

  16. Erich,
    I stopped bashing CN because the agreement from people in the field that CN’s model is deeply flaw is very broad based. I’ve done my part by trying to explain this to donors. But I think that for any real change to occur, a strong alternative must be presented rather than just pointing out the flaws of the leading evaluator. That’s why I’m so interested in the potential of Google doing something in this area.

    I’m sorry you don’t feel that nonprofit bloggers have improved the discussion of nonprofit effectiveness.

  17. Erich Riesenberg says:

    Sean –

    In your Dec 20 post you wrote: My Charity Navigator bashing ends today. I told Trent that he has all of my support in his new job.

    Now you write: I stopped bashing CN because the agreement from people in the field that CN’s model is deeply flaw is very broad based.

    Why can’t you ever stick to an answer? Are you writing that on Dec 20, the same day Trent resigned, you realized your work was done, that the world had stepped back from the brink of caring whether a nonprofit paid its professional fundraiser 75% of donations?

    Your circular comments are a pathetic waste of time because no criticism you have ever made of Charity Navigator’s system are anything different from what Charity Navigator states on its own web site, such as here: http://charitynavigator.org/index.cfm?bay=content.view&cpid=484#15

    Please don’t insult donors and everyone who has worked hard to bring accountability to the nonprofit sector by making such the broad, simple minded claim that there is general consensus that nonprofit expense ratios are of no value. In the past 60 days alone have been all over the map with your broad generalizations, as I have quoted numerous times.

    Also, please don’t feel sorry that I don’t think nonprofit bloggers have improved the discussion of the nonprofit industry. You should apologize to the many people who have come long before you and Holden and who will be here long after people get tired of reading your ramblings.

    Finally, it is great you are preparing a response to the controversy surrounding Holden and Givewell. Holden originally wrote about it Dec 31. You must be thinking very deeply about it. We await your findings!

    Joy in 20078!
    Erich

  18. Philippe Bradley says:

    I had something typed out but in fact, I’ve just come across this, and it describes almost perfectly what I was advocating – much more eloquently, of course. http://triptronix.net/ishbadiddle/archives/2007/02/15/15.20.40/

    THIS is what we need, and it’s not something Google.com (especially not Google Finance!) can do for us. Goofle Finance should stick to the 990 info (plus the forums, the links to news coverage and to blog posts)

    Perhaps, however, Google.org could seek to stimulate progress in this area by putting up a prize/challenge. Not that this area should need incentivisation – the drive to open up to potential donors, and build strong relationships based on trust and reputation, should already be a high priority for the philanthropic sector.

  19. Jeremy Gregg says:

    It is difficult to create such metrics across the board without heavy investment of funds, but the metrics that we use at Central Dallas Ministries to cultivate our donors is “social return on investment.” In other words, the cost savings to the community generated by our work.

    For example, we have determined that our health clinic saves nearby Baylor University Medical Center (BUMC)approximately $203 for every patient that we see (through decreased utilization of the emergency department). This sort of metric helps us to prove the worth of our clinic to BUMC.

    Similarly, we provide a program for housing the chroncially homeless that saves the community $4 for each $1 invested in our program (i.e. the city typically spends $40K per chronically homeless person living on the street, but our program costs around $10K per person).

    These metrics are fairly specific to our organization, and also to these particular programs. We do not have the capacity to determine comparable metrics for our after-school programs, law firm, employment training programs or food pantries. In general, all of these programs yield significant long-term benefits to society that outweigh the initial investment, but proving that requires a significant amount of funding in itself.

  20. Jason says:

    I’m a non-profit employee and I have found that more than anything else donors want to know what you are doing. They want stories about what they are doing and they want to know what the organization as a whole is doing with their money.
    I am continually frustrated when donors ask for administrative rates as I think that it is a very subjective number. That would be my biggest concern regarding metrics is that I’m afraid that if it is all metrics non-profits will just record what they need to for their metrics to sound good.

  21. Evaluations of NPO’s should include quantitative and qualitative evaluations on an even par with each other.

    As argued very articulately above, there are outcomes that can be measured, metrics by which to measure them, and tools for reporting the findings, including the information contained in the revised 990. Google, could contribute to the evaluation of NPO’s by making this quantitative information more easily available to the public, and have the additional effect of reminding NPO’s that it’s our duty to set goals, evaluate outcomes, report findings and respond to outside review.

    This quantitative evaluation process, however, is only half the equation. Unless Google succeeds in communicating this fact and steers the public adequately towards a qualitative evaluation, it should not develop any kind of evaluation aggregator.

    As was articulated very clearly above, there are ways in which to make qualitative evaluations (i.e. links to CEO’s bio’s, narratives by reviewers on sites like Great non-profits, project profiles on websites, links to the NPO objectives and corresponding evaluations, open discussions linked to the NPO’s “portfolio” — a collection of all the evaluation tools being deployed.) I might add a new tool — peer review — a process resembling the tenure process in academia, but with the flat and constructive structure of the National School Reform’s “critical friends” groups for teachers.

    Perhaps I’m stating the obvious or restating opinions already expressed above, but I recommend Google devise an approach that is equally quantitative and half qualitative or stay away from the enterprise altogether. The last thing Google — as a search engine whose great contribution to humanity is more and diverse locus of control — is to participate in any kind of activity that smacks of centralization and homogenization.

  22. Correction: I recommend Google devise an approach that is equally quantitative and qualitative or stay away from the enterprise altogether.

  23. “This quantitative evaluation process, however, is only half the equation. Unless Google succeeds in communicating this fact and steers the public adequately towards a qualitative evaluation, it should not develop any kind of evaluation aggregator.”

    Maureen, I’ll definitely pass this on. I agree.

  24. Erich Riesenberg says:

    Sean, you wrote: Phil, 990 data is clearly important. But it only addresses the cost side of the equation, it says nothing about the social good being produced.

    Still curious if you are even aware of the new 990?

    To quote http://www.charitygovernance.com/charity_governance/2007/12/part-iii-mrs-ir.html

    “As for Karnofsky, Hassenfeld, they may want to rethink their view of the Form 990 based on Parts I and III of the Core Form, which are front and center—pages 1 and 2 of the package. Here organizations are given the opportunity to describe their mission and significant activities. Part III is particularly important, requiring a description of the organization’s three largest services, with a focus on achievements. Don’t fault the Form 990 if organizations don’t tell you enough about outcomes. The IRS certainly has asked about them and given organizations the opportunity to tell their stories. For Karnofsky, Hassenfeld, and others, the solution is actually quite simple: Start with the Form 990 and if you don’t see what you want, move onto the next organization. If everybody did this, organizations would become more forthcoming in providing the sort of information that so many want. In fact, Karnofsky and Hassenfeld would be doing everybody a favor if after each unsatisfactory review, they sent a form letter explaining why they moved to the next candidate.”

    Best of luck to donors and anyone who actually cares about the nonprofit sector.

  25. Erich,
    Yes I am aware of the new 990. Although I am not aware of any authority on the subject of nonprofit evaluation who thinks that the new 990 is a game changer. Describing your mission and your accomplishments is not the same as adequately demonstrating the social good that you produce. I don’t see why we should even expect this kind of info in a tax form.

  26. Erich Riesenberg says:

    Ah, you really are arrogant Sean. It is remarkable that you would dismiss so many as irrelevant and not being an “authority.”

  27. Erich Riesenberg says:

    Oh, and, if groups like Charity Navigator are not authorities, what does that make you?

    Good luck, you obviously need it.

    Erich

  28. Trent Stamp is clearly an authority. He would be the most natural person to celebrate the new form. However, Trent said that the biggest difference between the old form and the new form is that the new form is longer.

    Erich, I’m going to withdraw from this conversation now. As you point out, even Charity Navigator thinks that their ratings are not enough to evaluation a charity. My beef with them is that they’ve successfully marketed themselves to the press to the extent that many donors think that expense ratios are an effective way to evaluate a nonprofit.

    I’ve said my piece on this topic. You’ve said yours. You can feel free to continue leaving comments, but I’m going to take a break from answering your sarcastic and condescending questions.

  29. Erich Riesenberg says:

    Oh Sean,

    It is truly sad you know so little about the nonprofit sector, and are unwilling to engage in any discussion which does not meet your preconceived biases. Instead, you repeat yourself, change your answers and divert attention.

    You are clearly wrong about Trent’s comment on the new form. Trent has not commented on the new 990, which came out in December. The link you cite is from June.

    Yes, in the post you cite he states he would like to see the IRS take strict enforcement action, rather than simply revise the 990, but I won’t debate that here, as it is well beyond your scope of knowledge, and there is no need for me to speak for him.

    I merely feel the need to point out that for you to imply Trent Stamp is critical of the new 990 is without any basis, period, for the benefit of others. He may indeed be critical, but you have no basis to make that claim.

    You may also be language deficient, if you truly think that where the form calls for nonprofits to list their achievements, you somehow think that is different from a nonprofit describing its “social good.” Any donor would expect a nonprofit to describe its measurements of success. The 990 is the only standard form most nonprofits must file, and is the obvious source to turn to for information. You should be happy about its siginficant improvements, but instead, because it does not meet your preconceived bias, and you clearly are ignorant of the new form, you choose to pan it.

    The only reason I post here is because you post with such authority, and I think so much of what you post is plain wrong. You need to be held to account, in case you mislead readers.

    Best!
    Erich

  30. Teri Behrens says:

    RE: Sean’s question about foundations sharing evaluation reports via Google.

    I don’t see foundations making all the reports of evaluations they commission generally available on the web. This may happen eventually, but I don’t think the culture of most of the big foundations is there yet. Even if they are made available, the purpose of those evaluations is often to help foundations improve their own funding strategies (which vary widely in how well they are articulated). Their utility to potential donors to individual organizations is questionable — most foundation evaluations are looking at groups or clusters of grantees to see the overall impact, rather than focusing on individual grantees. They COULD be useful to donors who want to understand in general what change strategies are being used and what is being learned about their effectiveness.

    Anyone other views on this?

  31. Mike Everett-Lane from DonorsChoose.org here — also the author of the above-referenced blog post on microformats and microphilanthropy.

    I think there are a few things that could be helpful re: Google and the nonprofit marketplace.

    The first, of course, is helping implement my brilliant idea about microformats 🙂 [See Paul Ford’s essay on Google and microformats and marketplaces for a good intro.] Google can help set and propagate these markup standards.

    Of course, microformats are good for “microphilanthropy” which is good for orgs like Kiva and Global Giving and DonorsChoose.org. But I also think that it will be good for the sector in general. In general, I think that making nonprofit data more searchable in an intelligent way would be a Very Good Thing. Teri’s idea above is a good start. Let’s make it easy for a donor (or client! or potential partner!) to find organizations by services offered, geography (think the hyperlocal web, but for social services not just for restaurants), hours of operation, volunteer opportunities, need for materials, etc.

    Secondly, let’s take this idea a step further: make it easy for nonprofits to find funding. Let’s get all that data in the Foundation Directory out of the book and out of the library, so we can easily find a potential funder. The grants market is terribly inefficient, and perhaps Google could do something about aggregating the information off of foundations’ 990s as well as nonprofits’ 990s.

    Just a few thoughts…

    Oh

  32. Teri, it seems to me that the Google interface would only really be useful if it is able to link to qualitative analysis of nonprofits. Program officers across the nation are working on this kind of evaluation every day. While I agree that most foundations do not have the “culture” that would be open to this idea, do you think that this information could possible not be a net positive to donors?

    I know that in the investment management business, quantitative information is critical, but it is qualitative information that drives most decision making.

  33. Mike, I like your idea of integrating foundation grantmaking info with the nonprofit data. Some foundations are already in Google Finance, like this page for the Hewlett Foundation (note my comment on the discussion board asking for information on which nonprofits I should support as a donor given my shared interests).

    Also, simply making nonprofits more “searchable” would be a big help. Imagine if Google built something like Grantsfire that connected nonprofits and foundations.

  34. @Erich: Can we return the conversation to one on nonprofit evaluation, assuming the Stannard-Stockton bash is over? I’d be interested in hearing your thoughts on evaluation…it’s a discussion that grew particularly heated, once upon a time, starting with a conversation at NetSquared last year.

    Not sure I need to defend philanthropic blogging here; those that see the value in it will stay, and those that do not will leave, I assume.

    @everyone: So, back to the topic at hand! I think successful nonprofit evaluation certainly lies far more in the qualitative than the quantitative. I think it’s also important to note that the very act of evaluation, to some degree, opposes the act of marketing (and I say this as someone with a background in internet marketing). Marketing is not necessarily obfuscation, of course, but it presupposes that your organization’s value needs to be presented in the best possible light, because that light *affects how you are seen* in the market. An attempt to evaluate is by definition an attempt to move beyond presentation, embellishments, cover-ups, and rhetoric of the salesman bringing his wares to market; this places the two irrevocably at odds. Evaluation, then, is at least partially concerned with penetrating the efforts of marketers. This becomes a chess game, instead of a one-sided consumption of marketed media…but its adversarial nature should be noted.

    But there are more important questions lurking in that particular issue. For example: Why do some folks prefer a quantitative approach to a qualitative one?

    I think the answer is that numbers are easier to verify, and we intuitively assume that verifiable facts do not lie. The problem there is that *every* nonprofit I’ve ever worked with has, more often than not, found itself trying to figure out the “right” numbers to put on a brochure, grant application, or press release. And for most organizations, it’s *easy* to come up with the numbers that represent them best. The problem is that those numbers are inherently the ones that least represent their problems, their struggles, or their learning experiences; they represent their perceived strengths. I’ve *written* them enough times; it’s what the powers-that-be think you should be putting out there to impress the rich folks.

    That’s good business marketing, I suppose…or was, back in the stone ages. But it’s more interesting for me to know an organization’s challenges than its poster stories.

    Highly flammable example: If Givewell had ten staff members and analyzed ten orgs this year (I’m making numbers up, for the sake of argument) and has 100 staff members analyzing 1000 orgs next year, their numbers will show that they have successfully gone to scale and created operating efficiencies to boot. I am interested in the Holden situation, and the organization’s reaction to it; where is that information in the numbers? Especially in the numbers most organization choose to make available to the market?

    With a qualitative approach, on the other hand, we’re slightly more aware that narrative can easily become rhetoric. However, successful experiments in the social content creation space suggest that, in large numbers, we are capable of keeping each other honest (think Wikipedia). Allowing everyone to comment somehow turns the expected logical outcome on its head, and results in value-neutral media. This makes me think a wiki-based platform might work best, with live feeds in a sidebar with the basic numerical background on an organization, and a strong organizing ideology on the tone, values, and rhetorical direction of the content created.

    As an exercise in forcing nonprofits to abandon traditional marketing for a more transparent approach, I think this would be highly effective. However, to succeed it would have to be implemented on a vast scale; otherwise, criticism is shaped into a targeting of one or more organizations who then become controversial. The entire *sector* needs to evolve; this might be a first step in that direction.

    The problem, of course, is that this won’t let me analyze nonprofit scores on a Dave’s Nonprofit Index ticker; it’s a little less neat, and requires a little more reading, but it might provide a far richer analysis and research space.

    To finish up this comment (apologies for the length), I’m not sure Google is the right home for this kind of project, though Google is certainly the right sponsor for the folks that do, eventually, build such a home.

    Thanks for reading.

    Dave.

  35. Erich Riesenberg says:

    David, grow up. Sean has been making nonsensical, spurious attacks and certainly should be called to account. If you or Sean can take a few moments to criticize my facts please spend your time doing that, until then I will take your criticism for what it is worth, more mindless chitchat.

    If you enjoy a good bashing, how about this Holden-esqe remark Sean made on Dec 20, the day Trent Stam announced he would be leaving Charity Navigator to head the Eisner Foundation: But let me say this about Trent. He knows charities. He will likely make a great executive director. He won’t just use overhead expense ratios to make grant decisions for the Eisner Foundation.

    So, here Trent Stamp is leaving Charity Navigator to begin a great adventure actually making grant decisions, and Sean feels the need to take one last swipe. How arrogant, and typical.

    I think Sean deserves the condescension award, and I am happy to deliver it.

    As an aside, it took Sean about an hour to comment on Trent Stamp’s departure. Holden’s implosion took place December 31, and Sean still hasn’t found time to comment.

    What is it with nonprofit bloggers? You people don’t even hold people like Holden and Sean to account when they make contradictory, superficial comments. I realize Sean and presumably others on here feel the need to tell donors they are not capable of making decisions, but anyone who has read a single Givewell evaluation can easily see it is nothing more than a superficial data dump. You should care more about nonprofit effectiveness and less about media attention.

    Best!
    Erich

  36. @Erich: Looks like Sean commented on the Holden thing as soon as the board decision came out (it was on the front page of this site before your previous comment).

    My earlier comment to you was an invitation to comment on nonprofit metrics (the subject matter of this thread); your contribution here would be valuable. I don’t want to get into an off-topic discussion here; that just disrupts information architecture and derails conversations in the hope of making a splash (kind of what you were accusing Sean of).

    I’m actually convinced you have valuable things to contribute, which is why I wish you’d written more about nonprofit metrics (other than the Holden sidestory, which is elsewhere). Put the anti-Holden stuff there, and the nonprofit evaluation stuff here?

    Thanks,

    Dave.

  37. Erich Riesenberg says:

    Gee David, thanks for the invite, but you are wrong to assume I have never commented on nonprofit evaluation here. You may want to peruse prior comments.

    Trust me, I don’t plan to waste a lot of time on a pointless, self gratifying blog. It is simply that the exciting moves at Charity Navigator and the implosion at Givewell provided nice confirmation of comments I previously made here, which Sean had hastily dismissed, and this is a GREAT opportunity to call him on his hypocrisy and ignorance, and I am taking it.

    I have made perhaps a dozen comments regarding nonprofit evaluation on this very blog, have carefully cited the maze of Sean’s circular arguments, and Sean constantly evades all the issues.

    Pushing Sean on Givewell is entirely appropriate. Givewell was the first source other than a nonprofit fundraiser that I became aware of which claimed fundraising expenses should be ignored, and Sean repeatedly praised Givewell. I merely tried to find out why, and Sean’s only answer, as posted in this thread, is that people in the industry don’t find it useful. I have learned that when people refers to anonymous experts, they are generally making it up.

    I think that any self professed nonprofit expert who claims paying attention to administrative and fundraising expense ratios is a waste of time is, in my opinion, ignorant, or worse. The fact that Sean is not even aware of the changes being made to the 990, and again dismisses them as being of no interest to the same anonymous “authorities” in the nonprofit sector, tells me not to waste my time trying to have a rational discussion with him.

    My guess is Sean’s goal is to tell donors nonprofit evaluation is hard,so they will hire him.

    It is obvious that aside from the simple financial metrics which provide so much basic information that everyone should be aware of, more complicated analysis requires examination of the individual charity. The 990 will provide information on the metrics the nonprofit itself uses and people will be able to judge that. It will flow back to Charity Navigator, Guidestar, and other aggregators. Beyond that, people will have to learn about a nonprofit directly, to see if it meets their values. It is pretty simple.

    Sean isn’t even at the point of trying to reinvent the wheel, it is still trying to determine if there is a wheel.

    Glorious 2008!
    Erich

  38. I know there are a lot of new readers here right now. So if you’re interested in my professional background and how I get paid you can visit the section about my firm. My firm Ensemble Capital helps donors structure the financial aspects of their philanthropic plans (we set up private foundations, charitable trusts, etc) and then get paid to manage the assets inside these vehicles and in many cases the donors personal assets. We do not offer services related to nonprofit evaluation.

  39. Philippe Bradley says:

    Erich, might I venture a neutral observer’s opinion here? You’re coming off as a rabid beast. You’re trying to ram Charity Navigator down everyone’s throat and boring us with this incessant, completely irrational gloating about one/two unfortunate employees at GiveWell – a brush with which you perversely want to tar the whole organisation, even their entire philosophy and approach. I’ve had the misfortune to read you on this blog and others. It’s the same stuff again and again.

    You attacked Sean’s explanation on why he feels even the new 990 is not sufficient to evaluate the suitability of a nonprofit for donation. Yet you did this without actually refuting his criticism, merely whipping out your belief in his ignorance of the matter at hand… again. This seems like double standards when you so clearly decry ‘circular arguments’.

    You seem so hellbent on derailing this thread. Why? Why not email Sean and let others stick to the matter at hand?

    I can’t stress enough that I feel you ought to re-examine your writing tone if you’re seeking to have any influence over the people reading your posts. I might suggest you’re somehow affiliated with Charity Navigator, but even they wouldn’t be so blatant in their self-righteousness and unquestioning, unassailable faith, and besides, their zealotry wouldn’t splash onto other people’s blogs like this. So I really don’t understand your behaviour.

    So just give it a f*cking rest. It’s tiresome and it manages to miss not just THE point, but all the auxiliary rants you drag into the thread.

  40. Philippe Bradley says:

    Hey, maybe it’s Holden again, exploring more advanced strategies of using fake identities in public arenas to put people off rival initiatives.

    This time, rather than pretending to be a nice GiveWell fan, he’s pretending to be an asshole Charity Navigator zealot!

  41. Erich Riesenberg says:

    Phillipe, you are wrong, I am not trying to build up Charity Navigator, I merely called Sean a jerk for persistently bashing them.

    What I am drawing attention to is the nonprofit expense ratios.

    Have a great year!
    Erich

  42. Erich Riesenberg says:

    Oh, and no Phillipe, reading the babble that goes on at this and similar nonprofit blogs, I am definitely not trying to influence anyone. I am convinced it is an incestuous group of nonprofit industry hacks who cross post.

    Cheers!

  43. David Lynn says:

    This has gotten a bit sick (wish these personal sub-threads could disappear), but just in case somebody is still reading, a concept came out of a few non-profit meetings I had today:

    What if Google could cross-reference 990’s with volunteer hours, and other volunteer stats like # of repeat volunteers. Obviously this would require self-reporting by the organization or by the volunteering individuals (might be a business there, we’re thinking about it).

    Why, you ask?

    Number of volunteers / volunteer hours, combined with financial info, shows a few things:

    a) More volunteers generally means more leverage for your donated dollars. A 10-1 volunteer-staff hour ratio means each of your dollars should mean that your money goes further.

    b) More volunteer hours, and especially things like repeat volunteers, should mean that the public finds this to be an acceptable and worthwhile charity. A whole bunch of community members wouldn’t be donating their time if nobody believed in it.

  44. Erich Riesenberg says:

    Hi David,

    Sorry to butt in, but number of volunteer data is included in the new 990!

    Read all about it. Seriously, people should read about the new 990. This is the best nonprofit governance blog I have run into.

    http://www.charitygovernance.com/charity_governance/2007/12/part-i-hey-mrs.html

    Cheers!

  45. David Lynn says:

    Thanks for the link! But just # of Volunteers is not sufficient (although better than nothing). I know plenty of charities that have a ton of people that come in for a couple hours on some sort of organized activity (like church or classroom helping at a food bank). While that certainly shows some belief in the mission, that doesn’t necessarily translate into monetary leverage (tons of volunteer coordination for low output). And repeat stats would show more.

    Thanks.

  46. Fundraising expense ratios are incredibly important…when you’re hiring or reviewing the performance of a fundraising employee. It’s an internal metric. It’s meaningless externally, as an evaluation tool for the organization. The harder you press for this number as an evaluation metric, the more you’ll find organizations classifying things creatively to paint a prettier “metric” for you; it doesn’t reflect the actual world nonprofits operate in.

    @David: Correlating volunteer stats with narrative on volunteerism at the organization might create a very clear picture. I still like the wiki-based idea I suggested earlier, but perhaps it should be organized into tabs (such as “volunteerism”), with each tab correlating narrative and numerical data, but (key point) open to public editing and commenting, both anonymously and named?

    Glad to see the thread back on course. The conversation’s too important to loose.

    Dave.

  47. Phil Steinmeyer says:

    I see volunteering stats as only marginally interesting.

    I don’t think funders should essentially ‘take credit’ for volunteer time (i.e. my donations are leveraged because of all the effort of those poor sap volunteers).

    And the fact that volunteers choose to work for an organization doesn’t necessarily make that organization terribly effective. I’ve volunteered for one organization that I thought was relatively ineffective in use of money (and volunteer time). I did so because I enjoyed the volunteer experience and the social interaction.

  48. Dave, organizations will always put the best light they can on their numbers (publicly traded companies certainly do), but don’t you think that forward thinking nonprofits who really are effective can find a marketing advantage in being truthful and transparent in revealing their evaluation metrics?

    In other words, I don’t see marketing and evaluation at odds (although there is certainly a natural tension). Instead I see transparent evaluation as one of the most effective marketing strategies (for effective nonprofits).

  49. David, I really like your idea of volunteer hours. I continue to think that there is no one perfect metric, but the commitment level of an organization’s volunteers is part of the whole story.

    The public goodwill that high volunteer hours suggest is an asset of the nonprofit and so it increases the total resources available to them. But I don’t think it says much about effectiveness, since volunteers are not the ones receiving the “product and services” of the nonprofit.

    I don’t agree with Phil’s comment regarding his worry that funders would take advantage of the “poor saps”. Volunteers are a resources just like an endowment. Funders should also not fund an organization just because they are in strong financial shape.