When we talk about measuring the impact of philanthropy, we are in essence discussing how to know the truth. We are examining how we might best understand the reality in which we live and the ways in which our actions affect the world around us. While “impact assessment” can sound like an academic or theoretical concept, it is at its root, simply the search for truth.
When thought about this way, you can see why I care about holistic approaches to understanding the social sector. While statistical analysis or stakeholder reviews or expert reviews or financial analysis all offer a view into the truth, each are limited in their ability to understand the full truth. So it was with great interest that I read an article in the Foundation Review about the Knight Foundation’s use of investigative journalists to do impact assessment.
Starting in 2006, the Knight Foundation hired investigative journalists to write “clear, honest, factual and transparent quarterly reports” that attempted to “draw out something close to the truth” about the results of the Foundation’s programs. The journalists were given full access to internal Knight Foundation documents and employees, and free range to pursue the story of what really happened through interviews with grantees, community leaders and anyone else of the reporters’ choosing.
The Foundation Review article, authored by the former vice president of communications for the Foundation, concludes that the resulting reports were “more direct, even critical, than any prior Knight Foundation attempt to evaluate and assess,” and that they “produced deeper looks into the intent and outcome of major initiatives.”
Great journalism helps the community it serves to discover the truth about the reality of the world in which they live. Great journalism recognizes that the truth is not simply the facts, but is dependent on the context in which the facts exist. Importantly, journalism uses narrative to communicate its findings.Stories are not distractions from the truth, they are the vehicle through which humans have always come to understand the truth.
The role of investigative journalists in helping us understand financial markets and politics is well understood. Books like Too Big to Fail, by New York Times reporter Andrew Sorkin set the standard for helping us understand “what really happened” during the financial crisis, just as All The President’s Men by Woodward & Bernstein got at the truth of “what really happened” during Watergate. Importantly, these books and the newspaper articles by their authors are understandable by non-professionals and in fact are extremely popular with laypeople.
Too often when we think about “impact assessment” we think of statistical analysis and dense research reports. These techniques have huge importance, the are often the foundation on which great journalism draws, but we must not fail prey to the idea that research reports and statistical analysis represent “the truth”. The truth of reality is a multifaceted mosaic that all humans seek to understand. We don’t do ourselves any favors when we discuss philanthropic impact as if it is an arcane science through which only specially trained professionals can determine what works.
Philanthropic impact is nothing more than the extent to which philanthropy actually makes a difference. The field of impact assessment is nothing less than the search for The Truth.
8 Comments
Thanks for your great post. As a career investigative reporter myself (and co-founder of the Center for Investigative Reporting), I would just add that investigative journalists employ a methodology not unlike the scientific method of inquiry — developing a hypothesis that we adjust (as the facts dictate) before publishing. We also go through a type of peer review, with the fact-checking process; plus we tend to be our own strongest devil’s advocates.
You mentioned stakeholder reviews in your post — these days I am working with GreatNonprofits, where as you know we gather user reviews from donors, volunteers, clients, etc. To a journalist’s eyes, these user reviews represent a valuable source of authenticity, as stakeholders tell their stories in their own “voice.” And, while, as you correctly note, these should never be used as an exclusive way to evaluate an organization, they do bring a valuable new element to the table.
Call it the “wisdom of the crowds,” or “community feedback,” or “transparency,” these reviews would provide any investigative reporter a treasure trove of useful information when evaluating how a foundation or other nonprofit is operating in the real world.
Sean, great post about Knight Foundation’s “real world” approach to figuring out if they are actually having an impact. Thanks for sharing.
Sean, I’m unclear how this differs from expert evaluation, especially since Knight funds journalism-related grants. Aren’t investigative journalists just a class of experts? Would it be different if it were a panel of forensic accountants? Thanks.
Steve,
Most of the programs that were subject to the investigative reports were not journalism related. So the journalists were not evaluating the programs as experts themselves. Instead, investigative journalists gather information from many different sources to build a coherent narrative about “the truth”.
I’m not suggesting that this approach is totally unique. In fact, I would argue that nonprofit analysts such as New Philanthropy Capital and GiveWell use a similar approach. Note David’s comment above on the process that investigative journalists use.
What makes the Knight project interesting to me is that the evaluations were carried about via a synthesis of many different kinds of input and a coherent, plain English narrative was built.
If instead a team of forensic accounts were used, the inputs they would have looked to may very well have missed “off balance sheet” assets and liabilities such as the impact on the community’s intangible resilience.
At synthesizing generalists, investigative journalists have an opportunity to gather many different kinds of inputs and create a mosaic that has a shot at being more accurate than any one of its elements
Now THIS is one “elegant” post. Beautiful.
Sean, great post and a fascinating approach by the Knight Foundation. I agree that investigative journalism shares many similarities with NPC’s approach – triangulating a number and range of sources, from interviews to research, stories to statistics – to arrive at a judgment of what really happened.
There are two takeaways for me in this post. One, arriving at the truth requires effort. You have to build a multifaceted mosaic, as you say – each of the elements requires time and information, and synthesising them requires the analyst or journalist tasked with doing so. It’s great that the Knight Foundation chose to make that investment to arrive at a rich and detailed picture of its impact.
Second, we’re starting to see how parts of this information gathering and synthesis can happen in a distributed way, so the finances get pulled together and analysed by one group, while another digs into the research and evaluation, and another gathers crowdsourced feedback. Journalism’s changing in response to today’s online world – the search for truth about philanthropy’s impact will too.
Thanks Myra, that’s very kind.
Tris, thanks for your thoughts. I do think that the best analysts are essentially investigative journalists. So I’m glad you agree?
Sean, great post about Knight Foundation’s “real world” approach to figuring out if they are actually having an impact. Thanks for sharing.