Along with my supportive column on the Pay For Success program, Peter York of TCC Group wrote a more caution piece:
President Obama’s budget proposal for 2012 includes a $100-million idea that could revolutionize the way social programs are financed—or could put a ticking time bomb next to the heart of some of our society’s most needed social programs.
…The main concern will be how to define success; in the real world, it is difficult, if not impossible, to agree on rigorous measures that make sense.
…If all government-financed programs were to rely on the social-impact-bond approach, the debate over measurements of success would more often than not set up social programs to fail, with a nonprofit’s program, not the unrealistic expectations attached to it, becoming the scapegoat.
And that could unwittingly undermine society’s support for government financing of programs to help the needy just at a time when relatively little support already exists.
Melinda Tuan, a co-founder of REDF and expert in social return on investment frameworks, writes:
I think very careful attention must be paid to the projected positive outcomes of a program and whether those are intended to generate cost savings for the government.
In my experience with social return on investment (SROI) at REDF, where we tried to calculate the social savings to society generated by nonprofit social enterprises that employed formerly homeless individuals, I clearly recall at least one important finding from that work.
We funded a fabulous nonprofit called Youth Industry that tried to get homeless youth off the streets and into wage-earning jobs in social enterprises such as a thrift store, bicycle repair shop, neighborhood restaurant, etc. The positive outcomes achieved in the lives of these homeless youth were remarkable. Based on follow-up interviews conducted two years after youth were hired at a Youth Industry enterprise, 72% of youth were still employed, and 45% were living in a stable home (as opposed to 81% homeless at time of hire).
However, the youth were also using more social services at two-years after hire than they were at the time of hire — which meant they were costing more to society than they had previous to being employed in Youth Industry social enterprise. Their “SROI” numbers were not good.
But the real story is that these were youth who were largely disenfranchised from society at the time of hire: 76% had used illegal substances, 68% had mental health issues, 41% had a previous criminal conviction; and as a staff members said, “they were not working the system, they wanted to get out of the system.”
The youth outcomes showing an increase in usage of social services provided by the government were a positive outcome for the youth. For example, they were re-engaging in society, accessing much-needed mental health services, and obtaining food stamps to feed themselves (as opposed to scrounging in the streets). In the case of these formerly homeless youth, costing more money to society was a step in the right direction towards their ultimate greater self-sufficiency and well-being.
Sean, I was glad to see that you cautioned any Pay for Success program be piloted using effective programs with evidence of success AND where positive program results would produce cost savings to the government. It would be a huge mistake to simply assume that cost savings are tantamount to positive outcomes in individuals’ lives.
Reader Jacob M comments:
While all this sounds great, to me the question whenever anyone sets up an incentive-based system that uses outcomes is a question of monitoring. No Child Left Behind’s effect on state education standards is probably the best recent example of this – when the Feds asked the states to hold their schools accountable for passing State tests, most states just dumbed down their tests so students would pass.
There’s a similar issue at play here. If the intermediaries get compensated for the outcomes which they fund, there better be some independent entity that is collecting the data and ensuring quality. Otherwise, there will be strong pressure, especially if investors get involved, to deliver on those outcomes.
While some of this pressure might lead to program efficiencies, it also will encourage people to game the system. Outright lying about outcomes might be possible to catch, but other more subtle methods such as “creaming” for the least needy may prove more difficult. And even if on paper outcomes may look good, the neediest people may become more likely to be overlooked.
I’ll respond to some of these comments tomorrow as all of them raise important points.
6 Comments
Fascinating discussion, and I certainly agree that there is a huge need and huge difficulty in defining and measuring the outcomes.
However, there is a giant benefit to the Pay for Success program running through intermediaries: Without the normal direct relationship from funder to grantee (or govt to agency), the market is essentially open for competitive bidding. The onus isn’t on the government or the end agencies – it’s on the intermediary to prove that they are an effective service picker.
Much like picking a mutual fund, all mutual fund managers essentially have the same basket of public companies to pick from. It’s up to each manager to build their portfolio, and prove that their portfolio will outperform others. If you don’t like that manager, you can move to a different mutual fund – but the underlying basket of available stocks to pick from doesn’t change.
That’s the real value here that I see. You can fire one intermediary and choose another, which in some ways insulates the agencies. It’s up to the intermediary to prove that their portfolio of service providers is outperforming the rest, and it’s their investor’s money on the line, not the taxpayers.
David. I disagree with part of your analysis on the benefits of working via intermediaries. Specifically, I’m not a fan of the mutual fund analogy.
For me, the primary advantage of working through intermediaries is the local/specific knowledge of intermediaries. For example, community foundations may know of high quality local providers that the Federal government would have difficulty finding. In my mind, the biggest benefit is that the intermediaries have already spent resources identifying these high-quality providers for their own funds. In other words, the intermediaries can provide the Federal Government with a “short list” to evaluate.
In other words, it’s precisely because everyone isn’t picking from the same basket that this is a helpful idea. Where I think your “mutual fund” analogy starts to break down is that foundations are often highly restricted. Whether’s it’s geography, charter or even leadership, foundations often have blinders on (whether they’re self-imposed or legally imposed). Note this isn’t actually always a bad thing – in fact, it probably makes them more effective than trying to do everything.
One of my chief concerns about the program is exactly what you think this the “real value”. There has to be a balance in the pressure placed by these sorts of schemes. If the government were to rountinely “fire” a large percentage of intermediaries, it would be incredibly disruptive for them and more importantly the nonprofits they support. I think you would want to be sure to temper the “firing” in favor of stability and predicatability in funding.
Jacob, I agree that I might be a bit idealistic to think there would be a sufficient set of qualified intermediaries. However, I like to think if that there were say 5 intermediaries all focusing on an issue like homelessness or obesity in one metropolitan area, even though they would essentially be picking from the same overall set of agencies, each of them would mix it differently and try to bring something special. It may also push them toward root cause solutions as opposed to treating symptoms, as you’re unlikely to be in business very long if you don’t bring solutions – the blinders will have to come off on occasion.
But most importantly, the burden of proof for sector change would be on the intermediary. That’s a highly needed role, and outside the scope of most small-medium agencies. Which is why I accept the mutual fund analogy: the burden is on the fund manager to show that they have a track record of building a portfolio of good companies that deliver performance, and deciding how and when to invest in those companies (equity, loans, bonds, etc). The companies just keep doing what they’re doing, and will have some good years and bad years.
Additionally, the pay for success model re-invigorates the sense of risk capital in the philanthropic sector. Dollars that don’t need a financial return should be the risk capital. A group of intermediaries essentially competing for service dollars and taking some risk should “temper the firing” effects. They may not even need stable, predictable funding – if I had an idea to move the needle on homelessness in my city and could invest in it and get paid back with some return over a 3-5 year period, that would be exciting.
I recognise that this is an easy solution to suggest, not so easy to implement, but the issue here is not being able to calculate a proper “cost to society”.
In this case, the youths involved were seen to cost little to society. They took advantage of no benefits, etc. The problem is that this is to simplistic. What about the cost of criminal systems from their activity, or related activities.
Assuming there is no cost unless it is directly attributable is a dangerous (and obviously false) assumption. I have no doubt that in this example, if the true “society cost” could actually be calculated, that the scheme would have been seen in a positive light.
I think Melinda’s comment points to the larger question I have in my mind when thinking about SROI frameworks and initiatives focused on paying for outcomes: what is the time frame we are concerned about?
As her story pointed out – if you tend to have shorter time frames (less than a year to 2-3 years) you might actually see a negative SROI for some of the highest risk populations. That’s because initiatives for very high-risk populations focus on getting people connected to resources – which means they are accessing systems, and leading to higher societal expenditures. It’s only in the longer term (4+ years) that you start to see true societal savings in these types of programs.
So the cautionary question I have in my mind is: what time frames are people using to define success or to calculate SROI. Too short a time frame means that only short-term programs will be highlighted as successes. But waiting 20 years for a true SROI might be something that very few people would be willing to commit to.
As is so often the case with innovation, there are many perils to be wary of. I agree completely that Pay For Success can’t work without better methods for measuring impact. There is a real risk of building incentives around metrics that might sound good in a soundbite, but, under the harsh light of RCT social science, wouldn’t hold up. Also, as Melinda correctly points out, we could easily be surprised when, a la “freakonomics” behavioral economics, social metrics look good, but fiscal savings fail to materialize. I also worry about problems of teaching to the test, or, as my friend Nigel Morris used to call it, “monovariablitus”. And then there’s the problem of putting wealthy foundations into a position to call the shots, in a sense, for how taxpayer money is allocated. These are just a few of the many challenges.
All that said, I like to think of Pay For Success as being one of many innovations that can be tested and improved without causing great harm. Let’s not jump to judgement before getting some experience under our belts.
Perhaps we will end up thinking of this as trading one set of difficulties for another. Remember Churchill’s quote: “It has been said that democracy is the worst form of government, except for all the others that have been tried.”
For example, I certainly do not support the bludgeoning of smaller nonprofits with onerous evaluation requirements. But when the checks get big, (as they do with government), and when taxpayer money (not voluntary philanthropic money) is involved, it seems to me that we might have an obligation to raise the burden of proof, whenever it might be possible to do so.
Moreover, I have been persuaded by Jon Baron and others of two truths: (1) There exist highly proven and scalable interventions that are deeply undercapitalized relative to the good they can do, and (2) Under the harsh light of social science, it is clear that hundreds of billions of government dollars are spent each year towards programs that may have once worked, but, today, bring about no discernible change in the life course of the individuals who are served, as compared to randomly selected peers who are not served.
To me, there is a moral imperative to reallocate the “fund what once worked” dollars towards whatever “here’s what we can prove that works” programs may exist.
So while it is true that most programs, indeed most areas of social need, lack sound evidence-based stories of impact… some actually do have RCT proven programs that work. For those, Pay For Success seems like a good way to experiment.
Finally, I am optimistic that the cost of evaluation, both in terms of money and in terms of entangling operations, is dropping precipitously, and that we will therefore soon detect a growing number of high impact/proven programs. The reason: huge databases are now coming online that are a byproduct of running schools, jails, homeless shelters, emergency rooms, etc. Our ability to track individual outcomes, (while also protecting their privacy) is being revolutionized.