Last week I mentioned CalHospitalCompare.org, a website that shows comparative rankings for the effectiveness of California hospitals.
A reader emails today with some thoughts on problems with the measurements used by the site. He points out that the vaunted Cedars Sinai of Beverly Hills has only an “average” ICU mortality rate according to the site. The site assumes that a hospital should strive for a lower mortality rate (less patients dying is good right?):
Why are we pedantically told “lower is better”? Why is lower better? Is this not exactly, specifically, the dead-nuts center of the “metrics” conundrum?
If the Cedars ICU mortality rate was 50%, wouldn’t that mean Cedars must be where everyone in the know knows to go when intensively sick?????? Included among them, obviously, are the intensively sick at death’s door — thus inflating the ICU mortality rate with folks whose only hope is for more time in ICU — not escape from it. “Mortality” measures nothing meaningful about the care and skill and hand-holding services provided AT DEATH’S DOOR in the Cedars ICU!
That 50% ICU croak rate might just mean Cedars is the place to go to be kept ticking long enough for all my family stragglers and long-losts to fly in and come kiss me goodbye? I’d still add to the croak rate, but… Wow! What service!
What the hell kind of “metric” measures that PRICELESS service???!!!
What about the exact opposite “metric” finding: Suppose Cedars ICU mortality rate was 1%. This means, of course, if the “average” is 13%, that Cedars is either magical — or lying. Let’s say they’re just wonderful. And… so… via “metrics” getting into the hands of salesmen… everyone discovers how wonderful Cedars ICU can be. So business goes up on the word — logically, in the fullness of time. Now… More and more people at death’s door come to them for those precious last few hours or days… But. In the end, folks croak. So? Little by little, Cedars ICU mortality rate gets back in line with “average” and, like all intended measurements for goodness, is rendered meaningless.
Why, why, why, why so much time on this obsession with measuring The Good?
The emailer then takes me to task for focusing on measuring nonprofit outcomes. I found his argument above excellent. This is a great example of how important it is to measure the right things. We cannot depend too heavily on any one metric. There is no magic, simple way to determine how effective a nonprofit is.
Right now, any decent economist could give you a long list of statistics that show we are in a recession… and a list of statistics that suggest we are not. That doesn’t mean that metrics are worthless. It does not mean we should not strive to seek knowledge. It means that understanding our world is hard. But through understanding our world, we can create a better one.