Robert Waldmann Has a Big Problem with the Anti-Prozac Meta-Analysis Study
Robert Waldmann has a big problem with and talks back to the anti-Prozac meta-analysis study of Hirsch et al.*.
Mark Liberman summarizes the meta-analysis data in a nice picture:
The x's are studies. The vertical axis shows the improvement in mood for people being given the placebo--the sugar pill. The horizontal axis shows the improvement in mood for people being given the antidepressant, both according to the Hamilton Scale of Depression.
People being given the placebo improved their mood a lot--by 7.8 points, which is a relatively big deal on the Hamilton Scale: feeling that you are taking control over your Depression by getting involved in a cutting-edge medical study, the fact that a group of research scientists are paying attention to you, and the passage of time together do a lot of good. But the people being given the actual anti-depressants improved their mood by even more. Let's turn the mike over to Hirsch et al.:
[W]eighted mean improvement was 9.60 points on the HRSD in the drug groups and 7.80 in the placebo groups, yielding a mean drug-placebo difference of 1.80 on [Hamilton] improvement scores.... [which] easily attained statistical significance [at the 0.001 level, in fact--much better than the 0.05 level]...
Subjects given Prozac improved their mood by an extra 1.8 points on the Hamilton scale. This difference is not due to chance sampling error--it is, statistically, very significant. The pills are really cheap to make. There is an upside. Better Living Through Chemistry.
So what's the problem with Prozac? The problem, according to Hirsch et al., is that the difference of 1.8 points on the Hamilton Scale:
does not meet the three-point drug–placebo criterion for clinical significance used by NICE [Britain's National Institute for Health and Clinical Excellence]...
Where does this requirement that no therapy for Depression is worthwhile unless it improves the Hamilton Scale score by three points come from? The weblog "Pyjamas in Bananas" finds a quote:
Pyjamas in Bananas: No research evidence or consensus is available about what constitutes a clinically meaningful difference in Hamilton scores, but it seems unlikely that a difference of less than 2 points could be considered meaningful. NICE required a difference of at least 3 points as the criterion for clinical importance but gave no justification for this figure...
Who wrote this? Irving Kirsch, lead author on the anti-Prozac study.
And it is at this point that the economist in me wants to reach for his revolver. A declaration that a real-world solid statistically-significant improvement in people's quality of life is not "clinically significant" is inadmissable unless it is motivated by a proper analysis of opportunity costs: a conclusion that the resources devoted to this therapy would have a higher value and a better alternative use in some other therapy. It cannot rest on an arbitary number that some organization pulls out of its a--.
Even worse, Robert Waldmann points out, is that the Guardian's health editor Sarah Boseley doesn't understand the article she is reporting on:
Prozac, used by 40m people, does not work say scientists: Analysis of unseen trials and other data concludes it is no better than placebo: Prozac, the bestselling antidepressant taken by 40 million people worldwide, does not work and nor do similar drugs in the same class, according to a major review released today.... When all the data was pulled together, it appeared that patients had improved - but those on placebo improved just as much as those on the drugs...
Waldmann comments that Boseley is:
totally dishonest, totally innumerate or both. 1.8 > 0. Patients on Placebo did not improve just as much as patients on SSRI's... this isn't even a case of treating a statistically insignificant difference... as... proof that the true value is zero.... "Irving Kirsch, Brett J. Deacon, Tania B. Huedo-Medina, Alan Scoboria, Thomas J. Moore & Blair T. Johnson" find a significant additional benefit of taking a SSRI rejecting the null of no benefit with a p value of "<0.001"... overwhelmingly strong evidence that SSRI's cause improvement in depression.... Oddly big Pharma, which spends huge amounts of money on advertising, doesn't seem to have managed to hire anyone intelligent enough to point out that 1.8 > 0...
