Having read Katherine Baicker et al. (2013), "The Oregon Experiment — Effects of Medicaid on Clinical Outcomes", New England Journal of Medicine, I conclude that the study was--as natural experiments are--significantly underpowered, that the jury is still out on most of the effects, but that the success of the Medicaid expansion in reducing depression and improving financial security are rock-solid and nailed benefits, by themselves worth more than the net cost of the expansion.
Therefore I wish to withdraw my tweet from last night:
And substitute for it, instead, the correct:
Aaron Carroll and Austin Frakt:
Oregon and Medicaid and Evidence and CHILL, PEOPLE!: We’re somewhat annoyed that the NEJM sent out press releases and the study to journalists, but not people like us, because we now have to rebut the gazillion stories that have already been written on a study I just found out about an hour ago. Maybe they should let some knowledgeable people see it early, too. Or just wait until it goes live to tell everyone. But we digress. Let’s get into it. To recap: Oregon ran an RCT of Medicaid, because of a lack of funds to expand it fully. Early results showed some promising evidence that Medicaid improved process measures, self-reported health, and enhanced financial protection. This update, at 2 years, was intended to give us some harder outcomes. The results are “mixed”:
We found no significant effect of Medicaid coverage on the prevalence or diagnosis of hypertension or high cholesterol levels or on the use of medication for these conditions. Medicaid coverage significantly increased the probability of a diagnosis of diabetes and the use of diabetes medication, but we observed no significant effect on average glycated hemoglobin levels or on the percentage of participants with levels of 6.5% or higher. Medicaid coverage decreased the probability of a positive screening for depression (−9.15 percentage points; 95% confidence interval, −16.70 to −1.60; P = 0.02), increased the use of many preventive services, and nearly eliminated catastrophic out-of-pocket medical expenditures.
Let’s review. The good: Medicaid improved rates of diagnosis of depression, increased the use of preventive services, and improved the financial outlook for enrollees. The bad: It did not significantly effect the A1C levels of people with diabetes or levels of hypertension or cholesterol. This has led many to declare (and we’re not linking to them) that the ACA is now a failed promise, that Medicaid is bad, and that anyone who disagrees is a “Medicaid denier”….
From our full reading of the paper, let us add the following to the conversation:
Improvements in mental health are still improvements in health outcomes. The rate of positive screens for depression dropped from 30% to 21% in the Medicaid group. The rate of medication use for depression went from 16.8% to 22.3%. It wasn’t statistically significant (though it was close, p=0.07), but that doesn’t mean Medicaid failed. Which leads us to…
Non-statistical significance does not mean failure. It means that either (a) there is no treatment effect or (b) the study is underpowered. Since there does not seem to be a power calculation, we can’t tell. How much of a difference would there need to be in order for statistical significance? We can’t tell. But just because this difference wasn’t significant in with the sample studied doesn’t mean it wouldn’t be significant with a larger sample. Indeed, the authors note this in the discussion: "Hypertension, high cholesterol levels, diabetes, and depression are only a subgroup of the set of health outcomes potentially affected by Medicaid coverage. We chose these conditions because they are important contributors to morbidity and mortality, feasible to measure, prevalent in the low-income population in our study, and plausibly modifiable by effective treatment within a 2-year time frame. Nonetheless, our power to detect changes in health was limited by the relatively small numbers of patients with these conditions; indeed, the only condition in which we detected improvements was depression, which was by far the most prevalent of the four conditions examined. The 95% confidence intervals for many of the estimates of effects on individual physical health measures were wide enough to include changes that would be considered clinically significant — such as a 7.16-percentage-point reduction in the prevalence of hypertension. Moreover, although we did not find a significant change in glycated hemoglobin levels, the point estimate of the decrease we observed is consistent with that which would be expected on the basis of our estimated increase in the use of medication for diabetes."… The point estimates show that blood pressure did fall…. The percent of people with diabetes with a high A1C went from 5.1% off Medicaid to 4.2% (p=0.61). The percent of people with high total cholesterol went from 14.1% to 11.7% (p=0.45)….
Financial hardship matters. Here Medicaid shined. It hugely reduced out of pocket spending, catastrophic expenditures, medical debt, and the need to borrow money or skip payments.
Preventive care matters. We’ve been cautious about the ability of prevention to save money. But some preventive care improves outcomes. More people on Medicaid got colonoscopies, cholesterol screenings, and prostate cancer screens (whether or not you support them). The percent of women over 50 who got mammograms doubled from 28.9% to 58.6%. Results once again weren’t always “statistically significant”, so people can claim “Medicaid failed”. But colonscopies in people over 50 went from 10.4% to 14.6% (p=0.33). Failure?
Health insurance is necessary, but not sufficient to improve health. It’s just the first step. We have never claimed that quality would go up just because of the ACA. Access will improve. We need to do a lot more work to improve quality….
So chill, people. This is another piece of evidence. It shows that some things improved for people who got Medicaid. For others, changes weren’t statistically significant, which isn’t the same thing as certainty of no effect. For still others, the jury is still out. But it didn’t show that Medicaid harms people, or that the ACA is a failure, or that anything supporters of Medicaid have said is a lie. Moreover, it certainly didn’t show that private insurance or Medicare succeeds in ways that Medicaid fails.
And Ezra Klein and Evan Soltas: Wonkbook: Everything you need to know about the Oregon Health Study