## Journal Refereeing Practices and Polynomial Regression: Threat or Menace?

A *cubic* of latitude!?!? So, other things being equal, it is really bad--14 years lower life expectancy--to live 5 degrees south rather than 10 degrees north of the Huai River? And as good to live 15 degrees south of the Huai River as to live 10 degrees north?

The feel (and I could well be wrong) as that at some point somebody said: "This is very important, but it won't get published without a *statistically* significant headline finding. Torture the data via specification search until we find a statistically significant effect so that this can get published!"

Andrew Gelman:

Evidence on the impact of sustained use of polynomial regression on causal inference (a claim that coal heating is reducing lifespan by 5 years for half a billion people): Yu Xie thought I’d have something to say about this recent paper, “Evidence on the impact of sustained exposure to air pollution on life expectancy from China’s Huai River policy,” by Yuyu Chen, Avraham Ebenstein, Michael Greenstone, and Hongbin Li, which begins:

This paper’s findings suggest that an arbitrary Chinese policy that greatly increases total suspended particulates (TSPs) air pollution is causing the 500 million residents of Northern China to lose more than 2.5 billion life years of life expectancy. The quasi-experimental empirical approach is based on China’s Huai River policy, which provided free winter heating via the provision of coal for boilers in cities north of the Huai River but denied heat to the south. Using a regression discontinuity design based on distance from the Huai River, we find that ambient concentrations of TSPs are about 184 μg/m3 [95% confidence interval (CI): 61, 307] or 55% higher in the north. Further, the results indicate that life expectancies are about 5.5 y (95% CI: 0.8, 10.2) lower in the north owing to an increased incidence of cardiorespiratory mortality.

Before going on, let me just say that, if you buy this result, you should still be interested in it even if the 95% confidence intervals had happened to include zero. There is an unfortunate convention that “p less than .05″ results are publishable while “non-significant” results are not. The life expectancy of 500 million people is important, and I’d say it’s inappropriate to wait on statistical significance to make that judgment.>Getting to the details, though, I’d have to say that I’m far less than 97.5% sure that the effects are in the direction that the authors claim (recall that 97.5% is the posterior probability associated with p=.05 under a flat prior). And, for the usual proper-prior Bayesian reasons, I’d guess that this “2.5 billion years of life expectancy” is an overestimate….

Table S.9 in the supplemental material gives their results trying other models. The cubic adjustment gave an estimated effect of 5.5 years with standard error 2.4. A linear adjustment gave an estimate of 1.6 years with standard error 1.7….

The headline claim, and its statistical significance, is highly dependent on a model choice that has no particular scientific (as distinguished from data-analytic) basis. Figure 3 above indicates to me that neither the linear nor the cubic nor any other polynomial model is appropriate here; that there are other variables not included in the model that distinguish the circles in the graph…. I’d like to ditch that approach of estimating high-degree polynomials….

Let me emphasize that I’m not not not saying that particulate matter doesn’t kill, or that this topic shouldn’t be studied, or that these findings shouldn’t be published in a high-profile journal…. What I am suggesting is a two-step: the authors retreat from their strongly model-based claim of statistical significance, and the journal accept that non-statistically-significant findings on important topics are still worth publishing….

Let me say all this again:… Respiration is important. Increasing lifespan is important…. Air pollution is bad for you. Policies matter. Environmental policies and environmental outcomes can and should be studied by quantitative researchers. Data are never perfect but we still need to move forward. Subject-matter researchers spend decades of their life establishing subject-matter expertise, and they don’t always have full statistical expertise. That is fine. There is a division of labor…. Regression discontinuity is a great idea. Causal inference from observational data is difficult but still needs to be done. There’s often no easy way to control for background variables in a quasi-experiment. These researchers did the best they could…. Even if their work is imperfect it should not be dismissed. I focus on a statistical concern with this paper because statistics is what I do…. I am skeptical about an effect of 5 years of life but I could be wrong. I think it would be fine for the journal to publish an article just like this one, but without the 3rd-degree polynomial and with a smaller and non-statistically-significant estimate of the effect. I have the impression that this and other journals have an implicit rule that under normal circumstances they will publish this sort of statistical paper only if it has statistically significant results. That’s a regression discontinuity right there, and researchers in various fields have found evidence that it introduces some endogeneity in the selection variable.

The above post is not an attempt to shoot down the article by Chen et al…. I… wouldn’t mind stirring up some post-publication peer review here, and maybe some subject-matter experts can contribute usefully. Again, this is an important topic. Lives are at stake, it’s worth capturing our scientific uncertainty as well as we can here, even if it means that I critique a published paper and possibly embarrass myself in the process.

Got it? Got it. Good.