Dear Mr. Shepard:
I was puzzled by your article: http://www.nationaljournal.com/politics/gallup-blew-its-presidential-polls-but-why--20121118.
Were I writing an article about Gallup's remarkable miss this year, I would have started by Googling "Gallup 2012 election criticisms", which would rapidly have led me to Nate Silver's "Gallup vs. the World" http://fivethirtyeight.blogs.nytimes.com/2012/10/18/gallup-vs-the-world/ --an October 18 real-time analysis of the Gallup anomaly in considerable depth. Reading that, I would have learned--back last October 18--that:
[Gallup's] results are deeply inconsistent with the results that other polling firms are showing in the presidential race, and the Gallup poll has a history of performing very poorly when that is the case. Other national polls show a race that is roughly tied on average, while state polls continue to indicate a narrow advantage of about two points for President Obama in tipping-point states like Ohio....
[T]he Gallup poll accounts for 5 or 10 percent of the information that an election analyst should evaluate on a given day. The Gallup poll’s influence on the subjective perception about where the presidential race stands seems to be proportionately much greater than that, however — especially when the poll seems to diverge from the consensus. This simply isn’t rational, in my view. As I discuss in my book, our first instincts are often quite poor.... We tend to put too much emphasis on the newest, most widely reported and most dramatic pieces of data....
Usually, when a poll is an outlier relative to the consensus, its results turn out badly. You do not need to look any further than Gallup’s track record over the past two election cycles to find a demonstration of this. In 2008, the Gallup poll put Mr. Obama 11 points ahead of John McCain on the eve of that November’s election.... The average of polls put Mr. Obama up by about seven points. The average did a good job; Mr. Obama won the popular vote by seven points. The Gallup poll had a four-point miss, however. In 2010, Gallup put Republicans ahead by 15 points on the national Congressional ballot, higher than other polling firms, which put Republicans an average of eight or nine points ahead.... Republicans won the popular vote for the United States House by about seven percentage points — fairly close to the average of polls, but representing another big miss for Gallup.
Apart from Gallup’s final poll not having been especially accurate in recent years, it has often been a wild ride to get there. Their polls, for whatever reason, have often found implausibly large swings in the race. In 2000, for example, Gallup had George W. Bush 16 points ahead among likely voters in polling it conducted in early August. By Sept. 20, about six weeks later, they had Al Gore up by 10 points instead: a 26-point swing toward Mr. Gore over the course of a month and a half. No other polling firm showed a swing remotely that large. Then in October 2000, Gallup showed a 14-point swing toward Mr. Bush over the course of a few days, and had him ahead by 13 points on Oct. 27 — just 10 days before an election that ended in a virtual tie.
In 1996, Gallup had Bill Clinton’s margin over Bob Dole increasing to 25 points from nine points over the course of four days.
After the Republican convention in 2008, Gallup had John McCain leading Mr. Obama by as many as 10 points among likely voters. Although some other polls also had Mr. McCain pulling ahead in the race, no other polling firm ever gave him larger than a four-point lead.
It’s not clear what causes such large swings, although Gallup’s likely voter model may have something to do with it.
Even its registered voter numbers can be volatile, however. In early September of this year, after the Democratic convention, Gallup had Mr. Obama’s lead among registered voters going from seven points to zero points over the course of a week — and then reverting to six points just as quickly. Most other polling firms showed a roughly steady race during this time period....
Little of this made it into your article. Nothing about Gallup's excessive within-election volatility and long-time lack of final-poll accuracy made it into your article. Short quotes from Alan Abramowitz appear toward the end of the article appear, but only as one of many perspectives--and little is done to convey to your readers the power of Abramovitz's (and Blumenthal's, and Silver's) real-time critique.
If I may ask, why did you choose to write the article you did?
Yours,
Brad DeLong
Professor of Economics J. Bradford DeLong
U.C. Berkeley
delong@econ.berkeley.edu
UPDATE: Steven Shepard replies:
Professor DeLong:
Thanks for your note. This was a reported story about Gallup’s ongoing self-evaluation, not a statistical analysis of their apparent bias this year (or in past election cycles). Mr. Silver did a wonderful job this cycle of chronicling the extent to which Gallup was producing results that were volatile and differed from the consensus of other pollsters.
Within editing constraints, and considering the election results have further established their poor performance, my story attempted to move the ball forward on the debate over why this happened.
Furthermore, I did cite the analysis of Prof. Abramowitz and Mark Blumenthal within the election cycle. Silver’s analysis, while helpful, offered little explanation for why Gallup’s results were what they were. I agree that their results receive outsized attention, and I don’t think anything in my story suggested that they deserved the attention they got from some in the media (admittedly including me and my organization).
Please feel free to contact me with any further questions you may have.
Thanks for reading,
Steve Shepard
Polling Editor, National Journal Hotline
You readers will not be surprised that I find this deeply unsatisfactory.
Any "story about Gallup's ongoing self-evaluation" needs to begin by setting the scene: by providing the context and making readers aware of Gallup's polling errors over what is now the past generation--that is what the self-evaluation is about. Shepard does not do so. And his apparent belief that to inform his readers of what his story is really about would require a different animal than a "reported story… to move the ball forward on the debate over why this happened", would require a "statistical analysis", seems to me to miss the point rather completely.
