And Mark Kleiman snipes in the rubble:
Scientific illiteracy: Andrew Cohen at The Atlantic demonstrates that you can be a leading “legal analyst” without knowing what the word “random” means or the difference between probability estimation and prophecy.
Zeynep Tufekci launches grenades of reason from the remnants of the tractor factory:
In Defense of Nate Silver, Election Pollsters, and Statistical Predictions: Nate Silver… his work is guided by math, not left or right politics. Yet he’s become a whipping boy…. “The pollsters tell us what’s happening now,” conservative columnist David Brooks told Politico, trashing Silver. “When they start projecting, they’re getting into silly land.” In the same article, MSNBC’s Joe Scarborough added, “And anybody that thinks that this race is anything but a tossup right now is such an ideologue, they should be kept away from typewriters, computers, laptops, and microphones for the next 10 days--because they’re jokes.”
David Brooks is mistaken and Joe Scarborough is wrong. Because while pollsters can’t project, statistical models can, and do … and they do some predictions very well. We rely on statistical models for many decisions every single day, including, crucially: weather, medicine, and pretty much any complex system in which there’s an element of uncertainty to the outcome. In fact, these are the same methods by which scientists could tell Hurricane Sandy was about to hit the United States many days in advance….
Dismissing predictive methods is not only incorrect; in the case of electoral politics, it’s politically harmful… perpetuates the faux “horse-race” coverage that takes election discussions away from substantive issues… silly, often unfounded, time-wasting exercise in fake punditry about who is 0.1 percent ahead. There may well be reasons to consider Ohio a toss-up state, but “absolute necessity for Romney to win the state if he wants to be president” (as Chris Cillizza argues) is not one of them….
“If there’s one thing we know, it’s that even experts with fancy computer models are terrible at predicting human behavior.” So said David Brooks in his recent New York Times column…. [Y]es, there’s no point in checking individual polls every few hours. But experts with fancy computer models are good at predicting many thing in the aggregate. This includes the results of elections, which are not about predicting a single person’s behavior (yes, great variance there) but lend themselves well to statistical analysis (the same methods by which we predicted the hurricane coming)…. This isn’t wizardry, this is the sound science of complex systems. Uncertainty is an integral part of it….
So if Brooks wants to move away from checking polls all the time, he should support more statistical models. And we should hope for more people like Nate Silver and Sam Wang to produce models that can be tested and improved over time. We should defend statistical models because confusing uncertainty and variance with “oh, we don’t know anything, it could go any which way” does disservice to important discussions we should be having on many topics – not just politics.
Drew Linzer looks at survey bias:
Another Look at Survey Bias: [T]here’s essentially no time left in the campaign for preferences to change any further: if the state polls are right, then Obama is almost certain to be reelected…. A relatively small number of survey firms have conducted a majority of the state polls, and therefore have a larger influence on the trends and forecasts generated by my model… some leaning more pro-Romney, others leaning more pro-Obama. As I said at the outset, we’ll know on Election Day who’s right and wrong….
There have been hundreds of smaller organizations who have released fewer than a half-dozen polls each. Most have only released a single poll. We can’t reliably estimate the house effects for all of these firms individually. However, we can probably safely assume that in aggregate they aren’t all ideologically in sync…. We can then compare the overall error distribution of the smaller firms’ surveys to the error distributions of the larger firms’ surveys…. If the smaller firms’ errors are distributed around zero, then the left-leaning firms are probably actually left-leaning, and the right-leaning firms are probably actually right-leaning, and this means that they’ll safely cancel each other out…. [I]f the smaller firms’ error distribution matches either the left-leaning or the right-leaning firms’ error distribution, then it’s more likely the case that those firms aren’t significantly biased after all, and it’s the other side’s polls that are missing the mark.
What do we find? This set of kernel density plots (smoothed histograms) shows the distribution of survey errors among the seven largest survey organizations, and in grey, the distribution of errors among the set of smaller firms. The smaller firms’ error distribution matches that of Quinnipiac, SurveyUSA, YouGov, and PPP. The right-leaning firms – Rasmussen, Gravis Marketing, and ARG – are clearly set apart on the pro-Romney side of the plot.
If, on Election Day, the presidential polls by Quinnipiac, SurveyUSA, YouGov, and PPP prove to be accurate, then the polls by Rasmussen, Gravis Marketing, and ARG will all have been underestimating Obama’s level of support by 1.5% consistently, throughout the campaign. Right now, assuming zero overall bias, Florida is 50-50. The share of Florida polls conducted by Rasmussen, Gravis Marketing, and ARG? 20%. Remove those polls from the dataset, and Obama’s standing improves.
