Why Oh Why Can't We Have Better Pollsters?

Matthew Yglesias says that friends don't let friends read the Gallup tracking poll:

Matthew Yglesias » The Three Day Itch: One thing a week’s vacation from blogging helps you get perspective on is the Gallup tracking poll. On August 1 when I had my last day at The Atlantic it was time for panic as McCain had tied things up. Then Obama started to regain ground, going up to a four point lead. Then the race tightened again, then Obama opened up a five point lead, and now it’s tightening again but with Obama back to a smallish lead having beaten back the strong challenge McCain was mounting around August 1. In short, McCain’s “Celebrity” ad and drilling attacks were working well, but when the McCain campaign went after Obama on the tire gauge thing he came up with effective countermeasures and regained his lead.

Maybe.

Or maybe none of that happened. As everyone knows, there’s sampling error associated with polling. As a result, if you poll 1,000 people on August 1 and then you poll 1,000 different people on August 2 you shouldn’t be surprised to see the results differ by several percentage points even in the absence of any change in the underlying public opinion. Beyond that, doing one poll per day throughout a long campaign would mean that you’d expect to see one or two relatively rare outlier results per month even under circumstances of total stasis. And as Alan Abramowitz points out if you look at the daily results this is actually what you see — incredible volatility with Obama’s lead oscillating violently around an average of 3-4 points. Since it’s not plausible that the public mood is really swinging anywhere near as rapidly as a very naive reading of the Gallup daily results would suggest, people could see that this is basically statistical noise in a stable race.

But Gallup doesn’t report its daily results, they report a multi-day rolling average. Abramowitz notes that if you report a ten day rolling average, you get a chart where nothing happens — Obama maintains a flat lead of 3-4 points. Again, a stable race. But if instead of doing either of those things you do what Gallup actually does and report a three day rolling average, you get these pleasant looking peaks and valleys in the race. The change over time here is large enough in magnitude (unlike on the ten day chart) but also slow enough in pace (unlike on the one day chart) to be plausibly interpreted as public opinion shifting in response to events. And since the human mind is designed to recognize patterns and construct narratives, and since it suits the interests of campaign journalists to write narratives, people interpret the peaks and valleys of the three day average as real shifts in public opinion. But while I have no way of proving that it’s just statistical noise and nothing’s really happening, the “nothing happening” narrative is completely consistent with the data, and it’s telling that the conventional narratives collapse when the data is presented in different ways whereas the “noise” narrative is consistent with multiple ways of displaying the information...

And, indeed, today the Gallup Organization Writes:

Gallup Daily: Obama Moves Ahead, 48% to 42%: PRINCETON, NJ -- Democratic candidate Barack Obama has gained ground in the latest Gallup Poll Daily tracking average from Monday, Tuesday, and Wednesday, and now leads Republican John McCain among registered voters by a 48% to 42% margin...

The truly diabolical thing about the Gallup Organization is that all the roots of the MA process used to construct the three-day moving average from the raw daily data are on the unit circle, so there is no way to back out the daily numbers from the averages. And even if you do know what the daily poll numbers were at some two consecutive dates in the past so that you can then deduce what the third day's polls in the next moving average were (the three having to add up to the reported average, you see), the rounding errors in each day's results propagate undamped over time and so grow without bound. Here, for example, you see what estimates of the daily numbers you get if you assume that the Obama share on July 18, July 19, and July 20 were all equal to 47%:

What is happening is that the rounding errors are being passed through an amplifying filter with a strong spectral peak at the three-day period--and so the three-day cycles in the estimated daily numbers are freaking out.

One way to get some insight into the data is to notice that the difference between today's moving average and yesterday's moving average is simply equal to one-third times the difference between today's results and the results from three days ago. Given that we have a 900 person daily sample and a vote share near 50%, the standard deviation of each day's sample should be 1.67%, which means:

(1) The standard deviation of the difference between today's sample and the sample of three days ago should be 2.35%--meaning that the daily change in the moving average has a standard deviation of 0.79%. only a one-day change in the moving average of 2% is interesting--smaller changes are likely to be statistical noise from a hypothesis-testing point of view.

(2) The difference between today's moving average and the moving average of two days ago is one-third the difference between the sum of today's and yesterday's sample and the sum of the samples from three and four days ago--meaning that the two-day change in the moving average has a standard deviation of 1.11%. A two-day change in the moving average of 2% is not nearly as interesting as a one-day change of 2%--statistical noise grows from the one to the two-day change.

(3) The difference between today's moving average and the moving average of three or more days ago is one-third the difference between the sum of the most recent three days and the previous or further back three days--meaning that the standard deviation of the three-day change in the moving average should be 1.36%--meaning that a two-day change of 2% is truly not very interesting to a hypothesis tester at all: only a 3% move over three or more days is truly statistically interesting (and, of course, the more persistent such a move is at the more than three-day horizon the more interesting it is).

The right way to deal with the Gallup tracking poll is, I think, to compare it to how it was at some benchmark date in the past, and then to ignore all changes in the Obama or McCain share of less than three percentage points--ignore all changes in the spread of less than six percentage points.

Comments

[degustibus]

I keep be amazed by otherwise seemingly intelligent folks who follow polls and report poll results. signifying nothing. nada. naught. Might as well be reporting on brittney or Lindsey or Paris.

oh why oh why can't we get a better blogger corps?

[George Smiley]

Even otherwise smart people in the United States are usually - not sometimes, but usually - functionally innumerate. In particular, almost everyone has no sense whatsoever of how sampling works, what noise is, etc. To a useful first approximation, nobody at all understands the central limit theorem.

It's hopeless.

[Robert Waldmann]

There is a fairly serious problem with Yglesias's argument "f you poll 1,000 people on August 1 and then you poll 1,000 different people on August 2 you shouldn’t be surprised to see the results differ by several percentage points even in the absence of any change in the underlying public opinion."

Gallup does no such thing. They poll around 900 then average over 3 days. Yglesias first wrote that it could be sampling error when discussing the alleged Obama bounce due to his world trip. The null that that alleged bounce was due to sampling error can be tested and rejected.

http://rjwaldmann.blogspot.com/2008/07/matthew-yglesias-hypothesizes-that.html

The convention bounce is also too large to be sampling error (in my head I calcualted about 3 standard errors). There is a difference between normal fluctuations which tell us virtually nothing about who will be our next President and fluctuations due to sampling.

It is true that August polls are of almost no value in predicting election outcomes. It is truer that August changes of a few percentage points in Obama minus McCain are of even less value than the fact that Obama is generally ahead. It is truest (because it can be proven) that the Gallup tracker is not bouncing around due to sampling error alone.

[Robert Waldmann]

You can't back out daily polls but you can test the null that true opinion hasn't changed. for non overlapping polls this is easy. variance of Obama - McCain in one poll is around 1/3000 (less than 1/(sample size) because some people are undecided so the correlation of "for Obama" and "for McCain" is not exactly -1). Var of dif of dif is about 1/1500 so se of dif of dif is about 2.7% (just tried to calculate a square root in my head).

With one day overlap you can calculate the change in 2 day averages (3/2)(change in 3 day as one day is the same). So about 10% convention bounce so far. sample sizes only around 1800 so var dif around 1/2000 so var dif of dif around 1/1000 se around 3.2% so change over 3 standard deviations. The evidence of a convention bounce (including Michelle and Hillary but not Bill and Joe) is statistically significant.

People do change their minds based on cheering Germans, dumb dumber dumbest negative ads and party conventions. Those people might be so flighty that there is no way to guess what the hell they will do on election day, but they do exist.

"Normal fluctuations which you shouldn't have a cow about because they tell you virtually nothing about who will be elected" and "fluctuations due to sampling error" are not synonymous statements.

[Robert Waldmann]

I had to google "Why Oh Why Can't We Have Better Pollsters?" to get a permalink. I tried with firefox and internet explorer (grrr) and all I got was your main URL. I wonder if anyone else has this problem. No prob googling, but I hope this message is useful.

[Measure for Measure]

The tragic thing is that Gallup presumably has statisticians on its staff who understand all of this.

"(1) The standard deviation of the difference between today's sample and the sample of three days ago should be 2.35%--meaning that the daily change in the moving average has a standard deviation of 0.79%. only a one-day change in the moving average of 2% is interesting--smaller changes are likely to be statistical noise from a hypothesis-testing point of view."

Er - help me out. 1.96 x 0.79 = 1.548 ~= 1.5, so that a one day change in the moving average of over 1.5% is interesting, which is meaningfully less than 2.0%. Or am I wrong? Or was one significant digit chosen intentionally (2% = 1.548% != 2.0%)?

[jamzo]

another way to deal with polls is to check in with mark blumenthal

http://www.pollster.com/blogs/

[Zeno]

It's exasperating how commentators react to natural random fluctuations in polls as if each little wiggle is full of significance. Polls are samples. They vary! Sheesh. If only they had paid attention in statistics class (or bothered to take one). I drew some graphs about random variation in my blog post Roller-Coaster Polling. Click on my name for the link.

[bigTom]

Somewhere in the bowels of both campaigns (and probably some other quarters as well) you can bet that there are folks is job it is to discern from polls, and other studies, if the various campaign tactics are working or not. For such a person, the slope of the poll result -or even just the sign of the slope, is the important data. I wonder how many tactics/ advertisements etc. are continued, or yanked from the air due to false signals arising from sampling error.

[David Lloyd-Jones]

.

I'm old enough that I remember when Pinky Lee and Nick Samstag and Doctor Dichter and Bill Benton (the first Senator to call tailgunner McCarthy down) were inventing all this stuff.

One way of keeping the social sciences, in which I include polling, in perspective is to remember that Enrico Fermi and Edward Teller were at University of Chicago together -- about the time Fermi was buried in the squash courts under the stands. And before Teller had been taken on by the Manhattan Metallurgical District...

So anyway, one day Fermi hears that this guy is giving a lecture on "sociology," a whole new thing. So he rushes into the Faculty Club and tells Teller "Edvard, Edvard, you've got to come and hear this."

They go to the lecture. Fermi asks Teller "Hey, hey, what did you think?"

Teller's dry observation: "Sociology? Just another Jewish cult."

.