The Law of Large Numbers

I continue to shake my head in amazement as I consider the most bats* ignorant thing I have read all summer: the claim in National Review that in order to get a picture of income distribution and mobility in America:

Intellectual Garbage Pickup: you'd have to track hundreds of millions of individuals.... [N]one of this is reliable... the Panel Study of Income Dynamics... tracks only 8,000 families out of a U.S. population of 295 million individuals...

The whole purpose of the science of statistics is to tell us that this is simply not true. As long as you can take a random sample of your population, you can find out an enormous amount about the population from a relatively small number of observations. You can find out what proportion of rich people had poor paretns, or what proportion of twenty year olds think they will graduate from college, or pretty much any other average proportion that you want.

Now the "random sample" part of this is very important. But if your sample is random--if the fact that the yes-no pattern of observations so far makes it no more (or less) likely that you next observation will be a "yes"--then the law of large numbers tells us that the sample average you compute will converge to the true population average at a frighteningly rapid speed.

The standard demonstration of this is to repeatedly flip a coin and count the excess proportion of heads over tails. We know that--with a coin flipped and caught in the air by a human being at least--the population average taking all coins that have ever been flipped of the excess proportion of heads is zero. How many observations do we have to take--how many coin flips--before the sample average converges to this population average of 0% excess heads?

Let's see. Here's one run of 1,000 "flips" from Excel's internal random number generator:

Here are ten more:

Impressive, no?

Try some yourself.

You could have a population of 295 million flipped coins. Yet you don't need to look at "hundreds of millions" of them to determine what is going on. Looking at 1,000 will do.

This is the principal insight of the science of statistics. it is an important insight. It is a powerful insight. It is also not an obvious insight--that's what makes it powerful and important.

Yet because statistical studies sometimes produce results ideologically inconvenient for the Republican Party, National Review feels it has to pretend that this insight doesn't exist.

That's really sad.

Comments

[zzi]

As they say, there are three types of lies: lies, damn lies, and statistics. The folks at the National Journal obviously have decided to stick to the first two, but only because they don't seem to have figure out how to make the third work for them yet.

[ken melvin]

Statistics work real well when the results agree with my opinion. They don't prove a damn thing when they disagree.

[howard]

the summer's only half over, prof: who knows what wonders await you as the weeks go by?

[P O'Neill]

What makes this War on Sampling even more dissonant is that the same scribes who are suspicious of small sample sizes will, in the next breath, cite "the law of averages" as a rationale for everything from Dubya's mission to Mars (by the "law of averages," Earth will eventually be hit by an asteroid, so we'll need Mars), to justification for WMD hype (by the "law of averages," terrorists will eventually get a nucular weapon).

[CapitalistImperialistPig]

You got it right in your penultimate paragraph. Their problem is not stupidity but mendacity.

[Mike Maltz]

I used to use the following example with my students:

Suppose you're making chicken soup. You cut up a chicken, an onion, some carrots, and throw in a bay leaf (my late mother's advice), and add salt. After cooking and stirring it for an hour, you check to see if you put in enough salt. How much do you have to taste to test it? [A teaspoon.]

Now you're making soup for the Eighth Army. In the soup vat you cut up and put in a few hundred chickens, 50 lbs. of onions, 100 lbs. of carrots, and of course the bay leaves -- and maybe a few lbs. of salt. After cooking and stirring it for an hour, you check to see if you put in enough salt. HOW MANY GALLONS do you have to taste to test it? [A teaspoon, of course.]

This got them to realize that the precision in sampling had little to do with the size of the population, and led to discussions on how they would see if there was enough chicken in the soup (you would sample from many different parts of the pot/vat), because you could assume that the salt was uniformly distributed but not the other ingredients.

[CapitalistImperialistPig]

It's their readers who are (apparently) stupid.

OK. OK. Their writers are pretty stupid too.

[Walt]

Mike: That's a clever example.

[Hal Varian]

"We know that--with a coin flipped and caught in the air by a human being at least--the population average taking all coins that have ever been flipped of the excess proportion of heads is zero."

How naive! Let's look at some real data:
http://www-stat.stanford.edu/~susan/papers/headswithJ.pdf

OK, I'm not entirely serious. But there is an interesting bit of probability hidden in your sequence of coin clips in graph 2. One might think that the "excess proportion of heads" would oscillate around 0, spending about half its time on one side and half its time on the other. But a look at your plot shows that this doesn't seem to be the case. In fact, the majority of the time, the excess of heads will stay on one side of the line, consistent with your simulations. (See Feller, volume I.)

[Jonathan Goldberg]

The National Review's arguement is not as funny when the Supreme Court applies it to statistical correction of Census data, thus underrepresenting guess who.

[Dennis]

Just for the record, the theorem that tells you this exactly is called the (or in fact a, as it's so useful the name has been used for many other things) Chernoff bound (wikipedia: http://en.wikipedia.org/wiki/Chernoff_bound). In words, the chance that the deviation of the average of a sample of size n from the expectation of a single trial is more than epsilon decreases exponentially with epsilon and n. This is probably my favorite theorem with political applications of all time, since it tells us quickly why things like health and social insurance should be universal as well as justifies polling...

If only I could find a politics-y use for the version on expanders graphs...

[A. Zarkov]

“The whole purpose of the science of statistics is to tell us that this is simply not true.”

Finite population sampling is not the *whole* purpose of statistics. It’s a very important activity within statistics, but not the only one. It also has its controversies. For example do we use model-based inferences in population sampling? There are whole books dealing with this question.

“ ... law of large numbers tells us that the sample average you compute will converge to the true population average at a frighteningly rapid speed.”

This statement is generally true, but borders on dogmatism. If the finite population itself is a sample from a stable distribution, then the statement is false as far as inference goes about the parent distribution.

For a simple thing like gross income, a longitudinal study consisting of a relatively small sample does provide some information income trends for the US population as a whole. So I completely agree with De Long that Luskin statement is ignorant. I’m not sure how PSID handles the effects immigration. Perhaps someone can expand on this.

It’s important to realize that finite population sampling becomes gets difficult when dealing with high dimensional data. For example the National Halothane Study, which dealt with anesthesia and postoperative hepatic necrosis ran into this problem. Moses et al did some very clever things to get the correct inference about the safety of Halothane. Even though the study had a large sample size (850,000) the analysis wasn’t easy.

[Dave]

The observation that "the majority of the time, the excess of heads will stay on one side of the line" holds because, while the relative discrepancy converges to 0, the absolute discrepancy diverges. (the former occuring because of dividing the latter's n^.5 by n) One may also think of partition functions: there are many more possible ways for a sample to have a slight excess than for it to have no excess.

If one looks at wealth spreads among different age cohorts, one finds that the distribution, which is lumped among the young, slightly flattens out to produce a very long tail among the old. Some people, looking at that long tail (especially those who find themselves out along it) are apt to see Social Darwinism at work; others may find a simple brownian diffusion process more parsimonious.

[Tom Womack]

If you take the run out well past a thousand coins, plot the deviation from n/2, and use increment-pixel rather than set-pixel as your plotting function, the Law of Large Numbers gives you a lovely comet-shaped display, essentially parabolic and bristling decoratively at the ends.

[Ronald Brak]

Actually I was just teaching my friend's kids how to flip a coin so it always comes up heads. (It's easy when you use an Australian fifty cent piece which is an enormous chunk of metal and contains more mineral wealth than some small countries.) My intent was to terach them not to gamble by showing them how easy it is to be cheated, but I think I may have actually succeeded in turning them into con artists.

[anne]

What is destructive is that repeatedly for awful political maniulative purposes such publicans and writers are falsifying what is knowable. The idea of reading such a publication is to me unthinkable.

[Barry]

One important thing to keep an eye on is how this spreads. This theory trashes economics; any econ Ph.D. who based a dissertation on sampled data has commited fraud, by NR standards. And any economic theory which uses probability models to produce population models is similarly trashed.

But how many right-wing economists will utter even the feeblist peep?

A while back the Heritage Foundation sponsored a symposium on Intelligent design. Somebody described this as the convergence of the Laffersphere with the Lysenkosphere.

In this war on reality, there seem to be none on the GOP side who support reality.

[anne]

Darn, I think the problem is more serious. The problem is a continual distortion of what is knowable and known. Statistics is a meaningless science when it does not support just what we are supposed to know. Biology is built on science when it should be built on religious doctrine. Environmental science and ecology are simply entirely beyond foundation.

[TimW]

Anne... I share your outrage. The more that publications like the National Review publish this crap, the closer we get to the day when people stop accepting it.

[Francis]

this is being the worst kind of pedant, but the x-axis is incorrectly labeled. It should be "flips" not "trials".

[Gareth]

The only people who can do anything about this are numerate righties. Every one of them must mock National Review once a week. Before we listen to them on any subject, we must ask if they have mocked National Review this week. Until they do, we will not discuss CAFTA or Social Security or anything else...

[PrahaPartizan]

Wouldn't the NR's philosophy void every political poll as well? How could we know what the voters favor if we have to query millions of them on each and every point? Better yet, is a ballot in which only 50% of the population votes valid? Wouldn't it take at least 90% of the registered voters actually participating to yield a valid election? By the NR's own doctrine Dubya and the posse are illegitimate. QED

[eyelessgame]

It's a necessary caution to say that you need an unbiased sample -- though of course a big part of the applied science of statistical sampling is there to ensure an unbiased sample.

But it becomes far too easy for anyone to dismiss anything by claiming bias, which disheartens me, as I just brought this up to a number of intelligent friends and they responded by saying 'but you can't be sure you've eliminated bias!' as though it was a mantra for avoiding statistical results they dont' like.

[Uncle Jeffy]

Once again, Brad, your mastery of understatement serves you well. I was trying to think of where I've encountered the NRO view of the world before ... wait ... I've got it:

"Doubleplus ungood"

Somewhere, George Orwell is laughing himself silly.

[derek]

I'd like a couple more asterisks on the bats*** word, please. I spent a minute or so looking for the footnote for "bats".

[anne]

TimW

Clever point :) May they publish and publish, for we can properly respond. A nice excuse for playing with statistics.

[Saudi Albertan]

Since false dichotomies are the flavor of the day in political discourse (with us/against us;good/evil, etc.), I propose one for the NR: vulgar propagandists or innumerate morons (in other words, liars or dupes).

A query for those of you in academia: how do people graduate from university without some slight knowledge of basic statistics?

[natasha]

"The National Review's arguement is not as funny when the Supreme Court applies it to statistical correction of Census data, thus underrepresenting guess who."

The point of the argument against that decision is that the sample wasn't sufficiently random, and it very well may not have been. As an extreme example, a nationwide random poll of around 1,000 individuals can be highly predictive of election results. However, if you took half your respondents from retirement communities in Florida and the other half from households in South Central L.A., your sample is then pretty close to worthless.

There are only two possible values for a coin toss, no matter how many you try. Any sample of 1000 flips from one coin or one flip on a thousand different coins gives similar results, but this obviously doesn't apply to people. As the number of variables increases, the difficulty of randomization increases with it.

[Kaleberg]

In the twenty five years we lived in a Boston suburb we were never called once by a polling company. In the five years we have lived in a small town on the west coast, we have been called about 50 times. Is there some sampling bias going on? Do pollsters figure they need a couple of people from Boston and a couple of people from Port Angeles to get a fair sample?

Let's face it. The empiricists are losing. Western culture is dead. The assault is from the left and the right. Statistics will be a forgotten art, and our descendents will pay for it. We can only hope it is someday rediscovered.

[AllenM]

Lies, damned lies, and statistics.

Western culture is not dead. We did a fine job building up western culture before we fully got ourselves to the point of infinitely infitesimal variation in statistics. Brad did you run your thousand point simulation one thousand times and account for the rounding bias in the random number generator? Remember when the pentium was found to be flawed....

[Auros]

Hal Varian (er, hi, I kinda dropped out of the program you run *sigh*) said: "One might think that the "excess proportion of heads" would oscillate around 0, spending about half its time on one side and half its time on the other. But a look at your plot shows that this doesn't seem to be the case. In fact, the majority of the time, the excess of heads will stay on one side of the line"

Though he didn't say, this is of course a consequence of independence. Given the tendency of the total number of heads over an ever-longer sequence to converge, if we know that at flip F we were on one side of the line (more heads, or more tails), then we should predict that in the limit as N-->infinity, the at flip F+N we will be on the SAME side of the line.

[Movie Guy]

This is a good post.

I sent the link to a few economists and political types who always complain that the samples are too small.

They just don't like the results...

[derrida derider]

I agree the article should be pilloried, but there is a half-truth in there buried under the innumeracy and dishonesty.

The PSID is a panel survey, and such surveys are inherently prone to become ever less representative of the population over time due to attrition and recruitment bias. Further, a panel survey is specifically geared to research questions about changes in individuals over time. Many 'fixed errors' will wash out; that is, as long as the bias is constant over time we can still track important changes. So the panel's designers are usually more willing to trade off population representativeness for other things than a cross-sectional survey's designers would be.

[cynic]

I suggest you remember this NR comment for the next time they start trying to prove anything they say with statistics. I wonder if they realize inflation, unemployment, GDP, dollar stability, ... is all based on sample statistics. They probably believe the CPI is calculated from data on every sale of every product at every moment of every month.

[Alex R]

While I agree with Brad about the willful ignorance about random samples in articles such as this, I also agree with those such as derrida derider who note some of the difficulties in sampling.

I would also point out that if you're interested in where the money goes, the extreme skew in the distribution of incomes means that what would ordinarily be considered a large sample (N=1000 or even 8000) might have some considerable inaccuracies. You can't really understand income distribution without knowing about what's going on in the 99th percentile, the 99.9th percentile, and perhaps even the 99.99th percentile -- which would correspond to the 20,000 highest-earning households in the country, assuming 200 million households (WAG). But you'll (probably) completely miss those households in a random sample of 1000 -- it'd better to get a separate sample from just the top 1% to get fine-grained detail of the top of the income distribution.

[Brad DeLong]

Emmanuel Saez down the hall (who knows more about the upper tail than anyone else I know) is confident that it follows a Pareto distribution. If you're willing to believe that, then we can make statements about the upper tail.

[cm]

Small caution to sampling -- there are bound to be methodology biases based on how you select and contact individuals. I imagine that it is difficult to catch people without stable residences, e.g. the homeless or "nomads". In a "stable" economy, you can take a shot at modeling them, but in times of change the modeling may become unreliable. But then there is always the escape route of dropping them under the table altogether. Can't hurt the results, no?

[Alex R]

While I was not previously familar with the Pareto distribution, it certainly looks like a reasonable model for income distribution. Verifying that it works at the 99th percentile and up, though, will still require sampling techniques that capture a sufficiently large sample of high income households... I can think of lots of reasons why this might be hard to obtain -- such households have good reasons to keep information about their income away from the prying eyes of economists and the IRS.

(Clearly the only way to solve this problem is to sample high income directly: I volunteer for the $1 million/year and up cohort :-)

[Stephen Harris]

"How many observations do we have to take--how many coin flips--before the sample average converges to this population average of 0% excess heads?"

With 10 random flips of a coin the difference
might be 6 heads and 4 tails. A percentage
difference of 20%. A discrepancy value of 2
extra heads. In a million flips, the law of
large numbers predicts something like .50001
heads and .49999 tails, a percent difference
which is converging. But the discrepancy
difference diverges; there will be more head
then tails tossed by more than 2. Your post
seemed to indicate that the law of large
numbers meant that there would be a balance
achieved after some huge number of coin flip
and that is not so. The deviation increases.