Elementary Philosophy of Probability and the War on Nate Silver: The (Not Very) Honest Broker for the Week of August 2, 2014

Of all the weird things that have happened in the American public sphere in my life, the most weird was the War on Nate Silver--launched in the fall of 2012 by David Brooks, Joe Scarborough, Dylan Byers, and a remarkably large company.

The underlying argument appeared to be that Nate Silver was doing something wrong and unfair by using... evidence. By... counting things. By... using statistics. By... estimating probabilities...

A few of the "best" examples:

David Brooks on the PBS NewsHour:

What I hate are the forecasts, when they say so and so has a 66 percent chance of winning or a 32 percent chance of winning.... If you tell me you think you can quantify an event that is about to happen that you don’t expect, like the 47 percent comment or a debate performance, I think you think you are a wizard. That’s not possible...

Dylan Byers: Nate Silver: One-term celebrity?:

So should Mitt Romney win on Nov. 6, it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and--one week from the election--gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent...

Morton Kondracke: Election Oddsmakers Suffering From Fuzzy Math::

Nate Silver, the New York Times’ election modeler, gives Obama a 65.7 percent chance of winning.... Intrade... had Obama as a 64.2 percent favorite as of today.... Romney came off as a plausible president... while Obama didn’t engage--and that the election’s been moving in Romney’s direction ever since.... The Real Clear Politics average of recent polls has Romney up by 1 point--meaning, a tie--but the Gallup seven-day tracking poll shows Romney up by 7 points among likely voters.... [If you] place a bet in... London [on Romney], you might just make a pile...

Peggy Noonan: Obama and His Team Have Lost That Loving Feeling:

The other day a Republican political veteran forwarded me a hiring notice from the Obama 2012 campaign. It read like politics as done by Martians.... "Analytics Department"... "predictive modeling/data mining" specialists to join the campaign's "multi-disciplinary team of statisticians".... Is that what politics is now? Or does the Obama re-election effort reflect the candidate and his flaws?...

And it seems to be not, or not entirely, a Democrat-Republican thing:

Natalia Cecire: The Passion of Nate Silver (Sort Of):

The "Nate Silver phenomenon" is a perfect example of Second Gilded Age puerility... a form of boyishness... a Tom Sawyer who insists on playing even when a slave's freedom is at stake... it is entirely appropriate that his statistical forecasting began not in politics but in sports. Nate Silver's models cannot... explain, say, the role of race... cannot give definitive predictions either, only probabilities.... Statistics is an inherently puerile discipline.... It's not understanding.... Silver's most ardent defenders are wholly immersed in the logic of puerility... most notably Silver's fellow statistical Wunderkind, Ezra Klein.... The moral absence at the heart of statistical methods.... A Nieman Lab defense... celebrates that "FiveThirtyEight has set a new standard for horse race coverage".... That this can be represented as an unqualified good speaks to the power of puerility...

And there's more! A lot more!! An awful lot more!!!

But what has reminded me of all this?

The reopening of the War on Nate Silver by National Review and its Charles Cooke: Smarter than Thou:

Insecure hipsterism... unwarranted condescension... self-professed nerds... (a) the belief that one can discover all of the secrets of human experience through differential equations and (b) the unlovely tendency to presume themselves to be smarter than everybody else in the world... Melissa Harris-Perry, Rachel Maddow, Steve Kornacki... Chris Hayes... Ezra Klein, Dylan Matthews... Matt Yglesias... Nate Silver... Paul Krugman... Richard Dawkins... Al Gore... Bill Nye... anybody who conforms to the Left’s social and moral precepts while wearing glasses and babbling about statistics.

The pose is, of course, little more than a ruse.... I’m smart... not... southern, politically conservative, culturally traditional, religious in some sense, patriotic, driven by principle rather than the pivot tables of Microsoft Excel, and in any way attached to the past.... the fadlike fetishization of “Big Data” is merely the latest repackaging of old and tired progressive ideas.... “It’s just science!” is... a bullying tactic... to pretend that Hayek’s observation that even the smartest of central planners can never have the information they would need... was obviated by the invention of the computer. If politics should be determined by pragmatism, and the pragmatists are all on the left... well, you do the math....

Progressives... believe... unscientific things--that Medicaid, the VA, and Head Start work; that school choice does not; that abortion carries with it few important medical questions; that GM crops make the world worse; that one can attribute every hurricane, wildfire, and heat wave to “climate change”; that it’s feasible that renewable energy will take over from fossil fuels anytime soon--but also do their level best to block investigation into any area that they consider too delicate.... Perhaps the greatest trick the Left ever managed to play was to successfully sell the ancient and ubiquitous ideas of collectivism, lightly checked political power, and a permanent technocratic class as being “new,” and the radical notions of individual liberty, limited government, and distributed power as being “reactionary”...

But why is this tired table-pounding worth writing about?

Because there is an interesting intellectual point underneath it all in at least one of the fronts in the War on Nate Silver.

We see it, a bit, in Jonah Goldberg's Nate Silver's Numbers Racket:

An intense kerfuffle broke out over the poll-prognosticator Nate Silver and his blog at the New York Times, FiveThirtyEight. Silver, a statistician, has been predicting a decisive Obama victory for a very long time, based on his very complicated statistical model, which very, very few of his fans or detractors understand. On any given day, Silver might announce that--given the new polling data--"the model" now finds that the president has an 86.3% chance of winning. Not 86.4%, you fools. Not 86.1%, you Philistines. But 86.3%, you lovers of reason...

And it is developed more fully by the truly extraordinary Gregg Easterbrook's Absurd Specificity Watch:

Americans seem to love hyperbolic claims of precision--perhaps it makes us feel that science is more efficient than it really is. When Nate Silver of The New York Times forecasts, as he did on the morning of the 2012 presidential voting, that Barack Obama will win re-election with "314.6" electoral votes to "223.4" electoral votes for Mitt Romney, such numbers are received with gravitas--as if the decimal places made them deep, rather than silly. In just two days, Obama's chance of re-election increased from "80.8 percent" to "83.7" percent. A claim of a "83.7" percent chance rather than "a good chance" is seen as turning the speaker into Mr. Spock, when actually ought to make readers giggle...

But why should Silver's claim at the start of November 2012 that his model predicted that Obama had an 80.8% chance of winning the election should "make readers giggle"?

What was wrong with it?

In what sense was the "specificity" "absurd"? If one is going to lay odds after all, one has to pick a real number between zero and one to do so, and of the real numbers in that interval the "absurdly specific" ones are those with a representation with just one or two non-zero digits. You should giggle if someone claims the odds are even, or 3-2, or 2-1, Or 4-1 exactly.

So there is something there that is not obvious.

Let's unpack it: time for a dialogue! [UPDATE: Cf. : Cosma Shalizi (2009)]

Thrasymakhos: So tell us, friend Simplissimus, what your cohorts' objection is. Gregg Easterbook and Jonah Goldberg do not tell us why Nate Silver should make you giggle. Instead, they use that figure of rhetoric that I call "Mean Girls": to mock, and in the process of mocking implicitly warn you that if you admit to not understanding their mockery you will be mocked too. That keeps them from having to outline exactly what their mockery is so the underlying argument can be examined...

Simplissimus: I will give it a try.

Let us think about the kinds of knowledge that we could have about a forthcoming future event, and about the overweening pride of those who falsely pretend to forms of knowledge and certainty that they cannot have...

Sokrates: Let's be specific. Is there something that was once in the future but is now in the past that was bothering you? Something we can examine?...

*Simplissimus: OK. The Obama-Romney presidential election. Nate Silver's false pretense of knowledge that he knew the odds down to the last decimal point.

Sokrates: Back up. Let's start with claims to knowledge that we can all agree are false, just so we can all start on the same page...

Simplissimus: OK...

Sokrates: We could claim about some future event that we know what will happen--that, for example, that we know at the start of November 2012 that Obama is going to win. Is that the kind of false pretense knowledge you are talking about?

Simplissimus: No. Everyone argues that certain knowledge of what future events will or will not come to pass is impossible.

Sokrates: So everyone agrees that claims of certain knowledge about the future are offenses against Tyche: impious and false. Only wizards and prophets claim such knowledge, and there are no true wizards and prophets.

Simplissimus: You speak truly, Sokrates.

Sokrates: But that is not the type of knowledge that Nate Silver claims, is it?

Simplissimus: No.

Sokrates: OK, so Nate Silver does not claim--call that first-order certainty. Nate Silver is much too epistemologically modest to fall into that trap of claiming to know who will be the winner.

Simplissimus: True.

Sokrates: There is another epistemologically arrogant claim to knowledge. Joe Scarborough makes it. He claims to know--with certainty--that the odds on who will win the presidential election are 50-50, and anybody who claims to know anything else is a fraud:

Nobody in that campaign thinks they have a 73% chance--they think they have 50.1% chance of winning. And you talk to the Romney people, it is the same thing. Both sides understand that it is close, and it could go either way. And anybody that thinks that this race is anything but a toss up right now is such an ideologue, They should be kept away from typewriters, computers, laptops and microphones for the next 10 days, because they are jokes...

Call that second-order certainty--that the principle of insufficient reason rules, and that the only true fair odds any honest and rational person can arrive at must be 50-50. Is that Nate Silver's offense?

Simplissimus: No.

Sokrates: So what, then, is Nate Silver's impious and false claim? What is it that offends the goddess Tyche, and because of which we should pay him no attention?

Simplissimus: The offense is that he claims that the odds of Obama winning the election are 80.8%. And then he claims two days later that the odds of Obama winning the election have risen to 83.7%. Nobody can know that. Nobody should claim that.

Sophia: So what is wrong with Nate Silver is that he claims to know the odds?

Simplissimus: Exactly.

Sophia: OK. Let's start dropping into some math here. Suppose there is, as a property of the world, some θ--written in the courses of the stars--that is the true chance that Obama will win the election.

Simplissimus: Why "θ"?

Thrasymakhos: It's a trick we math people use to make you feel insecure. It reminds you that there are valid forms of knowledge we know that you are totally clueless about. And it reminds you that these forms of knowledge are ancient and indeed "classical". It was Glaukon's brother, after all, who inscribed "let no one enter here who knows not geometry" above our gates. It is a way of reminding you to look up at that inscription, and remember that you do not belong in this conversation.

Sophia: We have to call "the property of the world--written in the courses of the stars--that is the true chance that Obama will win the election" something. And to call it "the property of the world--written in the courses of the stars--that is the true chance that Obama will win the election" makes discussion incredibly circuitous. Calling it θ allows us to think more quickly and more accurately.

Thrasymakhos: Perhaps...

Sophia: So your complaint is that Silver claims to know that true θ? And that knowing that true θ is something nobody can do, and something nobody should claim?

Simplissimus: Exactly.

Sokrates: So Silver's impious offense is not that he claims to know who will win--the first-order certainty-- the future, and not knowing that nobody can guess the future and therefore the principle of insufficient reason rules--the second-order certainty--but rather the third-order certainty of claiming to know what the odds really are?

Simplissimus: Exactly.

Sophia: But that is not what Nate Silver does.

Simplissimus: Huh?!

Sophia: Silver has a model. His model produces an estimate. Silver doesn't claim to know what the true odds θ are: all he claims is to have constructed (what he hopes is) an unbiased estimate θ' of the true odds θ.

Simplissimus: I don't understand...

Thrasymakhos: Does Nate Silver actually think that he knows is the chances of Obama's winning the election are 80.8% in any comprehensive sense? No. The most he will say is that that number is the point at which he, personally, would switch from thinking that betting on Obama is a good deal (for small stakes) to thinking that betting on Romney is a good deal (for small stakes).

Simplissimus: But Nate Silver says Obama's odds are 80%! He claims to understand the deep structure of the universe, and know what the true odds θ are! You cannot make a forecast of the odds without such knowledge!

Sophia: Yes you can.

Simplissimus: Huh?

Sophia: The odds you quote are simply the odds you would bet at. They aren't a claim to know what the true odds θ are. They are just your best unbiased estimate θ' of what the true odds are--which is why you shouldn't bet on whether the jack of spades will jump out of the deck and piss in your ear when you are betting against somebody other than Nature, but that is a different discussion...

Simplissimus: But you cannot calculate the true fair odds of Obama winning without knowing what θ is! Nate Silver's claim to have calculated those odds is an impious and false claim to know something about the deep structure of the universe that he cannot! He cannot honestly know that at the the start of November, 2012 the true θ was 80%! But that is what quoting odds requires!

Sophia: Nonsense!

In order to quote odds we do not need to know what the true θ is. We only need to have the best unbiased estimate θ' of θ that we can construct.

Let's consider an example: Suppose Nature has told us that θ is either 0.6 or 1.0, is equally likely to be each, but that we cannot find out any more than that. As long as we have our unbiased θ'=0.8, we can calculate the probabilities perfectly well.

Then--take a breath--there is a 50% chance that Obama has a 60% chance of winning, and a 40% chance of losing. And--take a breath--there is a 50% chance that Obama has a 100% chance of winning and a 0% chance of losing. We can then add up the probabilities:

• a 30% chance that θ is low and Obama wins
• a 50% chance that θ is high and Obama wins

And these up, and see that Obama has an 80% chance of winning. Compute the probability that Obama wins from our θ'=0.80, and see that Obama has an 80% chance of winning.

This generalizes.

We do not need to know the distribution of the true θ. All we need is an unbiased estimate θ' of the true θ. And then we are golden. Game and set.

Simplissimus: But what if the chance that the true θ=0.6 is not 1/2 but 3/4?

Sophia: Then our unbiased estimate θ'=0.7. And when we do the math:

• a 45% chance that θ is low and Obama wins
• a 25% chance that θ is high and Obama wins

And thus a 70% chance that Obama wins. Once again, what we need is an unbiased estimate θ' of the true θ. This generalizes: all we ever need to calculate the odds is an unbiased estimate θ' of the true θ.

Simplissimus: But what if we think the odds that the true θ=0.6 are 1/2 when they are in fact 1/4, and we have a θ'=0.8 and compute that Obama has an 80% chance of winning when in fact he has only a 70% chance?

Sophia: Then our θ' is not an unbiased estimate, is it?

But note that you are no longer claiming that Nate Silver impiously pretends to know something he cannot. You are, instead, merely claiming that Nate Silver's calculations are off, and that his procedure for arriving at his θ' does not produce an unbiased estimate of the true θ. Game, set, and match.

Complexificius: But suppose we don't know with certainty that nature is picking θ=0.6 and θ=1.0 with equal odds. Suppose nature picks a parameter λ first, and with a probability λ picks θ=0.6 and with probability (1-λ) picks θ=1.0. Then constructing an unbiased estimate θ' of the true θ no longer allows you to calculate the odds, does it? You have to know λ, right?

Sophia: Wrong! You can still calculate the odds with your unbiased estimate θ' of the true θ. But in order to construct an unbiased estimate of the true θ you need to know something about λ. Can you guess what that something is?

Thrasymakhos: Do you need to construct an unbiased estimate of λ?

Sophia: Correct! Suppose that there is a 50% chance that the true λ=0.5 and a 50% chance that the true λ=1.0. Then when we run the probabilities, we get:

• a 50% chance that λ is high (1.0):
  • a 50% chance that λ is high (1.0) and θ is low (0.6):
    • a 30% chance that λ is high (1.0), θ is low (0.6), and Obama wins.
• a 50% chance that λ is low (0.5):
  • a 25% chance that λ is high (1.0) and θ is low (0.6):
    • a 15% chance that λ is high (1.0), θ is low (0.6), and Obama wins.
  • a 25% chance that λ is high (1.0) and θ is high (1.0):
    • a 25% chance that λ is high (1.0), θ is high (1.0), and Obama wins.

That's a 70% chance that Obama wins.

If we knew what the true λ was, we could construct an estimate of θ', since θ'=0.4λ+0.4. We don't know the true λ, but if we start with an unbiased estimate of it λ'=0.75, we can still construct an unbiased estimate θ'=0.7. And so we figure Obama's odds at the same 70%. Game...

Thrasymakhos: You are leaning rather heavily on linearity here...

Supercomplexificissimus: But...

Sophia: I know what you are going to say: in order to calculate the true fair odds of Obama winning we had to get an unbiased estimate θ' of the true θ that nature chose. But in order to get that unbiased estimate of the true θ we needed an unbiased estimate λ' of the true λ that nature chose. But nature chooses λ according to some chance process, and we need an unbiased estimate of the α that governs Nature's choice, which requires we know the process, which requires an unbiased estimate of, in turn, β, γ, δ, ε, ζ, η, ι, κ, μ, ν, ξ, ο, π, ρ, σ, τ, υ, φ, χ, ψ, and ω...

Supercomplexificissimus: Correct. Where do all the unbiased estimates of all of these come from?

Thrasymakhos: You do realize you have just rediscovered the unsolved problem of induction?

Sophia: From the same place that your confidence that the sun will almost surely rise in the east rather than in the south tomorrow comes from. The point is that--wherever knowledge comes from--in order to calculate his odds, Nate Silver needs not to know what the true θ is, or even what the probability distribution of the true θ is, but only needs to have an unbiased estimate θ' of the true θ. And--wherever knowledge comes from--the claim that one has an unbiased estimate θ' is an epistemologically modest one, much more modest than knowing what the true θ is or what the distribution of the true θ is or that Obama will win or that Obama will lose or that it is a 50-50 tossup.

Supercomplexificissimus: But surely it matters that we do not know that the true θ=0.8! Surely it matters that all we have is an estimate θ'=0.8 that we hope is an unbiased estimate of the true θ!

Sophia: It doesn't matter for Nate Silver. It doesn't matter for anybody hoping to calculate true fair odds.

Supercomplexificissimus: Who does it matter for, then?

Sophia: Take your θ'=0.8. Suppose that you had 20 parallel-universe earths, and 20 Obamas, and placed 20 bets. Then you would expect to win 16 of them, yes?

Supercomplexificissimus: I suppose so.

Sophia: And if the true θ drawn by Nature were θ=0.8, you would have only one chance in 100 of winning all 20, correct?

Supercomplexificissimus: I suppose so.

Sophia: But if your θ'=0.8 came from a 50% chance of a true θ=1.0 and a 50% chance of a true θ=0.6, the odds of your winning all 20 would be one in two, not one in a hundred.

That is where the difference lies. It is in making repeated bets in similar situations that failing to recognize that you do not know the true θ drawn by nature but only have an (hopefully) unbiased estimate θ' turns around and bites you in the ----. Cf.: "Gaussian Copula", passim...

Sokrates: I am not satisfied. I hear--all the time--people talk about things like "non-ergodicity" and "Knightian uncertainty". I would not claim to know what they mean. But surely they are gesturing in a direction in which it matters whether a particular θ'=0.8 really means that the odds are 80%, or whether that θ'=0.8 is just the average of a large number of widely-different possible things that the true odds θ might be.

Sophia: I mentioned one way it matters--if the situation is not a one-off. It also matters a lot if you are betting not against nature but against another mind--if your estimate θ' is rock-solid, then you can gamble away; but if there is great uncertainty about θ then you need to be very aware of the possibility that perhaps they know more than you do, and you should on no account take the bet. Moreover, it matters a great deal for how you construct your estimate θ'. If uncertainty about what the true θ is is high, you should be prepared to radically revise your beliefs as new information comes in. And you should not make irreversible decisions if new information that might be important is on the way. and worrying about such things reminds you that there is always also the risk that you do not understand the situation, and that many things that look like sure things are not.

Sokrates: But?

Sophia: Nate Silver is perhaps the quantitative analyst least vulnerable to such "black swan" critiques. On election day he gave Obama's chances as 92%, while other similar-methodology forecasters like Princton's Sam Wang were well above 99%. Why? As best as I can tell, because Nate's model includes a 16% chance that the model was hopelessly and grossly wrong in some dimension.

Sokrates: So quantitative methods do have their limits after all?

---

UPDATE: And something that I should have linked to at the start: Cosma Shalizi (2009): On the Certainty of the Bayesian Fortune-Teller

---

• This File: http://www.bradford-delong.com/2014/08/elementary-philosophy-of-probability-and-the-war-on-nate-silver-the-honest-broker-for-the-week-of-august-2-2014.html
• Edit This: http://www.typepad.com/site/blogs/6a00e551f08003883400e551f080068834/post/6a00e551f08003883401a73df8d179970d/edit

4089 words