the real problem with Nate Silver's election prediction model

note: I've got an extended preamble here about the debate regarding modeling in general.  If you want to get straight to my particular issues with the FiveThirtyEight prediction model, scroll down to the bolded text.

Just before the Alberta provincial election in April I left for China, and a couple weeks ago I arrived back in Edmonton, in time for another election south of the border.

In the last week before Tuesday's vote, it seems that one of North America's most popular pastimes is poll watching.  This has created a significant, albeit temporary, demand for online one stop shops that aggregate the polls, and the most popular sites add value by assessing how each poll should be used to make a prediction. 

This means modeling.  You take the data, run it through the model, and out pops a prediction.

Most of these modelers are predicting a solid Obama victory (i.e. with several dozen electoral votes to spare).  Sam Wang's "Princeton Election Consortium", for example, is of the view that the "Bayesian Prediction" of Obama's re-election chance is 98.1%.  Now this particular site is partisan, so partisan, in fact,  that he's using his site to raise money for Democrat candidates.  The site that's getting the most attention, though is Nate Silver's FiveThirtyEight blog which is now hosted by the New York Times.  Silver is also an Obama supporter, but doesn't have the incumbent as quite such an overwhelming favourite, pegging Obama as a 3 to 1 favourite to win the popular vote and slightly more of a favourite to win the Electoral College (and thus the election).

After Josh Jordan wrote a piece titled "Nate Silver's Flawed Model" for the National Review (no link as NRO's servers are in NYC and are under water) the liberal economists Paul Krugman and Brad DeLong weighed in, with Krugman insisting that this challenge to Nate Silver's model be seen as part of a "War on Objectivity" assuring us that "if these people triumph, science — or any kind of scholarship — will become impossible."

I'll leave aside the irony of Krugman declaring "if it isn’t what the right wants to hear, the messenger is subjected to a smear campaign" while he smears the critic of Nate Silver's model without specifically taking issue with so much as a single word of the offending post, stating simply that the criticism amounts to a "disgraceful episode. And the fact that the National Review ran with this tells you all you need to know about the publication."

What stuns me is that these liberals riding in to defend the honour of Nate Silver and his modeling are the same crowd that insists Wall Street's wizards are the cause of America's ailments.  Now there are exceptions.  Dean Baker is another leftist economist and he's of the view that "We would be in pretty much the same place today even if the financial crisis had not happened."  That's a view I happen to agree with, after all, the fancy derivative products that overcompensated financial engineers came up with are just that, derivative, meaning they are dependent upon and derive their value from what's happening in the underlying "real" economy.  But that doesn't mean that these people didn't obscure the price signals in such a way as to aggravate the misallocation of the capital that was occurring in the real economy.  How did these hot shots get away with it?  Because people treated them like Krugman is demanding we treat Nate Silver, that's why!  Silver uses exponential smoothing with a decay factor and Monte Carlo simulation!  Do you plebes understand that stuff?  No?  Then sit down and show some respect!

This is, of course, why hedge fund managers are so grossly overpaid.  There is no clear need for a finance industry beyond various straightforward functions such as putting people with business ideas together with people with capital but we've got these great towers of High Finance because, well, it's fueled by flows of money that get directed at whoever appears to be the smartest, and that generally means whoever's got the most complicated sounding model.

The economic consequence of Krugman stepping up to praise Silver is that the market will conclude Silver deserves a million dollar salary.  Yet Krugman's "liberal conscience" will of course soon compel him to turn around and complain about income inequality!  Dean Baker has figured out how dumb this all is:

Progressives... have been losing not just because conservatives have so much more money and power, but also because they have accepted the conservatives’ framing of political debates. They have accepted a framing where conservatives want market outcomes whereas liberals want the government to intervene to bring about outcomes that they consider fair.
...
This "loser liberalism" is bad policy and horrible politics. Progressives would be better off fighting battles over the structure of markets so that they don't redistribute income upward.

Baker has noted that part of the reason U.S. doctors and lawyers make so much money is because they have monopolies.  More international competition (i.e. free trade in professional services) would lower prices for ordinary Americans, and the people hurt would be people who are doing very well already.  Trying to instead move the money around via the tax system buys into the argument that these service providers rightly own the fruits of their monopoly power.

I could go on here but this preamble is already quite lengthy so I'll turn to my particular objections to Silver's model.

I'm not going to take issue with Silver's prediction of who will win the electoral college except to note that a lot of people are wrongly interpreting the slide in Romney's probability of winning as Obama momentum.  This is in fact just a consequence of Silver's model having a time decay.  If the home team is down 3 to zero in the third inning, if they are still down three zip in the bottom of the ninth that doesn't mean the away team has scored any more runs, but it does mean that the behind team is running out of "at bats."  In Silver's line graph there is therefore a force that pushes in both directions away from the 50% line.  The fact Romney was able to move up closer to the centre line earlier in October was all the more impressive because it was pushing up against this time gravity.

What I will take issue with is Silver's prediction that the odds of Obama winning the electoral college and losing the national popular vote are just 5.1%.  A big driver of this is Silver's skepticism that Romney will win the popular vote, pegging as he does Romney's chances at that at 24.3%, little better than Romney's 21% chance of winning the electoral college.

How can they be less than 25% when Rasmussen's national poll has Romney ahead by two points and Gallup has Romney up by five?  There are other national polls, of course, but almost all of them have Romney either tied, ahead of Obama (albeit by less), or at worst behind just 1 point.

Silver's response would be that there is other data out there.  There are state polls and these can be used to supplement the national polls for a more accurate prediction of the national popular vote.  Now this is perfectly sound in theory but how exactly does he do this?  Silver went into detail on this back in July.  One of Silver's critical beliefs here is that "the relative order of the states [i.e. how they line up relative to the national trend in terms of "redness" or "blueness"] is extremely consistent from year to year."  Now right there you have the weakness of most models: the assumption that the future will continue to be like the past.  It is not irrelevant to note that a split between the popular vote and the electoral college verdict has been historically rare, but it should be tested against various narratives that might explain why this time it's different.  

The fact is that Silver makes some adjustments based on considerations unique to each cycle anyway, so the "extreme consistency" is limited even in his own eyes.  An example of this is his "adjustment for home state effects."  "On average, a presidential candidate gets a seven-point push in his home state," he observes, and he therefore has the model push Massachusetts somewhat redder in "the relative order of the states" because Romney's from there.  Now this is where you need to start asking yourself if Massachusetts is really going to go redder this year.  Sometimes familiarity breeds contempt, no?  Don't "identity politics" apply in other aspects as well?  One has to stop somewhere, of course, and that's my point: Silver stopped too early when it comes to considering what's different about 2012.

According to Silver's "FiveThirtyEight Presidential Voting Index - 2012" Pennsylvania is seven states bluer than Ohio in the state order.  If Obama is ahead in Ohio by 2, then he should be up in Pennsylvania by more than 8 points.  Wisconsin is even bluer; Obama should be ahead in Wisconsin by 9.5% if he's up in Ohio by 2%.  If Obama is just 2 points back in North Carolina, then Obama should have an 9.6% lead in Pennsylvania and have a full 11% lead in Wisconsin.  Thus spake the model.  But this doesn't square with what we are seeing on the ground.  David Axelrod may have promised to shave his mustache if Obama loses any of Minnesota, Michigan, or Pennsylvania, but that doesn't change the fact that the Obama campaign is committing resources to these states.  The liberal leaning Pittsburgh Post-Gazette says that Pennsylvania is being "flooded with television ads."  Now maybe Romney and his allies have decided to go after Pennsylvania because they've been banging their heads against the wall in Ohio for a long time with little progress and so now they're making the Hail Mary pass next door.  But that doesn't explain why the Obama campaign is spending money to counter this; - after all, didn't Nate Silver say that "the relative order of the states is extremely consistent from year to year," meaning that the "battleground" states don't change?  If we turn to Michigan, there are no less than 14 states between Michigan and North Carolina in Silver's rank index.  If we check against the current polling, we see that Realclearpolitics (RCP) has Romney just 3 points behind while Silver's index says that if Obama is 2 points behind in North Carolina  (which is how Silver currently rates the situation in NC) Obama should be a whopping 13.9% ahead in Michigan.  If we turn to Wisconsin, we see that instead of that 11% lead for Obama in this state that I noted above, RCP's polling average says Romney is just 4% back.  Despite these underestimates of Romney's support when using the "battleground states" of NC and OH to predict the not-so-battleground states of Pennsylvania, Michigan, and Wisconsin "we can use this index to calculate an implied national popular vote," Silver insists.  

Silver addressed some of these concerns Wednesday (October 31):  

Michigan is probably not as close as two or three points right now: most polls released after the first debate in Denver suggested a lead for Mr. Obama in the mid-to-high single digits. Usually, states do not shift all that much relative to others in their region. The fact that Mr. Obama’s polling has held up reasonably well in Ohio and Iowa, for example, is reason to suspect that some of the movement in the poll represents statistical noise, even if it comes from a good polling company.

Does anyone else see the circular reasoning here?  Michigan can't be that close because "states do not shift all that much relative to others in their region."  In other words, Michigan can't be that close because my model says it can't be that close!

Now it should be acknowledged here that Silver doesn't just use the "battleground" states to predict the national vote.  He does use the other states.  So wouldn't that balance things out?  It could, if weighted by expected 2012 turnout and we had good up-to-date polling of those other states.  But Silver doesn't do that.  Take Utah, for example.  Silver gives it a weight of 0.05, while Florida gets a weight of 1.58.  Florida thus gets more than 30 times the weight of Utah.  But Florida's population is less than 7 times Utah's.  Utah has only been polled twice since June, but both of these Utah polls were from mid-October and they've got Romney ahead by 51% and 53% for an average of 52%.  That's 14.7% ahead of where Silver's "index" says Romney should be, and that's relative to the United States (meaning Romney is even farther ahead in Utah of where he should be relative relative to NC or OH).  While Romney's Utah support gets underweighted (and this doesn't even go beyond the relative population difference to note that people who lean Republican turn out more frequently than people who lean Democrat), Obama's Florida support gets overweighted, where Romney is running 3.6% behind where Silver's index says Romney should be.  I could make the same point here using Michigan instead of Utah and North Carolina instead of Florida.

Silver calls this weighting a "wrinkle" that is not "that important."  He says that he cannot weight just by turnout because the "quantity and quality of polling" varies from state to state.  As Silver put it in July, "So — although technically it determines an implied national popular vote for South Carolina — the South Carolina value receives very little weight in the overall calculation since the only poll there was conducted months ago."  In response to this, allow me to quote from this article titled, "Why Scientific Studies Are So Often Wrong: The Streetlight Effect": 

The fundamental error here is summed up in an old joke scientists love to tell. Late at night, a police officer finds a drunk man crawling around on his hands and knees under a streetlight. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street. Then why are you looking over here? the befuddled officer asks. Because the light’s better here, explains the drunk man.

So in order to predict the national vote, Silver upweights "battleground" states like Ohio because the "light's better there."

Can anybody guess why Romney might be doing better in Utah than your typical Republican in the past?  I'm sure you can come up with a theory there.  Why might he being doing worse in Ohio?  Perhaps the "Not one of us" ads that Obama swamped the state with all summer in order to paint Romney as hostile to the working man brought Romney's support in blue collar Ohio lower without hurting Romney's support in the rest of the country (where the ads weren't aired) in the same proportion.

I'll make one final note here.  In his October 31 post Silver pointed at some other modelers and noted that they generally agreed with him.  As someone who works in finance I feel compelled to note that this is why so many supposedly brilliant hedge fund managers have their funds go bust.  They build these impressive appearing models that attract capital by virtue of their impressiveness without much consideration for the fact that the other hedgies are doing the same thing.  So when there is a market shock of some sort, these guys all end up trying to go in the same direction because their models are similar, and then they can't get out of their positions because there isn't anybody to take the other side of the trade!