the real problem with Nate Silver's election prediction model

note: I've got an extended preamble here about the debate regarding modeling in general.  If you want to get straight to my particular issues with the FiveThirtyEight prediction model, scroll down to the bolded text.

Just before the Alberta provincial election in April I left for China, and a couple weeks ago I arrived back in Edmonton, in time for another election south of the border.

In the last week before Tuesday's vote, it seems that one of North America's most popular pastimes is poll watching.  This has created a significant, albeit temporary, demand for online one stop shops that aggregate the polls, and the most popular sites add value by assessing how each poll should be used to make a prediction. 

This means modeling.  You take the data, run it through the model, and out pops a prediction.

Most of these modelers are predicting a solid Obama victory (i.e. with several dozen electoral votes to spare).  Sam Wang's "Princeton Election Consortium", for example, is of the view that the "Bayesian Prediction" of Obama's re-election chance is 98.1%.  Now this particular site is partisan, so partisan, in fact,  that he's using his site to raise money for Democrat candidates.  The site that's getting the most attention, though is Nate Silver's FiveThirtyEight blog which is now hosted by the New York Times.  Silver is also an Obama supporter, but doesn't have the incumbent as quite such an overwhelming favourite, pegging Obama as a 3 to 1 favourite to win the popular vote and slightly more of a favourite to win the Electoral College (and thus the election).

After Josh Jordan wrote a piece titled "Nate Silver's Flawed Model" for the National Review (no link as NRO's servers are in NYC and are under water) the liberal economists Paul Krugman and Brad DeLong weighed in, with Krugman insisting that this challenge to Nate Silver's model be seen as part of a "War on Objectivity" assuring us that "if these people triumph, science — or any kind of scholarship — will become impossible."

I'll leave aside the irony of Krugman declaring "if it isn’t what the right wants to hear, the messenger is subjected to a smear campaign" while he smears the critic of Nate Silver's model without specifically taking issue with so much as a single word of the offending post, stating simply that the criticism amounts to a "disgraceful episode. And the fact that the National Review ran with this tells you all you need to know about the publication."

What stuns me is that these liberals riding in to defend the honour of Nate Silver and his modeling are the same crowd that insists Wall Street's wizards are the cause of America's ailments.  Now there are exceptions.  Dean Baker is another leftist economist and he's of the view that "We would be in pretty much the same place today even if the financial crisis had not happened."  That's a view I happen to agree with, after all, the fancy derivative products that overcompensated financial engineers came up with are just that, derivative, meaning they are dependent upon and derive their value from what's happening in the underlying "real" economy.  But that doesn't mean that these people didn't obscure the price signals in such a way as to aggravate the misallocation of the capital that was occurring in the real economy.  How did these hot shots get away with it?  Because people treated them like Krugman is demanding we treat Nate Silver, that's why!  Silver uses exponential smoothing with a decay factor and Monte Carlo simulation!  Do you plebes understand that stuff?  No?  Then sit down and show some respect!

This is, of course, why hedge fund managers are so grossly overpaid.  There is no clear need for a finance industry beyond various straightforward functions such as putting people with business ideas together with people with capital but we've got these great towers of High Finance because, well, it's fueled by flows of money that get directed at whoever appears to be the smartest, and that generally means whoever's got the most complicated sounding model.

The economic consequence of Krugman stepping up to praise Silver is that the market will conclude Silver deserves a million dollar salary.  Yet Krugman's "liberal conscience" will of course soon compel him to turn around and complain about income inequality!  Dean Baker has figured out how dumb this all is:

Progressives... have been losing not just because conservatives have so much more money and power, but also because they have accepted the conservatives’ framing of political debates. They have accepted a framing where conservatives want market outcomes whereas liberals want the government to intervene to bring about outcomes that they consider fair.
...
This "loser liberalism" is bad policy and horrible politics. Progressives would be better off fighting battles over the structure of markets so that they don't redistribute income upward.

Baker has noted that part of the reason U.S. doctors and lawyers make so much money is because they have monopolies.  More international competition (i.e. free trade in professional services) would lower prices for ordinary Americans, and the people hurt would be people who are doing very well already.  Trying to instead move the money around via the tax system buys into the argument that these service providers rightly own the fruits of their monopoly power.

I could go on here but this preamble is already quite lengthy so I'll turn to my particular objections to Silver's model.

I'm not going to take issue with Silver's prediction of who will win the electoral college except to note that a lot of people are wrongly interpreting the slide in Romney's probability of winning as Obama momentum.  This is in fact just a consequence of Silver's model having a time decay.  If the home team is down 3 to zero in the third inning, if they are still down three zip in the bottom of the ninth that doesn't mean the away team has scored any more runs, but it does mean that the behind team is running out of "at bats."  In Silver's line graph there is therefore a force that pushes in both directions away from the 50% line.  The fact Romney was able to move up closer to the centre line earlier in October was all the more impressive because it was pushing up against this time gravity.

What I will take issue with is Silver's prediction that the odds of Obama winning the electoral college and losing the national popular vote are just 5.1%.  A big driver of this is Silver's skepticism that Romney will win the popular vote, pegging as he does Romney's chances at that at 24.3%, little better than Romney's 21% chance of winning the electoral college.

How can they be less than 25% when Rasmussen's national poll has Romney ahead by two points and Gallup has Romney up by five?  There are other national polls, of course, but almost all of them have Romney either tied, ahead of Obama (albeit by less), or at worst behind just 1 point.

Silver's response would be that there is other data out there.  There are state polls and these can be used to supplement the national polls for a more accurate prediction of the national popular vote.  Now this is perfectly sound in theory but how exactly does he do this?  Silver went into detail on this back in July.  One of Silver's critical beliefs here is that "the relative order of the states [i.e. how they line up relative to the national trend in terms of "redness" or "blueness"] is extremely consistent from year to year."  Now right there you have the weakness of most models: the assumption that the future will continue to be like the past.  It is not irrelevant to note that a split between the popular vote and the electoral college verdict has been historically rare, but it should be tested against various narratives that might explain why this time it's different.  

The fact is that Silver makes some adjustments based on considerations unique to each cycle anyway, so the "extreme consistency" is limited even in his own eyes.  An example of this is his "adjustment for home state effects."  "On average, a presidential candidate gets a seven-point push in his home state," he observes, and he therefore has the model push Massachusetts somewhat redder in "the relative order of the states" because Romney's from there.  Now this is where you need to start asking yourself if Massachusetts is really going to go redder this year.  Sometimes familiarity breeds contempt, no?  Don't "identity politics" apply in other aspects as well?  One has to stop somewhere, of course, and that's my point: Silver stopped too early when it comes to considering what's different about 2012.

According to Silver's "FiveThirtyEight Presidential Voting Index - 2012" Pennsylvania is seven states bluer than Ohio in the state order.  If Obama is ahead in Ohio by 2, then he should be up in Pennsylvania by more than 8 points.  Wisconsin is even bluer; Obama should be ahead in Wisconsin by 9.5% if he's up in Ohio by 2%.  If Obama is just 2 points back in North Carolina, then Obama should have an 9.6% lead in Pennsylvania and have a full 11% lead in Wisconsin.  Thus spake the model.  But this doesn't square with what we are seeing on the ground.  David Axelrod may have promised to shave his mustache if Obama loses any of Minnesota, Michigan, or Pennsylvania, but that doesn't change the fact that the Obama campaign is committing resources to these states.  The liberal leaning Pittsburgh Post-Gazette says that Pennsylvania is being "flooded with television ads."  Now maybe Romney and his allies have decided to go after Pennsylvania because they've been banging their heads against the wall in Ohio for a long time with little progress and so now they're making the Hail Mary pass next door.  But that doesn't explain why the Obama campaign is spending money to counter this; - after all, didn't Nate Silver say that "the relative order of the states is extremely consistent from year to year," meaning that the "battleground" states don't change?  If we turn to Michigan, there are no less than 14 states between Michigan and North Carolina in Silver's rank index.  If we check against the current polling, we see that Realclearpolitics (RCP) has Romney just 3 points behind while Silver's index says that if Obama is 2 points behind in North Carolina  (which is how Silver currently rates the situation in NC) Obama should be a whopping 13.9% ahead in Michigan.  If we turn to Wisconsin, we see that instead of that 11% lead for Obama in this state that I noted above, RCP's polling average says Romney is just 4% back.  Despite these underestimates of Romney's support when using the "battleground states" of NC and OH to predict the not-so-battleground states of Pennsylvania, Michigan, and Wisconsin "we can use this index to calculate an implied national popular vote," Silver insists.  

Silver addressed some of these concerns Wednesday (October 31):  

Michigan is probably not as close as two or three points right now: most polls released after the first debate in Denver suggested a lead for Mr. Obama in the mid-to-high single digits. Usually, states do not shift all that much relative to others in their region. The fact that Mr. Obama’s polling has held up reasonably well in Ohio and Iowa, for example, is reason to suspect that some of the movement in the poll represents statistical noise, even if it comes from a good polling company.

Does anyone else see the circular reasoning here?  Michigan can't be that close because "states do not shift all that much relative to others in their region."  In other words, Michigan can't be that close because my model says it can't be that close!

Now it should be acknowledged here that Silver doesn't just use the "battleground" states to predict the national vote.  He does use the other states.  So wouldn't that balance things out?  It could, if weighted by expected 2012 turnout and we had good up-to-date polling of those other states.  But Silver doesn't do that.  Take Utah, for example.  Silver gives it a weight of 0.05, while Florida gets a weight of 1.58.  Florida thus gets more than 30 times the weight of Utah.  But Florida's population is less than 7 times Utah's.  Utah has only been polled twice since June, but both of these Utah polls were from mid-October and they've got Romney ahead by 51% and 53% for an average of 52%.  That's 14.7% ahead of where Silver's "index" says Romney should be, and that's relative to the United States (meaning Romney is even farther ahead in Utah of where he should be relative relative to NC or OH).  While Romney's Utah support gets underweighted (and this doesn't even go beyond the relative population difference to note that people who lean Republican turn out more frequently than people who lean Democrat), Obama's Florida support gets overweighted, where Romney is running 3.6% behind where Silver's index says Romney should be.  I could make the same point here using Michigan instead of Utah and North Carolina instead of Florida.

Silver calls this weighting a "wrinkle" that is not "that important."  He says that he cannot weight just by turnout because the "quantity and quality of polling" varies from state to state.  As Silver put it in July, "So — although technically it determines an implied national popular vote for South Carolina — the South Carolina value receives very little weight in the overall calculation since the only poll there was conducted months ago."  In response to this, allow me to quote from this article titled, "Why Scientific Studies Are So Often Wrong: The Streetlight Effect": 

The fundamental error here is summed up in an old joke scientists love to tell. Late at night, a police officer finds a drunk man crawling around on his hands and knees under a streetlight. The drunk man tells the officer he’s looking for his wallet. When the officer asks if he’s sure this is where he dropped the wallet, the man replies that he thinks he more likely dropped it across the street. Then why are you looking over here? the befuddled officer asks. Because the light’s better here, explains the drunk man.

So in order to predict the national vote, Silver upweights "battleground" states like Ohio because the "light's better there."

Can anybody guess why Romney might be doing better in Utah than your typical Republican in the past?  I'm sure you can come up with a theory there.  Why might he being doing worse in Ohio?  Perhaps the "Not one of us" ads that Obama swamped the state with all summer in order to paint Romney as hostile to the working man brought Romney's support in blue collar Ohio lower without hurting Romney's support in the rest of the country (where the ads weren't aired) in the same proportion.

I'll make one final note here.  In his October 31 post Silver pointed at some other modelers and noted that they generally agreed with him.  As someone who works in finance I feel compelled to note that this is why so many supposedly brilliant hedge fund managers have their funds go bust.  They build these impressive appearing models that attract capital by virtue of their impressiveness without much consideration for the fact that the other hedgies are doing the same thing.  So when there is a market shock of some sort, these guys all end up trying to go in the same direction because their models are similar, and then they can't get out of their positions because there isn't anybody to take the other side of the trade!

new poll: in Edmonton Wildrose 36%, Lib 26%, PC 25%, NDP 12%

Angus Reid has released a poll of 1000 Albertans, the same size as the Environics poll released on November 5. A possible issue with this new poll is that it was a survey of Angus Reid Forum participants, and there may be some self-selection involved there. But leaving that aside, comparing these two polls is somewhat problematic, because while the pollster enumerated 5 options in both cases, Environics listed the Greens as the fifth option and Angus Reid (more correctly in my view since no Green party is currently recognized by Elections Alberta) listed the fifth option as "Other." Nonetheless, it appears that support for the PCs continued to slump across the province from late October to late November, with much of the drop in PC support in Calgary going to the Liberals and with the drop in Edmonton going pretty much entirely to Wildrose. The appearance of PC support bleeding to the Liberals in Calgary nonetheless strikes me as possibly just appearance, as the more likely explanation is that the Environics poll understated Liberal support in Calgary. Fact is, the Liberals took a greater share of the vote in Calgary in 2008 than within Edmonton city limits (33.9% vs 33.5%). Given that the Alberta Liberals have since proceeded to replace an Edmonton leader with a Calgary leader, the idea that the Liberals are at 30% in Calgary and 26% in Edmonton, as Angus Reid reckons it, strikes me as a lot more plausible then the Environics verdict that just 20% of Calgarians are inclined to vote for the red team.

It is also worth noting that Angus Reid confirms Environics' finding that NDP support in the capital city is down from the March 2008 election. Where have the 6 or 7 points the NDP appear to have lost from their 18% election share gone? Presumably to David Swann's Liberals, yet Edmonton Liberal support is also down 6 or 7 points relative to 2008 which means there may be a lot of Edmontonians who voted for Kevin Taft's team who are nonplussed with David Swann's group. That's something that makes a lot of sense to me, since while Kevin Taft and his talk about excessive PC spending led me to view him rather positively, I don't identify with David Swann at all.

The crosstabs are often the most interesting part of a poll. The province-wide breakdown for those in the $100K+ household income bracket is Wildrose 46%, PC 30%, and Liberal 16%. Given that the average household income in Edmonton Whitemud is $132K (and that is back in 2005), this may explain why our organizing in Whitemud has been so successful. I mean, look at this, Angus Reid has the Liberals in 2nd place after Wildrose in all demographics in Edmonton and for the six figure crowd (albeit province-wide for this tab) it is a 30 point gap, 46 to 16. The Edmonton Whitemud nomination for Wildrose is going to be hotly contested!

Besides Wildrose doing especially well in the rural south while the Alberta Liberals are quite weak in the rural north, the other crosstab of note is that Wildrose support skews noticeably older, with the Liberals skewing the other way (the PCs are inbetween on this tab, as on many). This is consistent with my experience on the ground and I have to admit I have been wearing out my welcome with a few older Wildrosers as I try to shove young, university educated people (uni grads being another demographic where Wildrose is weak and the Libs strong, with the Liberals running 8 points ahead across the province in this group according to this poll) to the front of the line. The mentality of the people on constituency association boards, on the provincial executive, and surrounding the leader is going to set the tone for the party as a whole, and accordingly I think the party needs to actively work to avoid adopting stances that preach to the choir. Environmentalism is a lot more popular with people of Danielle Smith's age and younger than those older, and I hope that older Wildrosers understand that there may be more to gain by expanding our foothold with 20 and 30-somethings than by messaging to older Albertans whom Wildrose has already won over by an absolute majority according to Angus Reid.

While I appreciate the idea of running up the score on favourable ground, my experience of politics has been that returns on investment decline after a certain point such that, for example, trying to shake the third round of votes off a tree is tough going because we are talking about fruit here that doesn't give up on the old branch easily. Find another tree and give it a first time shake. An effective shake means understanding what the tree is about and what makes it grow. I believe older Wildrosers are more than willing to cooperate so long as younger ones explain themselves (and the strategic value of bringing in their colleagues to positions whereby they could influence the party's stance) instead of just getting pushy and demanding with the old guard. In any case, whether new to the party or old, I would think most Wildrose Alliance members understand that the party is in the position it presently is because of one reason above all others.

Calgary polls

While one would think that this past week's news of a record provincial deficit would revive questions about the propriety of the pay raises the Alberta cabinet gave itself last year, the response we've seen instead from some quarters is a call to diversify the provincial economy. This suggestion avoids the fact that it is, in fact, a lack of diversity in government revenue sources that, combined with an inability or unwillingness to prudently manage revenue when it is abundant, is driving the deficit. A government program to diversify Alberta's economy into, say, banana growing would not be sound policy; if we have a comparative advantage in a particular area, basic economics argues that we should continue to exploit it. Instead of drawing revenue from a diversified source and taxing the economy at the broadest, least distortionary level (with, say, a provincial VAT), the Stelmach government has done the opposite and tried to increase royalties, increasing dependence on the most cyclical revenue source of all. The province has additionally kept general business taxes, which are also very cyclical, relatively high. This when business is ahead of both government and households in terms of propensity to invest.

A Macleans story says that according to ENMAX chairman Thompson MacDonald, the Wildrose Alliance is "too far to the right to appeal to disgruntled Tories." While this may indeed be the perception, the reality is that Alberta has the most fiscally liberal government in North America. While there are others who sport bigger deficits, I should hope that by now it's become obvious that whether or not a government is in deficit is a poor metric of its fiscal conservatism. The proper standard is its level of spending. Another jurisdiction with a higher income tax rate is more conservative if that jurisdiction does not enjoy windfall royalty revenue such that its spending per person is lower. What matters are the decisions that are taken in the context of scarcity. Are calls to get provincial finances in order truly "too far to the right" for Alberta? With respect to this question, I might also refer to Link Byfield's observations:

On actual policy there is almost no difference amongst all three leadership candidates -- nor can there be, because like the Reform Party of yore, policy is controlled by members, not the leader. But policy alone will not get the Wildrose Alliance elected. We will succeed (or not) on the credibility of our leader and on the caliber of candidates she can recruit. That's why I and most of the social conservatives I know are backing Danielle Smith.

In other words, how can Thompson MacDonald know that "the Wild rose Alliance isn’t going to change [the] equation" in the future?

The Hinman campaign has commissioned Dycap Research to poll Calgary Glenmore and has found that the Wildrose Alliance is running at 19% in the riding. Given that Ipsos announced on August 26 that the PCs were at 43% Calgary wide, with the Liberals at 23%, the Wildrose Alliance at 11%, and New Democrats at 5%, one might be inclined to think that 19% in Glenmore is too high an estimate. After all, the party drew 8% in Glenmore in the 2008 provincial election and the Alberta Alliance pulled in less than 5% in 2004. But 15% is within the realm of reason and 19% wouldn't be very far off the margin of error. The same Dycap poll revealed a drop of 12% in PC support since June, meaning that the recorded 7% gain for the Wildrose Alliance over the same time period is entirely plausible. Wildroser Danielle Smith has received a lot of media attention these past two months, and, as Lorne Gunther noted in the Edmonton Journal on August 16, "Smith has a deeper, subtler and quicker grasp of policy alternatives than has been exhibited by the entire Tory cabinet, combined." The PCs will take Calgary Glenmore again (they reeled in more than 50% of the vote last year), but if they do so this time with less than 35% and the Wildrose Alliance gets 15% or more (tripling the Alberta Alliance share of 5 years ago and then some), it will suggest that the WAP will be a real player in the next province wide campaign. 15% is within striking distance of 20%, and 35% support has been known to slump to less than 30%.

the power of perception

Nate Silver cites an interesting Gallup poll re perceptions of Fed chief Ben Bernanke.

What is interesting is that a very significant number of people seem to have changed their views of Bernanke from 2008 to 2009. What´s different between 2008 and 2009 for those who self-identify as Democrats or Republicans? The White House. Ergo, now "we" are in charge, for Dems, and "they" are in charge, for GOP voters. If there has been a substantial change of policy views on the part of Bernanke, I have not heard of it.

This holds an important lesson for why the Alberta Tories are so hard to unseat in Alberta. For a serious threat to materialize, many Albertans would have to conclude that "they" are in charge, and "we" are not. I don't see that happening soon... Stelmach will often take a rhetorical jab at "Ottawa" but never the "Harper Conservatives". Never mind that these two terms refer to the same thing: federal policy. "Ottawa" is THEM whereas the "Harper Conservatives" (to many Albertans) is US. Stelmach won't even go so far as to object to equalization, because many Albertans see equalization as a "Canadian" program and "Canadian" is US.

Over the last decade, there has nary been a union demand or a spending request that the Alberta P"C" has not accomodated with no less enthusiasm that the most explicitly "socialist" of social democratic parties. Yet this has not stopped Stelmach from warning of "socialism" and condemning it. After all, "socialists" are THEM.