Va. Governor Race Is Within the 'Margie Margin'
When people ask me “what went wrong with the polls in 2016,” I have three responses. The first, and most important, is that the polls weren’t unusually “wrong.” In fact, they were as accurate as they were in 2012, when polls were routinely celebrated as the best tool for predicting elections. Second, the polls were off in the particular places they were off because they systematically under-sampled white working-class voters in the upper Midwest.
The third point is related to the second: Pollsters have to make choices about what the electorate looks like, which can bias results in ways that poll averages simply can’t account for. What I mean is this: When you see a poll reported, it has an error margin, usually in the range of +/- 3 percentage points. The precise meaning of this error margin is a little bit difficult to explain, but you can usefully think of it as “we can be pretty confident that the actual value of a candidate’s vote share is within three points of what this poll is reporting.”
This error margin is dependent on the sample size: As sample sizes increase, error margins shrink. Polling aggregation effectively increases the sample size beyond that which any particular poll reports. So, we get a very small error margin. If the total number of observations in the aggregate were 5,000, the overall error margin would be +/- 1.2 percentage points, which suggests they should get the overwhelming number of races separated by more than two points correct (the error margin of a spread is 1.6 times the error margin, not twice the error margin). At least, that’s how it would work if pollsters were all taking perfect samples utilizing identical methods.
But this isn’t how polling works in the real world. Because we don’t know what the actual electorate will look like in advance, and because pollsters have difficulty getting a balanced sample (given caller identification and restrictions on calling cellphones), we encounter sources of error that go beyond the error that is inherent in random draws. People misunderstand surveys, pollsters weight their samples incorrectly, or pollsters report on different populations (although almost everyone samples likely voters at the end of a cycle).
All of this adds up to what I took to referring to as the “Omero-Gelman” margin during the last election cycle. This refers to a fascinating experiment performed by the New York Times’ Upshot, where the paper provided the same raw polling data (consisting of the poll respondents’ unweighted vote choices, along with various demographics) from Florida to four different pollsters. In return, it got different results, ranging from a four-point Clinton lead to a one-point Trump lead. Those end points were returned, respectively, by pollster Margie Omero and statistician Andrew Gelman; at Omero’s joking suggestion, I shortened this to the “Margie margin.” In any event, this is entirely independent of the error you see reported in error margins. Worse, you can’t minimize this error through aggregation. To borrow statistical terms, there’s no Law of Large Numbers for opinions; while I suspect these errors average out to zero over time, there’s no formal proof for this. The good news is that pollsters are pretty good at what they do, and there are generally accepted principles about weighting that keep results from going too far afield. The bad news is that this margin is nonetheless real.
Which brings us (finally) to Virginia. There seems to be little doubt that this race is narrowing in the final days, although there is disagreement among pollsters as to just how much. The RealClearPolitics Average currently has Ralph Northam up by 2.8 points over Ed Gillespie. But that’s where the Margie margin comes in. Some of this variability we see isn’t due to sampling effects; it is due to pollster assumptions about what the electorate looks like. Put differently, Quinnipiac doesn’t consistently find double-digit Northam leads due to random chance; it is because of how the pollster is viewing the electorate. We can’t really put odds on Quinnipiac being right in the way we can calculate probabilities for sampling error. It’ll either be right, or it won’t.
Which is a roundabout way of saying: Northam has a lead. Without any clear knowledge about what the electorate will look like on Tuesday, that’s still probably your best bet. But just keep in mind that this is well within the Margie margin, and pollster choices are probably the difference between the expressed result, a narrow Gillespie win, or a surprisingly comfortable Northam win.