Related Topics
election 2006
polls
Polls

President Bush Job Approval

RCP Average
Approve:36.8%
Disapprove:58.0%
Spread:21.2%
Send to a Friend | Print Article


Don't Bet on the Generic Ballot

By Jay Cost

Thomas Riehle and Lance Tarrance offered a thought-provoking piece last week discussing the generic congressional ballot, i.e. the question in media polls that asks which party respondents will prefer in the congressional election.

They write:

The new Cook Political Report/RT Strategies national survey...shows Democrats leading Republicans by 12 points among registered voters...the Democratic advantage in these generic ballot tests is into double digits, beyond the reach of the normal overstatement of Democratic prospects that critics correctly cite as an historic fact when looking at either the partisan Congressional control question or the generic ballot test.
It is true that critics (such as myself) cite the question's consistent Democratic skew. What Riehle and Tarrance do not mention is that, to use the poll as a predictive tool, one cannot simply correct this bias. Forecasting elections is not like golf -- where two players of different skill can compete once they have assigned handicaps.

The predictive value of a poll is based upon how it changes with the result it endeavors to predict. A poll that, on average, skews 7% in favor of the Democrats would still be valuable if it changes as the balance of power changes. In this instance, the skew would be like a thumb on an otherwise accurate scale. The final number is always wrong, but changes in the scale's estimate correspond with changes in the weight put on the scale. Riehle and Tarrance imply that this is the case - that we can correct for the Democratic thumb on the generic scale, and still read a distinct Democratic weight. Unfortunately, the generic ballot is more like a scale that is just plain broke.

At least it is this far from Election Day. Consider the history of Gallup's spring generic question (registration required), particularly the extent to which changes in it have preceded changes in the partisan composition of the House. This is the coefficient of determination, a value that tells us the percent of change in seats that can be anticipated by a change in Gallup's spring generic ballot. To return to the scale metaphor, a good scale with a skew would still have a high coefficient of determination - an increase in the weight put upon it would correspond almost perfectly to an increase in what the scale reports. A broken scale - one that offered results unrelated to the weight placed upon it, and whose average was still skewed in one direction - would have a low coefficient of determination.

For midterm congressional elections since 1958, the coefficient of determination of the spring question for the Democratic share of the House is about .335. In other words, changes in the spring generic ballot anticipate only about 33.5% of the change in the balance of power.

Why is this statistic such a poor indicator? The reason is that increases in the Democratic advantage barely correlate with increases in the Democratic share of seats. For instance, Gallup's spring 1958 poll gave the Democrats an 8% advantage; they would eventually enjoy a 153-seat majority in the next Congress. The spring 1966 poll gave them a 16% advantage; they would eventually enjoy a 60-seat majority. The Democrats doubled their lead in the poll, but lost nearly 100 seats. Another good example: in both 1974 and 1978, the Democrats had a lead of 26% this far from Election Day - in 1974 they won a 147-seat majority; in 1978 they won a 119-seat majority.

This low coefficient of determination is problematic in itself. You would not want to rely on the spring generic ballot too much in making your predictions! However, there are two additional problems, more theoretical in nature, that make me all the more wary. First is its inconsistent predictive power. If we consider a subset of the time-span in question - specifically 1958 to 1988, inclusive - the coefficient of determination drops to a paltry 2.25%. This is a sign that the statistic has more problems than we initially thought. When we find that an indicator performs unevenly, and we have no causal explanation for this unevenness, we should be skeptical of making use of it. This is a sign that chance, not causality, might be causing the relationship that we have discovered - and, if that is the case, there is no reason to believe that the spring generic ballot will be useful for future predictions.

The second concern is the high expectation we are inclined to place on the generic ballot. The spring ballot performs about the same when it is used to predict each party's share of the House vote (rather than share of House seats). This result is very unimpressive in light of the fact that the generic question claims to be a direct test of what will eventually occur on Election Day. The generic ballot does not claim to be one of many causal factors in the congressional elections (as we might take presidential job approval or candidate spending) - it claims to be a measure of the election results themselves. The fact that it does such a poor job predicting what we expect it to predict so well is another reason for concern. It is not just that it explains such a low amount of variation - it is that it explains an amount of variation that is so much lower than expected. We should be theoretically surprised by this result, and therefore skeptical of using the measure as a predictive tool, even to the limited extent the existing data allows.

In the mid-90s, several political scientists attempted to make use of the spring statistic by arguing that, when taken in the context of who is in control of the White House, its predictive value increases. However, these attempts were ultimately thwarted by the history defying 1998 and 2002 contests - and they were theoretically and statistically problematic to begin with. All in all, my sense is that the spring generic ballot adds more confusion than clarity to understanding this midterm. It is best left aside.

This might seem counter-intuitive. Should this not be an effective early indicator of what will happen? I do not think so. This is a candidate-centered age of politics. For the political elite, the party comes first; partisanship acts as a frame of reference and a cognitive screen, enabling them to understand the political world and evaluate new data. However, average voters do not think in the same manner - not because they are less intelligent, but because they possess too little political information. Partisanship is still very important to the average voter, but it is not like a religious or regional affiliation. Average voters can and do defect to the other party, and they almost always defect to the party of their incumbent. This is a consequence of the new political age in which we live. This is not the 1880s; parties do not exercise decisive control over who wins the nomination, what issues they promote, what they do in government, etc. Today, candidates are free of their party. Is it a surprise that the psychological power of partisanship has declined? Is it a surprise that voters often eschew their party identification to pull the lever for their incumbents, who spend millions to remind them of their independence?

If what will be decisive this fall is evaluations of candidates, not abstract expressions of partisanship, what good is today's generic ballot? The average voter has not yet put a name to the label "Republican" or "Democrat." When s/he thinks about 2006, s/he is not yet thinking about the candidates. When we enter the fall, it will be time to think more carefully about the generic congressional ballot - as that will be the time the average voter puts names to the labels.

The value pollsters inappropriately assign to today's generic ballot indicates why caution is prudent when approaching media polls. The mistake with the generic ballot is not exceptional. Too often, pollsters and pundits misinterpret the significance of polling results.

Consider again the spring generic ballot. It endeavors to predict the partisan division of election returns: if the Democrats poll 51% on the generic, they will win 51% of the vote. Behind this endeavor, however, lies the aforementioned false premise: the average voter, like the political elite, is actively thinking about congressional elections today along partisan lines. This is not true, and so the poll does not mean what its interpreters think it means. Similarly, consider congressional job approval. Its predictive value is based upon the premise that the average voter thinks about his representative in the context of Congress as an institution. Almost always, this is not true.

There are many interpretations of many questions that can be criticized along the same lines. Everyday and everywhere, one sees columns and articles that tout the electoral importance of this or that question, without any inquiry into whether the claim is justified. The fact that we know so many claims to be incorrect is why we should not give these polls the benefit of the doubt. I am not suggesting that we doubt the results. I am suggesting that we should doubt the electoral significance of the results. It is an issue of value - i.e. pollsters and pundits are too quick to assign a value to their results that is unwarranted.

We should ask media polls to meet a burden of proof prior to our use of them. The burden is a positive answer to the following query: does the poll have a history of predictive accuracy? If a question has electoral relevance, over time the poll will be a good predictor of electoral outcomes. Thus, we must surmise its historical accuracy before we use it as an analytical tool. If there is no history, or if the history is too short, the question must seem obviously well tailored to elicit the desired information from the average voter. In other words, it cannot be a question that presumes that average voters think in the ways that political elites think; it must, rather, be thoroughly grounded in what we know about how average voters think during the election season and how they act at the ballot box.

This is why I take presidential job approval seriously. It corresponds well to changes in the balance of power. This is why I do not take congressional job approval seriously. It corresponds poorly to changes in the balance of power. This is also why I ignore questions with a short history, unless they seem to obviously cohere with what I know about the electorate's knowledge base and thinking processes. So, for instance, the "whom would you rather have in control of Congress" question is one that I ignore. It is all well and good that the public responds however it responds; but I know that, when it comes time to pull the lever, the average voter is not thinking about Congress as an institution. So, of what value is this question? Absent a history of accurate predictions, it is only prudent to assume that it is of little value.

Ultimately, the spring generic congressional ballot fails to meet this burden. Despite whatever intuitive appeal it seems to have, the fact is that it poorly correlates with electoral outcomes; further, a closer inspection of 21st century American politics indicates that this should not be a surprise. When we get to Labor Day, it will be time to start taking this poll seriously. However, as long as we can still wear white - we can safely ignore it!

© 2000-2006 RealClearPolitics.com All Rights Reserved


Email Friend | Print | RSS | Add to Del.icio.us | Add to Digg
Sponsored Links

Jay Cost
Author Archive