About this Blog
Farewell!
Email Me

RealClearPolitics HorseRaceBlog

By Jay Cost

« Michael Steele Makes the Case for Party Reform | HorseRaceBlog Home Page | A Primer on the 2010 House Midterm »

Gallup's Bouncing Ball

This week, Gallup's "generic ballot" number - which asks people if they plan to support a generic Republican or Democrat in the upcoming congressional elections - found a big boost for the Democrats, who bounced out to a 6-point lead.

Dems Lead Generic by 6.gif

The jump was sufficient to merit comment from Allahpundit as well as other conservatives. And I'd like to toss my two cents in.

Polls tend to have house effects, and most of us tend to notice these effects when they result in a pollster falling on one side of an average or the other. Rasmussen, for instance, tends to have Obama's job approval on the low side while ABC News/Washington Post usually puts the President on the high side. But there are other types of house effects, and Gallup has an interesting one: it is kinda bouncy.

For instance, look again at that above graph. Over the course of the last two months, Gallup has shown as much as R+6 on the generic ballot and as much as D+6. In a small poll, that might be attributable to sampling error, but the Gallup sample is over 1,500 registered voters, which produces a margin of error of less than 3 points.

Is it really the case that there has been a 12 point swing in the last two months? I doubt it. Looking at the RCP generic ballot average, the only two pollsters who have done multiple generic ballot questions during this time are Fox News and Rasmussen, and neither found such substantial swings.

This is not the only example of Gallup's bounciness. If you are anything like me, you wind your way over to Gallup.com at 1 PM Eastern Standard Time, as that is when it releases it's latest numbers on the President's job approvals. Those numbers move around more than any others out there. On Sunday, for instance, the President's job approval number rose 3 points, and his job disapproval fell 3 points. That made for quite a swing, which is noteworthy because the poll is based on 1,500 total respondents over a three-day track. That means about 500 respondents every day. For his net approval to fall 6 points by cycling out one day and adding another must have meant a very substantial one day movement. If you watch Gallup every day like I do - you'll note that such swings are fairly common, much more so than Rasmussen, the other daily tracking poll.

This is not a recent phenomena with Gallup, either. Check out its trial heats for the later stages of the 2000 presidential election campaign. Note October 2000 in particular:

2000 Bush v Gore.gif

Holy cow! At the beginning of the month, Gore had a 12 point lead, but at the end of the month it was Bush who had an equally large lead. That's some bounciness. And I would note that ABC News poll had a much more stable track through the month of October, and none of the other pollsters showed such outsized movement.

As for the generic ballot itself, it is historically a difficult metric. For instance, in 1998 the Republicans won a solid if uninspiring 2-point victory in the House popular vote, but Gallup's November generic ballot showed the Democrats up 7. In 2002, the October generic ballot had the Democrats up by 6 points, but the GOP went on to win the House popular vote by nearly 5 points. In 2006, Gallup's post-Labor Day generic ballot was tied among likely voters, but a month later the Democrats were out to a 13-point lead. Its post-Labor Day poll found something similar in 2008, leading Gallup to declare that "The Battle for Congress Suddenly Looks Competitive." It wasn't. In both 2006 and 2008, Republicans had a rough month of September, and in both cases the party's trouble really began after the Gallup polls. Still, in historical retrospect, I do not think that the "actual" generic ballot numbers were ever as tight as Gallup found, or as far apart as it found just a few weeks later.

Now, don't get me wrong. The problems with the generic ballot question are due to the limitations of the question itself, not how Gallup handles it. Democrats have a generic edge in nationwide party identification, which often does not materialize on Election Day, so historically the generic ballot favors the Democrats. There's nothing that Gallup can do about that. Plus, there is nothing wrong with having a house effect. I'd be suspicious of a poll that had none. Gallup is a great, reliable pollster that has been doing good work since before most of us were born. It would not be in business today if it wasn't. And kudos to Gallup for offering up more and more of its historical data, which is not only interesting but also a real public service.

My point here is fairly modest: it's incumbent upon us, the consumers of polling data, to digest it properly - which means generally that we have to be aware of house effects, and specifically in the case of Gallup we should not get hung up on every little inflection point. Gallup bounces around quite a bit, and it has exhibited this quality for some time. The best approach in handling this is to average its results across a few weeks.

When we do that for the Gallup 2000 numbers on Bush v. Gore, we find Bush holding a modest 4 point lead, 47 to 43, which should make intuitive sense given the dynamics of that race in its final stages. Similarly, the average generic ballot since Memorial Day 2010 shows the two parties essentially tied, which sounds about right to me for a measure of registered voters several months before Election Day.

-Jay Cost