About this Blog
About The Author
Email Me

RealClearPolitics HorseRaceBlog

By Jay Cost

« The Culture of Type I Error | HorseRaceBlog Home Page | Thinking About the GOP Nomination »

The Previous Subject, Continued: Quantifying the Debate's Effect

A piece in yesterday's Politico by David Paul Kuhn argued that the debate has had a negative effect on Clinton's national numbers:

Hillary Rodham Clinton's lead nationally and in New Hampshire appears to have slipped following her shaky performance in last week's Democratic debate, according to several polls this week.

If you read on, you'll see the author quote several polls. But he quotes the polls selectively - i.e. in a way that corresponds with the thesis that her numbers are down. This is problematic because poll results vary randomly. That is, they go up and down not necessarily because of changes in the population, but from differences from sample to sample. This is why they have a margin of error. So, you can't just quote the polls that favor your thesis. There could be other polls that fail to favor it - and these need to be factored in.

This is why it is a good idea to average the polls. As you average them, you necessarily decrease the variance of the final result. The variation from sample to sample is inversely related to the sample size. So, a larger sample - or an average that includes many polls - reduces the differences that are due merely to sampling. An average therefore gives you greater purchase on whether the population as a whole has changed.

So - let's do that. RCP has recorded five national polls taken since the Democratic debate on October 30. These five polls put her at an even 45%. So we may have an apples-to-apples comparison, let's take the five national polls that preceded the Democratic debate. These put her at 44%. So, there is no evidence that Clinton's national standing has changed.

Nor should there be. This debate was watched by 2.5 million people nationwide. That's chump change compared to the population that these polls are trying to gauge. It means that everybody else who encountered the debate did so second- or third-hand - in news reports on television, in the newspaper, or from the comments of friends and family. The effect of the debate was therefore greatly diminished for those people.

What about New Hampshire? Well - that is more complicated because there has been only one poll taken since the debate. This would be Rasmussen's poll. Before the debate, Rasmussen found Clinton at 38% in New Hampshire. After the debate, she was down to 34%. The margin of error in this poll is +/- 4%, which means that we cannot reject the hypothesis that the difference is due to sampling variation.

In my previous post, I talked about Type I error. Specifically, I talked about how the press has in place insufficient protocols to prevent its occurrence. This is a perfect example of such poor protocols. Kuhn's analysis runs far too great a risk of Type I error. He is arguing that "something is there" - i.e. there has been a drop in Clinton's support - when in fact there is a very great chance that nothing is there.

-Jay Cost