About this Blog
About The Author
Email Me

RealClearPolitics HorseRaceBlog

By Jay Cost

« Consensus, Not Courage | HorseRaceBlog Home Page | Immigration Reform and the Structure of Congress »

Pitfalls of Presidential Punditry

This week I will start offering my thoughts on the major presidential candidates. Before I do that, I need to flag a potential pitfall inherent to the project. As usual, the best way to introduce a cautionary tale is via Monty Python's Flying Circus.

The moral is: be wary of undetermined hypotheses!

What is an underdetermined hypothesis? Wikipedia's definition suffices for an answer:

A theory (or statement or belief) is underdetermined if, given the available evidence, there is a rival theory which is inconsistent with the theory that is at least as consistent with the evidence. Underdetermination is an epistemological issue about the relation of evidence to conclusions.

Return to the Python bit. Archaeologists discover the toe, bring it to the museum, and - based upon that single piece of data - construct a complete, new, exciting explanation...that has absolutely nothing in common with reality! They presumably rejected many different theories just as reasonable as the one they accepted. The hypothesis of the archaeologists in the Python bit is thus undetermined.

This is a lesson for all of us, regardless of whether we work at the Field Museum. People, like cats, are curious. We like to ponder the unexplained, and develop explanations for it. Like cats, this can get us into trouble. We often acquire only a little bit of data; but from that tiny bit, we can still create a comprehensive story that is not the only one amenable to the data. When we develop underdetermined hypotheses, we overuse our data.

This is particularly relevant when analyzing presidential politics. Our desire for comprehensive knowledge on this subject is vast, but our data set is currently limited. We must be careful not to draw underdetermined hypotheses by overusing the data. I find that many pundits do indeed overuse the two main sources of data, namely polling data and financial data.

We are virtually saturated with polling. It facilitates much of the handicapping. Unfortunately, pundits often fail to see the limitations inherent to polling, and they overuse it.

A horse race poll is an attempt to gauge what people will do, not what they think now or what they have done. Thus, the poll is a valid predictor only insofar as (a) people are not expected to change their minds between their answers and the act of voting, (b) the poll itself is a reasonably good mimicry of the act of voting.

Obviously, condition (a) does not hold yet. And, as I discussed previously, condition (b) does not hold - for anybody except Republican voters in the Iowa caucus. Democrats in Iowa do not select a preference at the caucus in a way that resembles any poll. Meanwhile, voters outside of Iowa will have a constrained choice - as presumably candidates will drop out of the race after the Iowa caucus. Some polls offer us some purchase, even in the face of these problems. Both Gallup and RT Strategies, for instance, ask for some second choice preferences. However, this assistance is incomplete. Analysts do not have a full delineation of preferences for individuals, which can make a big difference. It might be the case that some first choices and second choices are correlated - e.g. a first choice preference for John Edwards implies a second choice preference for Barack Obama. In that case, the fact that Hillary Clinton is in the lead might be quite inconsequential - for her lead is predicated upon Edwards being in the race, which - for many people when it comes time to vote - he might not be.

I'm not saying that the inferences we draw from the polls are wrong. I am not saying that they are right. I am just saying that the polls allow for a multiplicity of inferences, all of which are as valid as any other. Is Rudy in the lead? Maybe. Is Romney in the lead because he leads in Iowa and New Hampshire and will be able to "slingshot" his way through Super-Duper Tuesday? Maybe. Is Hillary in the lead? Maybe. Is Obama in the lead because Edwards is not going to finish? Maybe. Is Thompson in the lead because McCain is not going to finish? Maybe! All of these positions are supportable by recourse to the polls. This means that all of these hypotheses are underdetermined. We must be mindful of this, and not use the polls to advocate one hypothesis over another.

The second data point that people use is money. This data is more valid than polls insofar as money today means resources tomorrow, whereas support in a poll today does not mean votes tomorrow. However, pundits still overuse this data.

To appreciate this, recall that by the summer of 1999, George W. Bush was a financial Goliath. He had so much money that he could impose his will upon the rest of the candidates. How much money did George W. Bush raise in the First Quarter of 1999?

$7.6 million. That's it.

So, why do we see stories like this:

Only a few months ago, political operatives were speculating whether Sen. Barack Obama could come close to Sen. Hillary Rodham Clinton's daunting fundraising machine. Now Team Obama is the legend, and the question is whether the junior senator from New York can keep up.

Hilary Clinton raised $26 million in the first quarter of 2007 - that is 342% of Bush's total receipts in the same period. The only way that this - "[Can] the junior senator from New York...keep up"? - is the question is if we have misunderstood the role of money. Is it a necessary condition for electoral success? Yes. Is it a sufficient condition? No. It is a necessary, but insufficient condition. If you don't have enough money, you won't win. But it doesn't mean that if you do have enough money, you will win.

Of course, we should be impressed by Mr. Obama's fundraising abilities - and we should take that as a sign that a large quadrant of the Democratic Party's elite support Obama. But to move from that observation to this question is to fall prey to overusing the data.

The problem is that pundits and analysts have turned the race for dollars into a proxy for the race for votes. This is inappropriate, which is why House incumbents who raise more money are more likely to lose. Incumbents who sense danger draw to themselves as many dollars as they can; but, as dollars cannot achieve victory, the danger remains. Money does not win an election. It just gives the opportunity to win.

By turning the race for dollars into a proxy for the race for votes, pundits implicitly turn fundraising into a sufficient condition, not a necessary one. They see that Obama has raised more than Clinton, and assume that Obama somehow has an advantage. In reality, the question that they should ask from the money is: are these candidates on track to raise enough to compete? And the answer is yes, they are! Obama is on track to raise enough, and so also is Clinton.

What I am suggesting by these points is that we are far too inclined to overuse our data. We want to draw more inferences than the data will allow - so we take them further than we should. The reason is perfectly human. We're all curious about what is going to happen, and we thus jump all over any piece of data that we can find.

Does this mean that we cannot say anything of value about the presidential election? Of course not. So, what should we do?

I think it is prudent first to lower expectations. Let us not expect that we can handicap the race per se. Otherwise, we will be inclined to overuse our data. Accordingly, over the next few days, I will merely offer broad remarks about the major candidates. My intention will not be to handicap, but to frame what I think are the relevant questions about their candidacies. This analytical expectation is more modest, and it will decrease the chance that I will overuse the data.

It is also prudent to be mindful of the limits of any data. This means that we should pause - as we have done here - to think about what constitutes valid, and invalid use. Doing so decreases the chances of accepting an underdetermined hypothesis.

Finally, we should keep an open mind. All arguments vary based upon their finality - and just because an argument is not final does not mean that it is not worthwhile. Only logical proofs are final. Everything else is subject to revision as more facts come to light. We should recognize that it is not just that the state of the race might change, our perceptions might change, too. Thus, we should not be too enamored of our own opinions, and not too inclined to reject the analyses of others.

-Jay Cost