Top Videos
Related Topics
election 2006
republicans
2008 Polls NationalIowaNew HampshireGeneral Election
GOP | DemGOP | DemGOP | DemHead-to-Head

Send to a Friend | Print Article


A Republican Meltdown?

By Jay Cost

Last week I argued that the media consensus that the GOP base is "dispirited" suffers from two analytical problems. First, it is underdetermined -- that is, there are other, equally reasonable but theoretically distinct, explanations that can account for the data. Second, using national data on dispiritedness to estimate the final balance of power in the House is impossible because of the ecological fallacy. All we have are polls that provide national data -- and the distribution of "dispirited" voters is what will make all the difference.

I'd now like to take an opportunity to amplify what I mean by both of these concepts using the most recent Democracy Corps poll of the 49 "most vulnerable" Republican-held districts, and the corresponding memo and graphs. First off, the problem of underdetermination. This is actually a tricky problem to identify because of their use of metaphorical language. Their hypothesis is that there is a "meltdown" coming. What, exactly, does "meltdown" mean? It is hard to say. Charlie Cook reads their metaphor in the following way:

The poll showed that while the 20 or so Republican seats most at risk had not moved much in recent weeks, there was a meltdown for the GOP in second- and third-tier races, which makes a certain amount of sense. In a highly adverse political environment, contested Republican incumbents with the most points to lose -- in terms of both job approval and actual support -- lost the most. Races that were already very close or where the races have been engaged for some time didn't have far to drop.

Fair enough, I think. Let's operationalize "meltdown" in that way: the GOP second- and third-tier seats are now in jeopardy. Their data "shows" that; the fact that the first tier dropped not at all is a sign that they already dropped long ago, that they are at bottom and there is nothing left for the GOP to lose there. .

Are there other, equally reasonable, inferences to be drawn from the data?

Oh -- you betcha!

The most reasonable is statistical error. The poll is a poll of 1,200 respondents. If each tier was sampled equally, then any given tier has 400 respondents. If that is the case, then any given value from any given tier has a sampling error of about 5%. And therefore, this poll gives us little concrete evidence about whom the partisan tide favors. Relative to their last poll of the most vulnerable GOP districts, the Democrats have increased their share of the two-party vote across all three tiers by, respectively, 0.5%, 6.5% and 4.8%. If we allow for a margin of error in both sets of numbers (the previous poll and the current poll), we cannot write off the possibility that, for the first and third tiers, all of this variation is due to normal statistical noise. For the second tier, statistical variation can account for all but about 0.5% of the observed movement toward the Democrats.

Thus, the true values might be consistent with "ripening and ready to be havested," "ripening but not ready to be havested" or, for two of the three tiers, "not ripening at all." The data itself points to no answers.

We also might be able to chalk much of this up to partisan intentionality. Cook mentions the partisanship of the polling company, but I think he fails to take that as far as it should go. He legitimizes the use of the data by saying that others are looking at the data. Well - sure they are, but the real question is whether they discount the Democratic numbers. I am sure that they do. Everybody discounts the preferred party's numbers in a partisan poll! If we assume that partisan polling depresses the other party's prospects a bit, which is a fair (probably the only fair) assumption, then we would bump GOP numbers in each category up. This would not speak to the rate of change between the polls -- but statistical error accounts for almost all of the rate of change. It does, though, speak to the absolute position of the GOP, which is just as important. If we increase their numbers to account for the source of the data -- then, rate of change aside, these seats are just less vulnerable than the authors would have us believe.

I think our cognizance of partisanship in polling should become especially acute in the late stages of the campaign. The Foley scandal has shifted about a half dozen seats away from the GOP (NY 26, FL 13, FL 16, PA 10, NM 01). This has, as I indicated this week, pushed a statistical aggregation of the major race rankers from about 13 to 15 seats flipping to about 15 to 18 seats flipping. What, then, is the correct strategy for the Republicans? You can read it right here. The message: we can hold both chambers. What is the correct strategy for the Democrats? We've been discussing it in this post. The message: we can blow the other side out.

I think we need to be especially cautious, then. Both sides have an electoral interest in inclining us one to one opinion or another. When the parties spin the media -- which they so often do -- they effectively launder a partisan perspective, transforming it into an "objective" analysis. We must be careful not to fall for this -- it is in their interests for this to happen, but it is not in ours. That does not mean that both sides are wrong (though that is of course possible). But it does mean that Democracy Corps' interest cannot simply be assumed to be the dispersion of what I like to call "Capital-T Truth." We need to be mindful of that.

Relatedly -- I cannot help but wonder about the presence of question-order effects in this poll. The interviewer asks the generic ballot before he/she asks the named ballot. Why would they do that? What is the payoff from that? It seems to me that the cost is non-zero: the generic ballot tends to skew toward the Democrats. If you ask a question that inclines people toward the Democrats before you ask a neutral question, might you not be "priming the pump" for the Democrats?

There are other, equally reasonable, inferences as well. The second- and third-tiers feature races where there is relatively little campaign activity. On the top-tier, candidates are unloading huge war chests to crowd out any other political information except what they wish to communicate. On the latter two tiers, this is not so much the case, which is not to say that there is not activity. There is just less. Might these responses be more affected by the national political environment at the time the poll was taken because of less campaign-related information? And, if that is the case, are they "real" responses? Or are they more akin to what we might call "non-opinions" that are drawn not from deep-seated feelings and orientations, but rather from top-of-the-head answers that simply sample the overall media take on the situation? If the latter is the case, then these results are unlikely to be stable because incumbents in these races will soon to unload their war chests to focus the voter not on the generalities of national politics, but national politics in the context of that incumbent. And, furthermore, since the poll was taken during the nadir of the GOP's position in the national press - should we not expect these numbers to improve, independent of candidate activity, just as top-of-the-fold headlines like "Were GOP Leaders involved in Illegal Sex Cover-up?" disappear from the newspaper?

Think of it this way. The top-tier races feature loads of money being spent. This money serves as an anchor that keeps voter opinions from drifting with the media tide. So, they were the least likely to change. In races where the candidates have not been as active, voters are still more inclined to go with the media flow. Hence, the decline. Insofar as Republican incumbents start spending money to anchor these voters -- those numbers will change, especially in light of the fact that Democratic challengers in those races tend to be under-funded and therefore unable to fully counteract incumbent campaign communications.

This alternative hypothesis also does not suffer from a pressing question that emerges in the "bottom dropping out" thesis. Charlie Cook is simply not right -- these races in Tier 1 have not dropped as far as they could go. Just as the GOP is falling apart in the Ohio and Pennsylvania Senate races -- they could also fall in the IN, CT and Philadelphia House races. The Democrats can begin to mount huge leads. These races can be put away. Indeed, if there is some kind of national movement of real significance toward the Democrats, this is actually the first thing we should expect! We should expect the most vulnerable races to trend toward "lean Democrat" and away from "Toss-Up." We should expect the less vulnerable races to trend toward "Toss-Up" and we should expect the non-vulnerable races to trend toward vulnerable. What the Democracy Corps poll shows is the non-vulnerable and less vulnerable races trending toward "Lean Democrat" while the most vulnerable races now becoming the least vulnerable. That is counter-intuitive. If these races in the First Tier are not dropping, then something is preventing them from dropping -- this alternative suggests that it is Republican spending, which -- importantly -- is replicable across most Republican-held districts.

Again -- these are just alternative hypotheses. I am not promoting one over the other. I am not making any claims about which of the four -- Cook, statistical effects, partisan effects, media effects -- is true. What I am saying is that the data itself does not arbitrate between them, and therefore we have no business choosing one over the other.

Now -- as for the ecological fallacy. Does this data point toward any number of seats switching? Stan Greenberg says "Yes." Writing for the similar results of an NPR poll conducted this week by GQR (and "fielded" by POS), he argues:

The NPR survey is identical in its findings to the survey released by Democracy Corps last week, which showed the Democrats in a position to win a majority of the 49 Republican held seats and showed a country determined to vote for change.

Translation: the Democratic lead across the board means 24-25 seats switching, possibly more.

This is not valid. Greenberg has committed the ecological fallacy.

The only way this data has electoral relevance is if these respondents are distributed in a certain way. But the Democracy Corps poll offers no information about their distribution. The inference that is (psychologically) easiest to draw is that they are distributed evenly - and therefore that most of the 49 races show the Democrats with leads. However, this might not be the case. It is also possible that the GOP has a lead in all but few, including (importantly) all but 14. The bottom might only be dropping out in a few places. It might be dropping out everywhere. The important point is that we cannot know. All we have is data of a whole -- and there is no way to draw an inference about the parts based only upon data of the whole.

The rubber hits the road right here. Question: Does the Democracy Corps indicate that the Democrats can put many second- and third-tier seats on the table? Answer: No. Reason: The ecological fallacy.

The slightly-different-but-essentially-the-same NPR Poll offers some helpful cross-tabs that mitigate the problem of the ecological fallacy -- specifically, they break the races down by region or by which party holds the seat. This reduces the severity of the ecological fallacy -- and probably reduces the number of "perverse" distributions that are supportable (i.e. distributions in which, in the macro, the GOP loses by a lot, but on the micro picks up 90%+ of the seats). But it does not eliminate the scenarios in which the GOP loses less than 15 of its seats (or even less than 10 seats) -- nor, importantly, does it arbitrate between the different scenarios that are supported by the aggregate data. Also, such cross-tabulations come at a severe price. As we sub-divide a 1,000 person sample (the NPR poll's size) into 4 quadrants, we are left with only (on average) 250 respondents, and therefore an intolerably high margin of error. Where the NPR poll would be of most help in terms of the ecological fallacy -- a cross-tabulation of both geography and partisanship -- is where the error term probably hits about 7% or 8%, which makes it completely unhelpful. In other words, Greenberg's conclusion is justified neither by the NPR poll nor the Democracy Corps poll.

As a method, I find this "polling of the X most vulnerable races" to be quite suspect. Not in and of itself, but rather because it inclines one to draw race-by-race inferences. But these sorts of inferences cannot be drawn. At all. I view polls like this as akin to entrapment - they are goading you into making an inferential error.

Polls like this are consistent with a general phenomenon I have noted: media polls are designed to maximize news value, not truth value. I see a lot of that. Media polls always seem to offer lots of heat and very little light.

I'll close with a caveat. Am I saying that Cook, Carville, Greenberg, etc are objectively wrong? Nope. Not at all. This is a methodological critique only. Not a substantive critique. They could all be right. But if they are, it will be for the wrong reasons. Accordingly, none of this should be taken to imply that I am favoring one hypothesis over the other. I am merely a "methods hound." And yesterday evening I sniffed out a fox. That's it.


Email Friend | Print | RSS | Add to Del.icio.us | Add to Digg
Sponsored Links

Jay Cost
Author Archive