Are Poll Sampling Complaints Legit?

Are Poll Sampling Complaints Legit?

By Sean Trende - April 20, 2012

Every election brings a raft of complaints about polls. I don't expect this one to be any different -- Republicans will bemoan the overabundance of Democrats in the samples, while Democrats will claim that the polls under-sample minorities.

What make these criticisms more salient this year, however, are signs that "working the refs" is having an effect on polling companies. In 2008, in response to complaints from many observers that minority and young voters were undercounted, Gallup actually produced two models: “traditional” and “expanded.”

The Battleground Poll likewise produced two different results, reflecting the differing assumptions of pollsters Celinda Lake and Lance Tarrance Jr. about the makeup of the electorate. In 2010, Gallup once again produced two separate samples after being continually harangued by Democratic observers.

The results of this “jawboning” of polling companies are mixed at best. In 2010, the criticism of Gallup was correct; in 2008, the traditional model had it right. So it was jarring to see the criticism taken to a higher level when Obama campaign strategist David Axelrod accused a recent Gallup poll of having “serious methodological problems.” He then directed anyone interested to this column from National Journal’s Ron Brownstein to explain why:

“The Gallup track, which is conducted among registered voters, has a sample that looks much more like the electorate in 2010 than the voting population that is likely to turn out in 2012: only 22 percent of the Gallup survey was non-white, according to figures the organization provided to Emory University political scientist Alan Abramowitz. That was close to the non-white share of the vote in 2010 (23 percent), but in 2008, minorities comprised 26 percent of all voters, according to exit polls; the Obama campaign, and other analysts, project the minority share of the vote will increase to 28 percent in 2012. In its survey, Pew, for instance, puts the non-white share at 25 percent.”

Obama advisers have likewise objected to Gallup’s “noisiness” -- in other words, its tendency to show the president’s job approval shifting up and down without any intervening news story that would suggest that there should be movement (although this probably is simply a reflection of movement within the error margin from a pollster that does not weight its sample heavily).

Expect to hear a lot about this in the months ahead.

So, what to make of these criticisms from the left and from the right? I don’t make too much of the latter’s complaints regarding partisan weighting. Every election since 2004 -- the first cycle I wrote about online -- has featured debates among the pundits about whether pollsters should re-weight their samples by partisan identification. The problem with doing so -- and I think the outcome from 2004 shows this -- is that partisan ID is fluid; the partisan composition of the electorate really can change substantially in a relatively short period of time, as it did from 2000 to 2004, and again from 2004 to 2008.

More importantly, what we’re seeing right now is probably a function of the fact that pollsters are largely employing registered-voter screens, or no screens at all. Polls of adults and registered voters have tended to be more heavily Democratic than the actual electorate (Nate Silver estimated that in 2010 the difference was around six points), although this gap can disappear in a year like 2008, when enthusiasm is high on the Democratic side and low on the Republican side. (This year is currently looking a lot more like 2004 than 2008 in that regard.) I suspect these oversized Democratic samples will shrink once we get to the fall and likely-voter screens become ubiquitous.

The objections regarding the minority-vote share, and whether pollsters should aim for a sample that is somewhere over 25 percent minority, require a more in-depth discussion. There are two salient points involved:

1) The minority share of the electorate will probably stay flat, or even decrease, in 2012.

The reason for anticipating a 28 percent non-white electorate is pretty simple: That share of the electorate has been rising steadily since 1992. Since it grew from (using the final exit-poll data) 22.2 percent in 2004 to 25 percent in 2008, we might expect a similar rise this year, to 28 percent. In particular, given the well-documented, rapid growth of the Latino population in the United States, the non-white share of the vote might be expected to increase even further (you can see Abramowitz’s argumentation here).

But as I explain in my book, “The Lost Majority,” predictions about the ethnic composition of the electorate have been incorrect in the past, and are likely to be incorrect in the future. In late 2009, Abramowitz himself predicted that a GOP takeover of the House was highly unlikely because “[b]ased on the average rate of change in the racial composition of the electorate over the past two decades, by 2010 we can predict that no more than 76 percent of voters will be white while at least 11 percent will be African-American and at least 13 percent will be either Hispanic or members of other racial minority groups.” Due to this, Republicans would need to win around 58 percent of the white vote, something Abramowitz judged to be “exceedingly difficult.”

Of course, Republicans won 60 percent of the white vote in 2010, probably the highest share of that vote for Congress won by either party since 1822 (assuming an almost entirely white electorate pre-1952). More importantly for our purposes, whites made up 78 percent of the electorate in that year, in excess of Abramowitz’s 76 percent ceiling.

Let’s analyze this a little closer. Why did the white share of the electorate decline from 2004 to 2008, and why might we not expect the feat to repeat in 2012? To answer this better, let’s break down the electorate from 1996 through 2010 by racial/ethnic group. Please note that I don’t have the final raw data for the 1996 or 2010 exit polls; without precise numbers, there may be rounding errors here:

Of the 2.8 percent decline in the white vote from 2004 to 2008, 1.2 percent is attributable to the growth of the African-American vote, about 0.5 percent to the growth of the Asian vote, and about one point to the growth of the “Other” vote. Note that the Latino vote increased by only 0.2 percent.

1 | 2 | Next Page››

Sean Trende is senior elections analyst for RealClearPolitics. He is a co-author of the 2014 Almanac of American Politics and author of The Lost Majority. He can be reached at Follow him on Twitter @SeanTrende.

A President Who Is Hearing Things
Richard Benedetto · November 12, 2014
Obama Is No Clinton
Larry Elder · November 13, 2014
Bret Stephens' Call for Robust U.S. Foreign Policy
Peter Berkowitz · November 16, 2014

Latest On Twitter