Handicapping the Presidential Race: Watch the Samples and Weighting
August 13, 2012 · 11:07 PM EDT
It’s time for another caution about polls and what they mean.
The most recent set of six CBS News/New York Times state polls conducted by Quinnipiac University and the latest Fox News national poll should remind us how important each survey’s sample is in understanding the ballot test.
A sample that seems disproportionately Democratic based on recent elections, as was the case with the Quinnipiac’s Florida poll, or disproportionately Republican, as with Quinnipiac’s Colorado poll, will produce results that should raise eyebrows – and doubts about the ballot test.
And, as the most recent Fox News national poll demonstrates, when the partisan make-up of a sample differs significantly from the previous month’s, you may see a trend that says more about the samples than about the race.
In Quinnipiac’s Florida survey, which found President Barack Obama leading Republican challenger Mitt Romney by six points, 51 percent to 45 percent, the survey recorded the results of 373 respondents who identified themselves as Democrats (31.7 percent), 359 who called themselves Republicans (30.5 percent) and 393 who identified themselves as Independents (33.4 percent).
That slightly larger Democratic sample is reasonable considering that the 2008 Florida exit poll showed the electorate was three points more Democratic than Republican (37 percent to 34 percent), and the 2010 exit poll showed that the state’s electorate was split evenly between Republicans and Democrats (each with 36 percent).
But since the 1,177 likely voters in the Quinnipiac/CBS News/New York Times survey didn’t reflect the state’s demographic make-up – for example, blacks constituted only 8.4 percent of the sample, well below the group’s percentage in the 2010 census – the responses were “weighted” to create a more representative sample. This isn’t unusual. In fact, it is standard industry practice.
However, the weighting in Quinnipiac’s Florida poll also altered the partisan composition in the revised sample. (Like many media polls, Quinnipiac’s CBS News/New York Times state polls don’t weight for partisanship, since it’s impossible to know what the precise partisan make-up of the electorate will be in November. Most partisan pollsters either weight for party or make sure their sample reflects the likely partisan composition of the electorate, since it is such an important factor.)
So while the weighting sought to correct one problem, in my view, it created another.
The weighted Quinnipiac sample in Florida was 36 percent Democrat and only 27 percent Republican – a dramatic shift from the unweighted numbers and a partisan mix that doesn’t come close to resembling the ’08 and ’10 Florida electorates, and, I would argue, is unlikely to reflect the electorate in November.
I have no idea what the exact partisan split will be in the Florida electorate in 2012, but I’m pretty sure that Democrats won’t have a nine percentage point advantage.
Given the increased polarization in the country, partisanship is an important predictor of the vote, so a sample that is more Democratic will produce a better showing for Obama and one that is more Republican will produce a better result for Romney.
Obama’s six point lead in the ballot test in the Florida Quinnipiac poll and Romney’s five-point lead in the ballot test in Quinnipiac’s Colorado poll aren’t surprising given the partisan make-up of each weighted sample. Making the sample less Democratic in Florida and less Republican in Colorado – as they probably will be in November -- would make both contests look much closer than they do in the Quinnipiac ballot tests.
In the August Fox News national poll, 44 percent of respondents were Democrats while only 35 percent were Republicans. Not surprisingly given the sample, Obama held a 9-point advantage over Romney, 49 percent to 40 percent. Just one month earlier, Fox’s sample was only 42 percent Democratic and 38 percent Republican, and, predictably, Obama held a more narrow 45 percent to 41 percent lead.
Has there been a dramatic shift in partisan self-identification over the past month? It’s possible but quite unlikely. The difference in the samples could come from the weighting or merely from the nature of random samples.
In focusing on these surveys, I wouldn’t want to give the impression that other surveys don’t have the same problems. Even the NBC News/Wall Street Journal poll has had these kinds of hiccups. The July sample was significantly more Democratic than the June sample, giving the impression that President Obama was expanding his narrow lead over Romney. Maybe the president had expanded his lead, but it’s hard to tell given the different samples.
I certainly am not suggesting that there is anything nefarious about weighting, nor do I question the professionalism of the organizations doing the polling. In fact, the transparency of the Quinnipiac/CBS News/New York Times survey results deserves to be noted and applauded.
But readers must understand the limitations of polling, and I find too few caveats by most reporters, television talking heads and, yes, pollsters when it comes to how samples and weighting are affecting the presidential ballot test.