Pollster Poll Purports Partisan Political Polarization

Nate Silver passes judgment upon the polling outfits of 2010 for accuracy, precision, and the swimsuit portion of the competition and finds out whose polls were the most dependable.  Hint:  it's not Scotty R's boys.

And yes, preliminary analysis showed that nearly all the major polling outfits favored the Republicans, it was just a matter of by how much. But Rasmussen earned themselves a special rebuke (emphasis mine):

The 105 polls released in Senate and gubernatorial races by Rasmussen Reports and its subsidiary, Pulse Opinion Research, missed the final margin between the candidates by 5.8 points, a considerably higher figure than that achieved by most other pollsters. Some 13 of its polls missed by 10 or more points, including one in the Hawaii Senate race that missed the final margin between the candidates by 40 points, the largest error ever recorded in a general election in FiveThirtyEight’s database, which includes all polls conducted since 1998.

Moreover, Rasmussen’s polls were quite biased, overestimating the standing of the Republican candidate by almost 4 points on average. In just 12 cases, Rasmussen’s polls overestimated the margin for the Democrat by 3 or more points. But it did so for the Republican candidate in 55 cases — that is, in more than half of the polls that it issued.

If one focused solely on the final poll issued by Rasmussen Reports or Pulse Opinion Research in each state — rather than including all polls within the three-week interval — it would not have made much difference. Their average error would be 5.7 points rather than 5.8, and their average bias 3.8 points rather than 3.9.

Nor did it make much difference whether the polls were branded as Rasmussen Reports surveys, or instead, were commissioned for Fox News by its subsidiary Pulse Opinion Research. (Both sets of surveys used an essentially identical methodology.) Polls branded as Rasmussen Reports missed by an average of 5.9 points and had a 3.9 point bias. The polls it commissioned on behalf of Fox News had a 5.1 point error, and a 3.6 point bias.

Rasmussen’s polls have come under heavy criticism throughout this election cycle, including from FiveThirtyEight. We have critiqued the firm for its cavalier attitude toward polling convention. Rasmussen, for instance, generally conducts all of its interviews during a single, 4-hour window; speaks with the first person it reaches on the phone rather than using a random selection process; does not call cellphones; does not call back respondents whom it misses initially; and uses a computer script rather than live interviewers to conduct its surveys. These are cost-saving measures which contribute to very low response rates and may lead to biased samples.

Rasmussen also weights their surveys based on preordained assumptions about the party identification of voters in each state, a relatively unusual practice that many polling firms consider dubious since party identification (unlike characteristics like age and gender) is often quite fluid.

Rasmussen’s polls — after a poor debut in 2000 in which they picked the wrong winner in 7 key states in that year’s Presidential race — nevertheless had performed quite strongly in in 2004 and 2006. And they were about average in 2008. But their polls were poor this year.

The discrepancies between Rasmussen Reports polls and those issued by other companies were apparent from virtually the first day that Barack Obama took office. Rasmussen showed Barack Obama’s disapproval rating at 36 percent, for instance, just a week after his inauguration, at a point when no other pollster had that figure higher than 20 percent.

Rasmussen Reports has rarely provided substantive responses to criticisms about its methodology. At one point, Scott Rasmussen, president of the company, suggested that the differences it showed were due to its use of a likely voter model. A FiveThirtyEight analysis, however, revealed that its bias was at least as strong in polls conducted among all adults, before any model of voting likelihood had been applied.

Some of the criticisms have focused on the fact that Mr. Rasmussen is himself a conservative — the same direction in which his polls have generally leaned — although he identifies as an independent rather than Republican. In our view, that is somewhat beside the point. What matters, rather, is that the methodological shortcuts that the firm takes may now be causing it to pay a price in terms of the reliability of its polling.

What Nate here is saying seems to boil down to "It's funny how all the strange methodology that Rasmussen uses seems to favor an overemphasis of traditional Republican voters".  Even in this year's election where Republicans did extraordinarily well, Rasmussen overshot the mark not only by a large margin favoring the GOP, but did so time and time again.  You would think Rasmussen would then adjust its final model based on this, as it did have precision hitting that 4-5 points in the GOP's favor repeatedly.  That didn't happen.

Rasmussen's game appears to be selling a narrative that support for conservative positions was stronger than it really was.  Rasmussen is a prolific poller, polling races that other pollsters would only poll less often.  Considering there were many times this year where the most recent poll in a particular race was Rasmussen's (and other national outfits would go 6-8 weeks or more without a poll of that race) that had to have affected the framing of the coverage of the race.

How many "Republican X is surprisingly close to incumbent Democrat Y" stories out there were based on Rasmussen polls when Rasmussen was off from 4 to nearly 40 points?

Then again, maybe that was the point.  Rasmussen may have sacrificed its credibility to boost the GOP, but if Nate Silver wasn't around to call them on it, would anyone have noticed?  Will that have any effect on polls taken towards 2012, and the Village media's use of them?  Doubtful.

And looking back at that chart, Rasmussen was hardly the only outfit that favored the GOP by a significant margin.  Rasmussen was just the most inaccurate of the lot.

Remember how PPP is considered a Dem leaning polling outfit too?  Real Clear Politics always slaps on (D) after any of their polls.  They actually shaded a bit towards the GOP, and with the least amount of bias towards a party.  I wonder if that D stands for "Damn Straight They're Unbiased".  Does Rasmussen and Marist get an (R) after each poll heading into 2012?

Somehow I doubt anyone in the Village will take notice, especially the guys on the conservative end of things.  Keep that in mind heading forward.