Best Pollster in NC Was... | Eastern North Carolina Now

    Publisher's note: This article appeared on John Hood's daily column in the Carolina Journal, which, because of Author / Publisher Hood, is linked to the John Locke Foundation.

    RALEIGH     Polling, it is said, is more of an art than a science. Pollsters are among the first to say this — particularly when the products, causes, or candidates they project to succeed fall flat. When their predictions turn out to be on the mark, however, pollsters hope you picture them with pocket protectors instead of palettes.

    Questions of methodology and transparency aside, political polling is fundamentally about generating useful, reliable predictions about how the general population thinks or will vote. During the 2014 election cycle, many pollsters didn't. According to polling analyst Nate Silver, publicly released polls on gubernatorial and Senate races were, on average, about four percentage points too generous to Democrats nationwide.

    How well did the pollsters do here in North Carolina? During the past three months, I tracked every public poll in the race between incumbent Sen. Kay Hagan and House Speaker Thom Tillis. I kept count not just of the "topline" results for the two major-party candidates but also the share of likely voters who said they favored Libertarian Sean Haugh, the share who said they were undecided, and the demographics of the likely voters in each sample.

    By my count, there were 17 different pollsters that produced at least one Tillis-Hagan-Haugh survey during the last month of the race. Several produced more than one poll. For starters, let's look at the final survey produced by each pollster. Just two of the 17 — Pennsylvania-based Harper Polling and Florida-based Gravis Marketing — came close to predicting Tillis's margin of victory, which was just shy of 2 percent. Harper actually had Tillis up 2 points. Gravis had him up 1 point.

    The next group of pollsters did fairly well by having Tillis and Hagan tied in their last surveys. Five yielded this result: Survey USA (which polls for several media outlets and institutions in North Carolina), National Research (which polls for the Civitas Institute), Marist College (which polls for NBC News), Public Opinion Strategies (a Republican firm that polled for the Tillis campaign), and High Point University. Two additional polling units produced estimates within about 3 points of the final result: Fox News and Rasmussen Research.

    Finally, there were eight pollsters whose estimates were far off, between 4 and 6 points: CNN, Monmouth University, Elon University, the new Republican firm Vox Populi, Suffolk University (which polls for USA Today), the Lukens Company (which polled for a pro-life group), YouGov (which polls for CBS and the New York Times), and North Carolina's own Public Policy Polling, which does surveys for liberal organizations and Democratic candidates.

    Did live-caller polls do consistently better than automated ones? Not in North Carolina, at least. The two with the best predictions, Harper and Gravis, are both automated surveys. Of the 17 pollsters I reviewed, six do robo-polls. Only two of the six, Vox Populi and PPP, had below-average results. On the other hand, the two pollsters that relied on Internet sampling, Lukens and YouGov, both did poorly this year in North Carolina.

    Did most of the pollsters at least detect movement toward Tillis at the end of the campaign? Yes. Of the 13 that conducted at least two polls, seven showed the Republican narrowing the gap with Hagan in the homestretch. Among the ones that didn't, however, were three familiar names in North Carolina: PPP, Elon, and National Research/Civitas.

    Some of the problem lay not in collecting and weighting the poll samples but in interpreting them. There was a widespread assumption that, when it came down to the last few days, North Carolinians still saying they were undecided would either break evenly between Hagan and Tillis or just stay home on Election Day. That's not what happened. There truly were some undecided but likely voters, and they ended up breaking 2-to-1 for Thom Tillis.

    So here's a tip for the next election: when final polls show two candidates within a point or two of each other and the undecided vote three or four times that gap, just assume it's too close to call. Pop some popcorn. And don't try to guess the end of the story.