Month: October 2011

Polling firms pay their bills by finding out what people think about things and then selling this information to corporate clients. But if you're a corporate client, how do you know which pollster produces more accurate results? Here's the problem: A toothpaste maker might want to find out if consumers prefer green or yellow packaging and might commission a poll or focus group to find that out. But, of course, there will never be a poll in which all toothpaste buyers participate. You can never for certain if, in fact, green was preferred to yellow. You have to trust the polling firm and its methodology.

But corporate clients looking for polling firms do have one yardstick they can use to measure a prospective firm's method and approach against a poll which includes the entire population that the poll measures. It's called an election. As David Coletto, CEO of Abacus Data, the polling firm used by Sun News Network, notes: “It’s the one time we’re held to account for our numbers.”

Polling firms, throughout an election, poll a very small group of voters and then use those results to explain and possibly predict the behaviour of all voters.

During the just-concluded Ontario provincial election, polling methods briefly popped up on the radar because one polling firm, Ipsos Reid,  called out other polling firms and journalists in Canada using remarkably strong language. Ipsos executives did not name names but said some of their competitors were “hucksters selling methodological snake oil.” This rather extraordinary letter warned journalists who used these polls that “we are distorting our democracy, confusing voters, and destroying what should be a source of truth in election campaigns – the unbiased, truly scientific public opinion poll.”

Obviously, for polling firms, arguments about who's right or wrong and why are important industry issues. But for the rest of us, we're probably most interested in the bottom line question: Whose poll was closest to the actual outcome? Which “unbiased, truly scientific public opinion poll” most closely matched the results? Well, unfortunately for Ipsos, it was not their poll. In fact, they had the largest variance from the actual results.

Notably, the polling firms that did best against the results used three very different methods. Abacus Data — which was one of the firms that appears to have prompted the outburst from Ipsos Reid — was nearly bang on its final poll and it uses online polling. Forum Research and Ekos were nearly spot-on and they use an automated interactive voice research (IVR) method. Nanos Research nailed Liberal support and was within the margin of error for support for the PCs and NDP using more traditional telephone polling.

Ipsos, too, uses telephone polling, but its last poll of the campaign had the biggest variance with the actual results. They found Liberals leading by 10 points over the Progressives Conservatives, 41 per cent to 31 per cent. By contrast Abacus was alone among the pollsters in calling a three point margin for the Liberals and that, sure enough, was the margin of victory when the votes were counted.

Eric Grenier, who is obsessed (in a good way!) with polling and politics through his excellent 308dotcom project assesses the performance of the pollsters in the election and writes that “ironically, in light of the controversy stirred up in the opening days of the campaign, the two most accurate polls came from the newest firms: Forum Research … and Abacus Data.“

Here's a review of how polling firms did compared to the results:

Darrell Bricker, the CEO of Ipsos Reid, has this comment on this chart:

“While the conclusion one could reach from this chart is that certain polls performed better than others in predicting the election outcome, that would only be true if all the surveys were done the same way, on the same days, and had used identical samples. All of these surveys are unique and should only be assessed against their own assumptions, including the appropriate margin of error”.

And here's some other interesting day-after responses from other polling firms:

• Abacus: On a roll: Two Election Calls in a Row
  • “… Most public opinion researchers did well last night and that is good news for the industry as a whole …”
• Angus Reid: Provincial Elections: What We Saw and How We Acted
  • “…Our record has proven that online panels—when combined with proper sampling techniques, thorough discipline in questionnaire writing and development, and an openness to work until the final moments of a campaign—can be used to accurately predict the views of the Canadian electorate. In fact, no other company in the country comes close to matching our record of both participation and accuracy…”
• Ekos: A Brief Post Mortem on Polling During the Ontario 2011 Election
  • “…at EKOS are pleased that not only did we accurately project the popular vote, we believe we did a good job in charting the direction of the election. For instance, we were the first to call a Liberal victory in our September 27th release …”

 

 

Should be an exciting night in Ontario Thursday as voters there pass judgement on the incumbent Liberal government of Dalton McGuinty. It should be exciting because, according to some polls out this weekend, the outcome seems far from decided:

• A Leger poll for my organization, Sun Media, shows: PC 32% / LPO 32% /NDP 29%. Leger polled a representative sample of 1,102 Ontarians drawn from its online panel of 350,000 Canadians on Sept. 28 and 29. The pollster says this kind of poll should be accurate to withign 3.1 percentage points, 19 times out of 20.  So, based on that, all three parties are within the margin of error. In other words, they're tied.
• The folks at Angus Reid, doing a similar online panel at almost exactly the same time, get a similar headline: Progressive Conservatives and Liberals Are Virtually Tied in Ontario with the following results: PC 34% / LPO 33% / NDP 26%. Click thru on the link for methodology, etc.
• Over at Environics it's a similar story — a tie between Conservatives and Liberals — though they find the NDP a little further back. The Environics scorecard: PC 36% / LPO 35% / NDP 25%. Environics, too, polled about a 1,000 online from Sept 26-28.

So far as the regions go — sample sizes are smaller and therefore, margins of error increase — all the pollsters find Tim Hudak's Tories doing great in the southwestern, eastern, and northern parts of the province. Andrea Horwarth's NDP are doing well in the ridings in her hometown of Hamilton. And McGuinty's Liberals are rock-solid in Toronto and are nipping at Hudak's heels in the 905 ring around Toronto.