As the recent Ontario election exemplified, election polls are usually quite accurate and provide voters with a window onto the intentions of their fellow citizens. This accuracy creates a feedback loop that may be helpful in that most democratic of acts: voting. Polls draw voters’ attention to particular candidates and issues and allow them to make tactical decisions, if they choose.

However, polls have never been as good as their famous disclaimers imply — plus or minus a specified margin of error, 19 times out of 20. That’s because the disclaimer captures only one kind of theoretical statistical error. It does not address, for example, cases where a pollster has written a poor or confusing questionnaire, or sampled the wrong people, or failed to weight the results properly for the characteristics of the population under study.

Sometimes polls go wrong, even badly wrong. And when they do, they can damage the election process. Misinformation may skew the media’s depiction of the campaign and create a mistaken narrative in the minds of the voters — and even among the candidates and their campaigns. Polls like these are an impediment to democracy, not to mention an embarrassment to the public opinion industry.

In the campaign for the Calgary mayoralty in October 2017, the polls were wrong. Badly wrong. The most widely publicized polls were conducted using Interactive Voice Response by Mainstreet Research in cooperation with Calgary’s two major daily newspapers, the Calgary Herald and Calgary Sun. Three polls spaced out over the length of the autumn campaign told a startling story. According to these polls, contrary to the expectations of most careful observers of the Calgary political scene, Mayor Naheed Nenshi, the two-time incumbent, was trailing a relatively unknown businessman, Bill Smith, who had never before sought elected office. And Nenshi, according to these polls, was not just trailing Smith, but trailing him badly, at one point by nearly 17 percentage points.

For the most part, these polls were reported uncritically by the two newspapers. When academics, other pollsters and, not surprisingly, the Nenshi campaign, raised methodological questions about the polls, including anomalous results among women and younger voters, the newspapers sometimes ignored them or reported them in a defensive tone. Executives at Postmedia, which owns both papers, were privately worried about the accuracy of the polls, we were told after the election by Mainstreet executives, but they did not share these concerns with their readers.

Within Mainstreet there was also concern. However, its executives maintained an astonishingly assertive and self-confident tone in the media and on social media, threatening legal action and menacing critics with “payback” after the election.

Smith did not win by nine, or seventeen, or twelve percentage points, as Mainstreet polls had suggested he might. In fact, Nenshi was returned by a comfortable margin of nearly eight percentage points – not just outside Mainstreet’s claimed margin of error, but many multiples of it. In the meantime, campaigns had been rocked and voters misled.

Strangely enough, two other polls of note appeared during the election campaign, both conducted online. These online polls, one conducted for an advocacy group, LRT on the Green and another by a group of academics, the Canadian Municipal Election Study, appeared late in the campaign and did not enjoy the megaphone of the Postmedia newspapers. As a result, they had less impact. Confoundingly, these polls painted the opposite picture from the Mainstreet polls, giving Nenshi a double-digit lead — also well outside their stated margins of error when compared with the election results.

In January 2018, the Marketing Research and Intelligence Association (MRIA), which was the industry association representing the public opinion industry, appointed a panel consisting of the three of us — all academics with experience working in the industry — to conduct an independent review of what went wrong in Calgary and to make recommendations for the future. We studied the polling information and media reports, and interviewed many of the principals in the Calgary controversy, including Mainstreet executives and Mayor Nenshi, as well as journalists, polling experts and academics in Calgary and Toronto.

We prepared a substantial report and submitted it in May 2018, expecting it to be released in time for the organization’s annual meeting in Vancouver in June. We were frankly mystified that it was not. Subsequently, in late July, the MRIA announced that it was disbanding because of financial issues. As panel members, we decided that we should release our independent report as originally submitted in May because we believe the observations, conclusions and recommendations are important for the public, the media and the polling industry.

What our report found was that Mainstreet began its polling by drawing its samples from a “directory” that was likely flawed in that it underrepresented younger voters. In reaction to criticisms of its first poll, Mainstreet changed its methodology in a way that inadvertently made its problems worse by stripping out many of its cell phone respondents. When younger voters turned out to vote in surprising numbers, the vulnerabilities of Mainstreet’s methodology were further exposed.

This performance would have been a terrible embarrassment as it was. However, as documented in our report, the bloated self-confidence and acrid aggression of Mainstreet’s comments on social media and in the mainstream media raised the stakes. We also found that the Postmedia newspapers did not apply routine journalistic scrutiny to the Mainstreet polls, but treated them as a proprietary resource to be defended rather than critiqued. In many cases, other media organizations struggled to address the issues raised by the polls — and by the end of the campaign, the contradictions among them — but they did not have the information, background or expertise to do this effectively.

The panel made many recommendations, and they can be summarized as follows:

• Political polls should be much more transparent, and should routinely report how the sample was derived, the proportion of cell phones (where applicable), response rates, callback procedures, etc., as well as who is paying for the poll and whether work is being done for other participants or interested parties in the election.
• The standards of disclosure should be monitored and enforced by the industry association (presumably the successor to the MRIA).
• The industry association should develop a system to calculate and communicate “historical margin of error” — that is, the accuracy of polls as compared with actual election results, which would capture real-world error rather than merely theoretical statistical error.
• Representatives of polling firms should conduct themselves, in the media and social media, in keeping with professional standards and as social scientists rather than pundits or self-promoting entrepreneurs.

Media organizations should

• clearly disclose the nature of their relationship with polling companies, including the financial relationship
• apply routine critical journalistic scrutiny to all polls, including those available to them exclusively
• establish, with the help of the industry association, professional standards for the reporting of polls and improve training for reporters who regularly cover them

Our report includes many additional recommendations as to ways the MRIA could have nurtured a better environment for the conduct and reporting of polls. We believe they apply equally to any future body that may represent the industry, and indeed a group of leading firms is  discussing the establishment of such an organization. We hope our report will encourage a debate that will increase transparency, inform public opinion research, and help the application of better standards by whatever new organization emerges.

At its best, polling is a democratic craft, but for that to be true, it must conform to the same standards of transparency and accountability that we should demand of all the institutions that play a role in our democracy.

The report A Review of the Public Opinion Polling Conducted During the 2017 Calgary Election was prepared for the Marketing Research and Intelligence Association by Christopher Adams, Paul Adams and David Zussman.

Photo: Naheed Nenshi celebrates his victory as Calgary’s mayor following municipal elections, Monday, October 16, 2017. The Canadian Press, by Jeff McIntosh.

Do you have something to say about the article you just read? Be part of the Policy Options discussion, and send in your own submission. Here is a link on how to do it. | Souhaitez-vous réagir à cet article ? Joignez-vous aux débats d’Options politiques et soumettez-nous votre texte en suivant ces directives.