One of the snapshot verdicts on the 2004 federal election was that the major polling firms largely missed the end-of-campaign shift in voter prefer- ences that boosted Liberal fortunes to 135 seats, from the 108 to 125 range projected by superimposing polling data on the regional distribution of seats from the 2000 contest.
A detailed analysis of polling data and electoral out- comes suggests that the technical models used to make these projections are fairly reliable. However, the use of polling for pur- poses of electoral forecasting suffers from four inherent limitations:
Canada is not a national political market but a highly regionalized one, reflecting differences in economic and social interests and, sometimes, values. Voter attitudes and trends do not move in a synchronized fashion in different parts of the country. As a result, national polling figures are functionally irrelevant to election forecasts.
In some regions, there are signifi- cant internal cleavages among urban, suburban, and rural seg- ments of a region which are largely ignored in published polling data. Moreover, the regional samples of individual polls are usually fairly small, with high margins of error.
Elections are dynamic environ- ments in which both parties and many voters adapt their respec- tive strategies and actions (including voting decisions) to external stimulae and feedback ”” including polls. Regional polling data may hint at trends, but they generally provide little indication of differences between firmly and marginally committed voters, or the extent to which undecided voters (whose last minute deci- sions can shift a close election) are engaged with the political process. Polls conducted late in the campaign suggest that as many as 25 percent of voters were liable to change their minds at the last minute.
The rapid growth of telephone marketing and solicitation in recent years is contributing to consumer fatigue and larger ”œrejection rates.” Although there are many techniques to test the statistical validity of survey sam- ples, high rejection rates add an additional dimension of uncertainty to survey results, especially during times of electoral volatili- ty. These concerns are compound- ed by widely varying turnout levels among different demo- graphic groups, even though most Canadians tell pollsters they intend to vote.
This article examines polling data for each region during the cam- paign, and compares it with actual outcomes by region. It concludes that although most pollsters missed the extent of last minute vote swings in Ontario, regional vote projections for three of four major polling firms that publishing regular polls during the campaign were well within national margins of error.
Based on five national and one regional election surveys carried out between June 20 and 24, 2004, 64 percent of final week polling projec- tions assessed regional levels of major party support within margins of error of 3 percent (see table 1). This compares with statistical mar- gins of error associated with typical survey sample sizes ranging from 5 to 6 percent in Ontario to more than 10 percent in Atlantic Canada, Manitoba and Saskatchewan. Major pollsters’ forecasts were most accu- rate in Alberta and British Columbia, with 80 and 72 percent of projec- tions accurate within 3 percentage points, and most vulnerable to last minute vote shifts in Ontario.
These findings suggest that pub- lished media polls are useful for con- veying information on election trends to the voting public that would other- wise be available only to party profes- sionals. However, the news media and other analysts could usefully make some adjustments in the ways that polling data is reported.
The 2004 federal election ”” and the sweepstakes for polling cred- ibility ”” were won and lost in Ontario. Not only did the province’s 106 seats include the largest number of competitive seats, particularly fol- lowing the merger of the Canadian Alliance and Progressive Conser- vative parties, but the combined effects of federal scandals and an unpopular provincial Liberal budget greatly increased the percentage of the electorate willing to consider shifting their votes. Larger polling samples in Ontario reduced margins of error compared with other parts of the country ”” particularly during the last two weeks of the campaign, when some polling firms significant- ly increased the size of their sam- ples.
Most published polls indicated a surge in Conservative and, to a less- er extent, NDP support during the first two weeks of the election cam- paign. Liberal support dropped from the 45-49 percent range in polls con- ducted in late April, to the 36-40 percent range in polls conducted at the end of May, to a nadir of 33 to 38 percent in polls conducted around the time of the leaders’ debate in mid-June (see table 2). Conservative support during this period increased from the 25-33 per- cent range in late April, to the 35-39 percent range by the end of May ”” a level at which it stabilized until mid- June. Assessments of NDP support ranged from 18 to 22 percent in late April, to 20 to 25 percent in late May, and 18 to 26 percent in mid- June. As in several regions, the pres- ence of the Green Party as an increasingly visible centre-left alter- native to the Liberals and NDP con- tributed to both the volatility and uncertainty of the race.
During the last 10 days of the campaign, most pollsters discerned the trend in public opinion back to the Liberals and away from the Con- servatives and NDP (see table 3). Lib- eral support increased to the range of 38 to 43 percent, with the Conserva- tives dropping back to 30 to 35 per- cent, and the NDP remaining fairly stable, but within the wide range of 17 to 25 percent. Two of the six polls conducted in the final stretch of the campaign, Ipsos-Reid (18-20 June) and Compas (23-24 June; 28 June) clearly caught the outlines of the final results, although Ipsos-Reid’s last poll (21-23 June) suggested a slight Conservative recovery. These trends were magnified by urban-rural dynamics in which the Liberals were returned with strong majorities in most of their urban seats, and Con- servative support was concentrated primarily in rural eastern and south- western Ontario and the fringes of the greater Toronto area.
Did the pollsters get it ”œwrong” in Ontario? With the two exceptions noted earlier, there is no doubt that, at 13.2 percent, the Liberals’ winning margin in the popular vote was substantially larger than that predicted by most pollsters. Fewer than half of the party vote projections made dur- ing the last week proved to be within three percentage points of the final outcome ”” the lowest of any region. The ”œhorserace” syndrome in media coverage was reinforced by pollsters’ ”œseat projections” that projected province-wide trends on to the results of the 2000 federal election without taking into consideration Ontario’s widely varying local and sub-regional dynamics, or the spuri- ous correlation of past Alliance and PC support in projecting constituen- cy outcomes. As these projections were widely touted by major newspa- pers and television outlets that spon- sored the polls, the result did little to enhance pollsters’ credibility.
The second major battleground of the 2004 election was British Columbia. A combination of open seats, caucus defections, and three- way contests made at least one-third of the province’s 36 seats highly competitive, in sharp contrast to most other provinces. In sharp con- trast to Ontario, the pollsters got it largely right in projecting BC vote results, despite considerable volatility during the election campaign.
British Columbia is characterized by four major sub-regions ”” Vancouver/Burnaby/New Westminster, Vancouver Island, the rest of the Lower Mainland, and the BC interior. All these sub-regions have distinctive voting patterns, as noted by Ipsos- Reid, the one national polling firm to survey a large enough sample of vot- ers to measure voter outlooks in different segments of the province.
The final result placed the Conser- vatives at 36.2 percent of the vote, down from 49.4 percent from Alliance totals in the 2000 election, the Liberals at 28.6 percent (up 0.9 percent), the NDP at 26.6 percent (up 15.3 percent), and the Greens at 6.4 percent. Although this result is close to voter loyalties reported in a March 25-27 Ipsos-Reid poll (28/38/18/12), except for a drift of Green voters toward the NDP, poll results reported during the following three months indicated a highly volatile electorate. Most poll- sters observed the late campaign swing from the Liberals to the Conservatives and the NDP.
Compas, SES, Ekos, and Ipsos-Reid-Reid (BC poll) all projected the final BC results accurately within a 3 percent margin of error for the three major parties. Two of Compas’ last week projections (together with its prediction of a 7 point Conservative margin over the Liberals) were accu- rate within one percentage point. Interestingly, all four major polling firms which published regular polls during the election projected the combined NDP/Green vote at 34 per- cent ”” compared with the actual out- come of 33 percent.
The ”œdistinctness” of Quebec society extends not only to its polarized federal electoral contests, pitting the Bloc Québécois as the dominant party of Quebec francoph- ones since 1993 against the Liberals as the principal opposition, but also to its polling dynamics. Veteran aca- demic Maurice Pinard noted years ago that the ranks of ”œundecided” voters tend to disguise a dispropor- tionate number of Liberal votes on election day. This phenomenon was on display once again in the 2004 federal election.
Our newsletter about the public service.
Nominated for a Digital Publishing Award.
Despite initial fallout from the Liberal sponsorship scandal in Que- bec, the Liberals appeared to be gaining ground on the Bloc in the days before the election call on May 23. However, the consensus of the polls by mid-campaign was that the Bloc had opened a 20-point lead, and that Gilles Duceppe performed more effectively than Paul Martin in both the French and English lan- guage leaders’ debates. As a result, out-of-province pollsters placed Lib- eral support in the range of 21 to 23 percent, compared to 50 to 58 per- cent for the Bloc during the week following the debate. Quebec-based Leger Marketing discerned a much closer race (48 to 30).
Polls in the last week of the elec- tion indicated some shift of support toward the Liberals, and a high per- centage of undecided voters ”” with Leger Marketing and Ipsos-Reid catching the public mood most pre- cisely (see table 5). However, other published polls significantly under- estimated the Liberal vote and over- estimated the vote of the other parties so that only 60 percent of vote projections for individual par- ties in Quebec during the last week of the campaign were within three percentage points of the actual out- come.
In Atlantic Canada, pre-election polls between late April and mid- May 2004 forecast a Liberal lead averaging 22 percentage points over the Conservatives, and an even larg- er lead over the NDP. However, small polling samples (ranging from 60 to 90 for most polls released during the campaign) contributed to high lev- els of volatility and high margins of error ”” making it difficult to deter- mine the extent to which Atlantic Canada was bucking the national trend.
Most polls published during the last week of the campaign projected major NDP gains, a slight growth in the Liberal vote, and a sharp decline in the combined PC/Alliance vote (see table 6). Election night results confirmed this trend with the Liberals gaining an average of 3 percent of the popular vote across Atlantic Canada, the NDP gaining 6 percent, and the Conservatives drop- ping 10.5 percent from their prede- cessors’ combined vote. Most polling projections caught the broad direc- tion of these trends, while tending to exaggerate them somewhat.
Manitoba and Saskatchewan are lumped together for polling purposes more for the sake of conven- ience, because of their individually small sample sizes, than for the simi- larity of their electoral markets. More than half of Manitoba’s votes are cast in Winnipeg, while electoral map- makers have designed ”œpie-shaped” ridings in both Regina and Saskatoon which result in large rural votes in seven of the eight notionally urban ridings. As in its provincial politics, rural Manitoba is divided between a largely conservative south and a large- ly NDP north. The resulting fragmen- tation of political outlooks within a small market ensures that pollsters’ typical sample sizes are far too small to inspire much confidence. Margins of error for federal election polls in ”œMan./Sask.” are usually in the range of 10 percent.
The effect of this electoral dynam- ic was particularly visible in 2004, as noted in table 7. The Conservatives won 13 of 14 Saskatchewan seats with 41.8 percent of the vote. The new party also entrenched its support in rural southern Manitoba while divid- ing Winnipeg’s seats with the Liberals and New Democrats (2/3/3). One poll- ster, Quebec’s Leger Marketing, accu- rately projected the Conservative and Liberal votes in the region.
Two of the four pollsters (Com- pas/Ipsos-Reid) were close in their estimates of the eventual Liberal vote. Two (SES/Ipsos-Reid) were close in their projection of the NDP vote. Compas consistently overesti- mated the Conservative vote, with the others failing to catch the extent or durability of the party’s support in the region. However, the inherent limitations of polling in such a small, diverse, and sparsely populat- ed market were clearly on display.
Projecting the outcomes of elec- tions in Alberta rarely requires great psephological skill. Baseline polls in late March projected a wide Conservative lead over the Liberals, which gradually increased during the course of the campaign. Turnout dif- ferentials do not appear to have made a significant difference in election results, with only a marginal difference between turnout in Edmonton (60.6 percent), Calgary (59.4 percent), and the province as a whole (59.4 percent), according to preliminary Elections Canada reports.
Three of the four major polling firms (Compas/Ipsos-Reid/SES) pro- jected the final Alberta results accu- rately within 3 percentage points for at least two of the three major parties ”” although most forecast a higher NDP vote than actually materialized.
Did the pollsters ”œmiss” the outcome of the 2004 federal election? Technically, no. Most results were within their statistical margins of error, even though few pollsters or their media clients bothered to qualify their regional data in this way in their published reports. Honourable men- tion to SES Research, which regularly did so in its daily tracking poll and weekly regional breakouts for CPAC.
It is often as misleading and unfair to talk generically of ”œthe pollsters” as it is of ”œpoliticians.” Breaking down the accuracy of final projections by party and region of national polls published during the final week of the campaign, Leger Marketing (distributed through Quebecor/Sun Media) provided the most accurate snapshot of regional vote totals, followed by Compas (National Post/Canwest Global) and SES Research (CPAC). Trailing were Ipsos-Reid (Globe and Mail/CTV) and Ekos Research (Toronto Star/LaPresse). Table 9 summarizes the percent- age of regional party votes accurately projected within one, two and three percentage points.
Media outlets tend to depend heavily on their proprietary polls, rarely bothering to compare polling data to place results in a broader con- text or to offset the effects of small regional sample sizes. Some pollsters were more transparent. For example, Ekos and Compas went to some lengths to test the stability of voter intentions and publish their findings, noting the degree to which poll respondents indicated the likelihood of changing their vote intentions before election day.
One way of offsetting the effects of small regional poll samples would be to conduct a ”œpoll of polls,” a tech- nique adopted by the British media for many years and used by Wilfrid Laurier political scientist Barry Kay in his regional seat projection model. Kay’s model, superimposed on surveys conducted during the final 10 days of the campaign, accurately predicted regional seat outcomes (plus or minus one seat) in all regions except Quebec and Ontario.
The most accurate projections from Kay’s aggregation of polling data come in regions where individual polling samples are too small to have more than minimal statistical validity (see table 10). However, aggregating polling data would not have allowed the media to forecast the Liberal land- slide in Ontario or the modest Liberal recovery in Quebec, although the lat- ter appears to have become a structur- al feature of pre-election polling in that province.
Pollsters (and their media clients) could also enhance truth in polling by disclosing both the size of regional samples in their published materials (a regular feature of reports published on Ipsos-Reid and SES Research Web sites), and the regional margin of error for regional surveys.
On balance, published opinion polls remain useful ways for the vot- ing public to observe political trends during elections to obtain some con- text for evaluating the organized manipulation inherent in major party campaigns. However, with an increasingly disengaged electorate, much of which is reluctant to answer pollsters’ phone calls, the best advice is still caveat lector ”” don’t believe everything you read.