Past Vote, Pollster Weights, And The Untold Story Of The Election

Scrimshaw’s Guide To Polling Wonkery

(This is going to be a wonky piece and there’s no real way around it. I’ll do my best to make this clear but you’ve been warned.)

One of the bigger questions in the polling space right now is why some pollsters have better results than others. Abacus still has a tie and Innovative Research Group has a C+1, while the untested Kolosowski has a 2 point lead nationally. Against that backdrop, every other pollster has Liberal leads, ranging from the small (Mainstreet) to double digits (today’s Ipsos). But it’s mostly not a disagreement on the Conservatives.

The three pollsters I mentioned first all weigh by past vote - ensuring their samples, or at least the vast majority of their samples who voted in 2021, look like the past election result - while the rest don’t. Last night I tweeted a list of Conservative vote share by whether the pollster does or doesn’t weigh by past vote, and there’s not a huge relationship there. (That list looks even less causal if you correct for Liaison and Nanos getting much better for the CPC in today’s tracking, even as Ipsos slips a bit.) Where you do see a relationship is with the NDP.

In all three of the WPV (Weighing by Past Vote) pollsters is an NDP vote at 11% or 12%. Here’s where they are in the others: 7% (Liaison), 8% (Nanos), 8% (Mainstreet), 7% (EKOS), 10% (Ipsos), 8.5% (Pallas), 7% (Leger), 7% (Angus Reid). This is about method in a lot of ways, because this looks pretty clearly like the difference in margin (in aggregate) isn’t about where the Conservative vote is, but where the NDP vote is, and therefore how much the Liberals can steal from them. So, who’s right?

Weighing by past vote is a certainly defensible choice, but that doesn’t mean it is inherently the correct one. In America, pollsters who weighed by past vote were more accurate, but it’s not a panacea. In 2020, Nate Cohn would tweet out the underlying metrics of his polls as a teaser sometimes, and it would always involve the sample’s recalled 2016 vote being multiple points to the right of the 2016 result in that state. Then the poll would come out and Trump would be suffering a 10% swing against him from 2016, which even an airquotes “more friendly” recalled past vote couldn’t change. (The NYT/Siena polls don’t weigh for past vote, but they include it in their releases as a measure of transparency.)

The other recent election where this was a running talking point was the UK 2019, where Labour supporters found polling samples were too low on Labour recalled voters, and this was somehow proof that the polls would miss low on Corbyn. This is instructive of the case for believing the WPV polls in Canada are wrong - in both cases, events had transpired to make the party in question (Labour and the NDP) much less appealing to voters. And that’s a problem when you deal with past vote.

Either through errors in memory or through a refusal to admit (either to yourself or to someone else) that you voted for someone you now hate, recalled vote can be wildly unreliable. Again, in 2020, weighing by past vote would have made the polling error worse for the NYT/Siena polls, not better. And in 2019, what clearly happened was some number of Corbyn voters in 2017 couldn’t admit they voted for him - either Brexit supporters who couldn’t believe they voted for someone who by then was a Remainer, Jewish voters and their allies appalled by the anti-semitism issues, and various others. Had you weighed to past vote and actually gotten the Labour past vote to 40% (or 40% of voters who voted in 2017), Labour would have been artificially high in the polls and underperformed.

The reason this matters in Canada is that the composition of which kind of 2021 New Democrats you get matters. Let’s take two voters, both of whom actually voted NDP in 2021, but one of whom claims they didn’t and one who did. I have no proof for this, but it passes a basic smell test, that past NDP voters who remember voting NDP and admit to are more likely to be NDP loyalists than wavering progressives who know they voted Trudeau at some point and are enthusiastically voting for Carney. It’s a well established pattern that people remember voting for the winner when they didn’t, as those US 2020 examples show. But, in cases of subsequent disaster, people remember their history differently - remember how there were all those polls of Americans claiming they always opposed the Iraq War, when we know from the contemporaneous polls they didn’t?

My strong theory is that some number of Carney voters think they voted Liberal in 2021 because they’re pissed as hell at Jagmeet and forgetting that prior to 2021 they kind of liked the guy, which means if you weigh your sample to get 18% of New Democrats you’re actually overweight on 2021 New Democrats in reality, because you’ve got the 18% who admit to being New Democrats and then some number of 2021 New Democrats who claim to have voted Liberal. At the very least, you could be overweight on loyal New Democrats.

Now, they could be overweight on loyal New Democrats and also be right in the end - it could be because of the intensity of hatred for Singh that many have, the 2021 NDP voters you’re getting (both who remember voting NDP and Liberal) are disproportionately likely to swing. There could be a loudest voices in the noise phenomenon here, where the polls that don’t weigh by past vote are getting people who are very excited by Carney (and in some places Poilievre) and more eager to answer polls than sad, mostly beaten down Singh voters.

The thing about polling is that methods decisions often look brilliant in hindsight even without looking so at the time. My instinct is that the NDP sample they’re getting is overly loyal to the party because of weighing by past vote, and so the numbers are probably in the high single digits and not low double digits. That said, it is undeniable that I’m a Liberal supporter in this election and it would be better for my party if that’s true. But this feels like a Corbyn-era question to my eye - an issue where a lot of people are regretful of their choice to vote for a party and a leader and are claiming they voted for another party. If that’s the case, then overweighting the smaller pool of voters who still remember voting NDP to compensate will mess up the sample, and imply a higher NDP floor than is real.

It’s also theoretically possible that there is some sort of “shy NDP voter” thing happening, where progressives who are wary of Carney but in very pro-Carney circles feel they can’t say something. I doubt it, but let’s be real - I can confidently say based on data that the reason the polls differ because of the handling of the NDP, but beyond that it’s theories all the way down. But if I’m going to make fun of others for living on the fence, let’s get off it - if the election were today the NDP wouldn’t break 10%.

Maybe Jagmeet aces the English debate, everything is just fine, and Leger, Abacus, Mainstreet, and IRG all show a 14% NDP vote in their final poll. Maybe he tanks the debates and everybody coalesces on a 7%. But right now, the NDP’s vote, and whether you weigh by past vote or not, is the distinction that distinguishes a close race from a comfortable LPC majority. And which side of that debate gets that decision right will likely be the ones coming up roses on April 28th.