H+K VP and national director of campaign strategy Stephen Carter contributed the following blog post to the Calgary Herald, originally published on June 2, 2014.
The race for PC leader has started. And, true to form, a poll has been released telling us all how the race will end. And the Calgary Herald dutifully ran the results today.
Polls are intoxicating. We love the idea of knowing the future. No need to wait for Sept. 6, we all know that Jim Prentice will defeat Thomas Lukaszuk and Ric McIver. It says so right there. Science has forecast the outcome.
Except that generally isn’t what happens. Polls, especially this far away from an election or leadership, are about as right as a 14-day weather forecast. They can be right, but that is more likely due to good luck than good forecasting.
Why? Well, here comes some science stuff that is important to know when reading polling stories.
Let’s start with selection bias. Selection bias is when the people selected to participate in the poll do not represent the overall population in general. We see reported that 1,470 Albertans participated in the poll. But who are these 1,470 people? How were they selected?
We need a random process to select the sample to ensure the poll is accurate. But participating in a panel is far from random. This isn’t to say that all online polls are inaccurate, they aren’t. But you have to be aware of the selection bias when you look at the results.
Which is why online panels aren’t supposed to include a margin of error. Sadly, this article includes a margin of error, implying the statistical validity of the poll. Oops.
The second selection bias this poll suffers from is regarding the nature of leadership races. Only members can vote in the PC leadership race. How many members were polled? Well, that’s not disclosed. I’d bet not very many.
Therefore, the poll uses a sample that likely doesn’t even represent the general population to suggest an outcome of a race that will be decided by a very select subset of the general population. Let’s call that an error multiplied by another error.
All that before we get to the second problem with the article: the desire by pollsters and journalists to project polling results forward. Polls are not crystal balls. In fact, they represent a past view. The leadership is conducted on Sept. 6. Three long months from now.
No one knows how the leadership vote will end. And we are certainly none the wiser for reading James Wood’s article about this poll.