Mind the gap! Why companies are polls apart


  By Ivor Knox 
IN OUR regular Sunday Times referendum polls, Panelbase has generally shown a No lead of about 10%-12% (excluding undecided), which is a smaller gap than most of the surveys conducted by other agencies over the past 18 months or so.

Political polling is a mixture of art and science, and the construction of an accurate snapshot of public opinion at a given moment in time includes several elements which can influence the results, such as methodology (online, phone, face to face), sample source, and question wording/order.

Despite variations in these factors, there are often plenty of similarities in the raw results of polls (for example, links between party affiliation and independence voting intention), but data analysis can explain a good deal of the variance between polling organisations.

It is common practice for polls to use quotas and/or post-fieldwork weighting to create a sample which properly represents the population.  Panelbase applies weights on age, sex, socioeconomic grade and household tenure, and other companies implement a variety of similar measures.

For our Scottish polls, we weight the results to match the 2011 Holyrood election, filtered to report only people who are very likely to vote. 

Among the undecideds 59% say they would vote Yes if they and had to choose, while 34% would vote No.

To demonstrate an example of the potential effect of analysis on outcomes, we reran our tables for this poll with the results weighted to match the Scottish vote in the 2010 Westminster election instead.

Still filtered on likelihood to vote, this produced a 63%-37% lead for No – more than double the 12-point gap of our Holyrood weighting approach.

While there is logic in weighting Holyrood voting intention questions to 2011 and Westminster voting intention to 2010, there is a debate to be had about which is more likely to produce an accurate result for the referendum question.

For example, it’s a Scottish issue but also a UK constitutional one.  We are sticking with our current model for two main reasons.  First, while we accept that Westminster or plain demographic weighting are perfectly reasonable, we still feel that weighting to Holyrood is also valid, and our respondent recall of that more recent election seems to be closer to reality.

Second, for consistency — trends are more important than individual polls.  If all the polls (or the average of all the polls) with their various methods continue showing No holding a fairly consistent lead (including ours, albeit with a narrower gap), then it would seem highly probable that No would win, if the referendum were to be held now. 

If they all started moving in one direction or the other, it would be a fair indication that genuine shifts in attitudes were afoot and that one of the campaigns may be changing people’s minds.

(c) Ivor Knox – Ivor Knox is managing director of Panelbase
This article first appeared in The Sunday Times Scotland