By Mark McNaught
Since my last column, I have become increasingly curious about the methodology employed by polling firms to determine whether or not some of them are simply made up, and if there was an institution charged with assuring their methodological integrity.
My web search brought me to the British Polling Council; the trade group for UK public opinion research firms. According to the Guardian, the BPC was founded through a Westminster cross-party motion to delegitimize ‘unscientific cowboy’ polls.
Its rules require members to disclose their sampling methods, the procedures used to assess raw data, and the full wording of their questions and answers. What it did not include is any requirement that the raw data exist in the first place.
Polling firms must pay to become members, ranging from £250 to £750 depending on the size of the company, after which they pay an annual fee based on an ‘honour system’, or they will be expelled. It handles complaints against individual firms for the breach of their rules, but firms cannot leave the organization while being investigated. The BPC does not evaluate the methods employed for arriving at their conclusions, potentially allowing polls to be made up without fear of sanction.
Among the other ‘Benefits of Membership’:
“The purpose of the British Polling Council (BPC) is to assist journalists, and through them, the general public and politicians, in the understanding, interpreting and reporting of polls. […] Journalists are actively encouraged to query the organisation to better understand public opinion industry practices and to keep better informed on trends and issues within the field.”
Their ‘Officers serving on the Sub-Committee on Disclosure’ include many representatives from well-known media outlets and public opinion research organizations.
Drawing from their list of ‘Members [Company representatives]’, we observe how polling firms have treated the question of Scottish independence.
The Angus Reid Canadian Public Opinion survey commissioned by the Sunday Express, which has been cited by the Scotsman, found that only 34% of Scots favour independence. According to the methodology, from August 14-16, 549 randomly selected Scots took an online survey. Whether they polled Canadians, actual Scots, or Scottish Canadians is unclear.
In 2011, Comres was commissioned by BBC4 to conduct a survey which showed only 36% felt that Scotland should become a fully independent country. The main methodological problem is that they only asked the English.
This ICM poll commissioned by the Scotsman showed 40% in favour of independence, and 60% opposed, in which 1002 adults allegedly participated. The question was “As you may know, a referendum on independence will be held in Scotland on 18th September 2014. Voters will be asked, “Should Scotland be an independent country”. Do you think you will vote “Yes” or “No”?” Exactly who participates in these online surveys, and how does ICM know they are not getting a response from someone in Uzbekistan?
IPSOS-MORI has been commissioned by STV to conduct 6 telephone polls over the next year in the run-up to the referendum. Their most recent poll shows 30% Yes, 57% No, and 14% undecided. Allegedly, exactly 1000 Scots were telephoned between the 9-15 September 2013 to determine this result.
As one looks over the statistics including gender, age, employment status, home ownership, area of affluence, urban/rural, national identity, etc. – one wonders how they obtain all this information? Did they ask all of these questions on the phone before asking their opinion on Scottish independence? Did they already have the profile before calling? How do we know that they had a legitimate representative sample? Did they call on mobile phones or landlines? Knowing more about their methodology would help determine whether their findings are valid.
‘Lord’ Ashcroft is believed to have commissioned ORB International to conduct a ‘super poll’ of more than 12,000 Scottish voters. Since he is unwilling even to disclose who supposedly conducted the poll, there is no way to determine the integrity and accuracy of the results.
The crowd-funded Panelbase polls commissioned by Wings Over Scotland seem to have a high degree of methodological validity. The most recent shows 43% voting ‘no’ and 35% voting ‘yes’ which clashes with an SNP commisisoned poll which gave Yes a one point lead. I would be curious to see the differences in methodology which could help explain this discrepancy.
The Mail on Sunday commissioned Survation to conduct a poll of an ‘online consumer panel’, asking 1001 from Scotland and 1019 from England and Wales about their views on Scottish independence. If the ‘consumers’ refer to Daily Mail readers, how did they find 1001 in Scotland?
YouGov is an interesting case. It employs innovative methodology, and features forums on Scottish independence, with arguments on both sides. I also found a comprehensive poll conducted by youGov/Channel4, in which they don’t explain their sampling method.
Another youGov poll cited by the Scotsman shows the ‘no’ vote ‘hardening’ at 59%, with a similar lack of information concerning the source of their empirical data. YouGov/Devoplus shows Scots supporting enhanced powers if they remain in the UK, but it is difficult to discern exactly how this poll was conducted.
While some of the methodological questions raised may have valid answers, it is incumbent on the polling firms to provide all the information necessary about how they reached their conclusions so that citizens can determine for themselves whether they are made up or not. It is not enough to provide a spread sheet with the results and the questions, because there is nothing to verify that they actually sampled the opinions of anyone.
Scots have been told by the BBC, the Scotsman, the Herald, and many other sources that they don’t support independence. This raises serious questions not only about actual levels of support for Scottish independence, but also the duplicity of UK poll-driven politics.
So many results are bandied about in parliamentary debate and in the media, many of which could have been entirely fabricated. How can the UK function as a democracy when the information upon which electoral decisions is manipulated to brainwash the masses?
It also raises the intriguing possibility of developing a more empirically-driven politics in an independent Scotland. According to a poll I read somewhere, exactly 78.2% of Scots are in favour of that.