Opposition justice spokesperson Graeme Pearson MSP published his review of policing in Scotland last week, in response to a succession of scandals involving the unified Police Scotland. As a former Deputy Chief Constable of Strathclyde Police and then Director General of the Scottish Crime and Drug Enforcement Agency, the Labour MSP supported that unification, but has since criticised its management. Here, Professor John Robertson challenges the assumptions behind Pearson’s report.
The Pearson report is based on unclear data collection methods, a complete lack of literature review to inform the design or the analysis of results and a credibility-undermining tendency to make regular unjustified value judgements, based on prejudgement rather than uncovered evidence, make this review utterly unreliable as evidence to be used in government policy adjustments.
Here is the background:
‘Announced on 31st August (2015), the aim of the review was to assess the effectiveness of the single police force and propose reforms to restore local accountability and reverse the impact of damaging cuts to staff and services.’ (Pearson Review: 3)
The Pearson Report took less than three months from announcement to final report. This is astonishingly quick and suggests the risk of it having been rushed to meet a politically-driven timetable. A typical undergraduate dissertation takes roughly twelve months from drafting the title and questions at the end of Year 3 to final submission in May of the following year. More accurately and more modestly, this quick and quite biased commentary should be presented as ‘commentary’ or ‘opinion’. The term ‘review’ implies method of which little has been made clear here.
The review does not use the word ‘research’ anywhere. The authors were well-advised to avoid it yet they seem to think it is research-like, has the qualities commonly associated with research – rigour, objectivity, clarity, transparency – and put it forward explicitly as useful for government policy–makers.
The presentation of the document using a high-status lead-author and Honorary Professor, is presumably intended to invoke or claim by association all of the same qualities – rigour, objectivity, clarity, transparency – one would expect in valid, reliable, research. Neither ‘valid’ nor ‘reliable’ make an appearance, sensibly, though ‘unreliable’ gets one mention.
The Pearson Review has featured heavily in broadcast and print news bulletins but it has not been subject to any critical analysis whatsoever, until now. Readers may remember the extensive critique of my 2014 study of Referendum Campaign coverage by TV news and the still-maintained view that my methodology was flawed. Professor Pearson’ has cleverly dodged that arrow by simply not appearing to have any methodology at all!
Here, relatively briefly are the flaws in the Pearson Review that make it unusable as evidence:
- Self-fulfilling prophecy syndrome – The researchers clearly ‘knew’ beforehand what they expected to find and, unsurprisingly, found it. Perhaps this is explained by the fact that the Labour Party in Scotland funded the review and that Professor Pearson is a Labour MSP and former Deputy Chief Constable but it instantly casts doubt on the impartiality of the work. Perhaps the most obvious evidence of this criticism is the early identification (page 3) of four aspects as central to the review. What is not clear is why they have been chosen. To be of worth, such a decision would require a pilot study or, using a grounded theory methodology, see their emergence from objectively gathered and read data provided by a representative sample of police staff and those whom their performance affects.
- Lack of a background literature review – the researchers provide no evidence from research into comparable developments of police centralisation in other countries. Is it, for example, common or rare for countries of Scotland’s approximate size/population/crime level profile to have a single police force? Where comparable centralisation processes have taken place in other countries, how long is it appropriate to allow for bedding in before launching a new review?
- Unclear methodology –There is less than one page of description of the methods adopted, on page 6. There were no research questions posed, yet 10 recommendations emerge, later. How answers can appear when questions directed toward them are not posed suggests that the researchers worked backward from what they hoped to deliver in a political context (Professor Pearson’s pre-existing knowledge) and then picked out examples of data to justify them. We are left unsure as to how many people attended the meetings, how these were organised, what data was gathered and how it was used to extract findings. Most worrying, in the absence of reassurance to the contrary and in the light of 1 above, we are left with a nagging concern that these focus groups were advertised and run so as to attract primarily critical comment. The data from the seven public meetings held by Pearson are, I suggest, likely to be entirely valueless as evidence of anything at all. The meetings with six organisations, likewise, are not explained in terms of sampling or methods used. Why these organisations and not others? Are they a representative sample? They seem, nearly all, to be rather close to trades unions or equivalents or to local authorities. Isn’t it obvious that such groups will find change of this nature problematic and wish to preserve previous arrangements, regardless of any objective assessment of the need for change? How were the interviews conducted – structured, softly-structured, un-structured? You could make a case for any of these as long as you make it clear in writing and justify it in terms of the research questions, if there were any of these, of course.
- Unreliable evidence extrapolation – in each of the four review submissions we see quotes which point to problems in policing which might be evidence of a wider concern but which are not evidenced as such. We only see a handful (8?) of quotes to exemplify the criticism discussed below. How many such quotes could have been chosen given space? We are not told. In the text following the attention-grabbing leader quotes, we find a confusing and frankly valueless list of what:
- One contributor stated
- Many contributors made
- Through many points was the perception
- Another issue raising concerns with (unspecified number) operational officers
- A private company had to uplift
- Several officers reported
What we never hear is ‘how many such comments’ or ‘out of how many in total’.
There are nine pages of these anonymous examples of the feelings of this sample from around 23 000 people working in Police Scotland. How big was the sample? It could be as low as 0.001% unless the open meetings were very well-attended. There is no indication, I can find, of how many attended the meetings, how many spoke or what they predominantly were saying
5. Clear bias – ‘The Scottish Government has failed’ (pages 13, 14, 15), is an allegation echoed throughout what should be a discussion of gathered data and not an excuse to make unsubstantiated allegations. On page 13 it is even suggested that the Scottish Government may have ‘wilfully ignored the importance of relationships’. ‘Wilfully ignored?’ What kind of language is that? For all the flaws in the Pearson Review, I have not accused the authors of wilful behaviour.
6. Piggy-backing bigger research – The Pearson Review, having done no extensive data collection itself, attempts to piggy-back the more substantial SPA/Police Scotland Staff Survey 2015. 50% of around 23 000 staff responded and the findings suggested they were often not happy. However, if you ask staff in any major institution how content they are, be ready for bad news. Look at the staff satisfaction studies done in recent years with GPs, nurses, teachers or firefighters and you’ll find very similar results. Further, look at the numbers. They’re not really that bad compared to some other surveys I’ve seen. Pearson makes extended use of these numbers but they cannot tell us much about the recent centralisation of policing.
So, overall, a surprisingly amateurish report which should have received fuller critique from the Scottish media.
Professor John Robertson, 11.11.15
Note: As of October 2015, I am a member of the Scottish National Party
