Be wary of policy-makers bearing evidence
The use of evidence in policy-making is subject to systematic bias, argues Professor Alex Stevens, who observed behind the scenes in a Whitehall government department
How do civil servants use evidence in their everyday work as they create policies? That’s what I set out to understand when I was seconded to a Whitehall department for six months. My findings, I should warn, were a bit dispiriting. There is a systematic bias in both the kind and content of evidence that is used. I observed policy-making being distorted by the filtering of evidence to fit the particular goals of the powerful. The result? Policy-making that may be supported in public by evidence but certainly has not been determined by it.
I found that civil servants have a wide variety of evidence available to them. There is too much of it to take in to all the policies they are required to write. So they develop rules of thumb by which to choose which bits of evidence to use. The civil servants I worked with were chiefly concerned with what evidence could be applied in the making of a usable policy document. So civil servants favoured certain forms of evidence over others – they considered quantitative to be more persuasive than qualitative. But there was also a bias towards using evidence that supported the existing policy beliefs of the people who had the power to accept or reject the policies that the civil servants put forward. Crudely put, we might say – as journalists sometimes joke – the facts are not allowed to get in the way of a good story.
For example, I was involved in a meeting between civil servants working on penal policy. There was a very interesting and informed discussion about the relative benefits of spending money on imprisonment. The conclusion of this discussion was that investment in prison was not a cost effective way to reduce reoffending. However, the civil servants ruefully concluded that this would not be the right headline to put in the policy document that was supposed to emerge from the meeting. Why? Because it did not fit the Government line of the day.
Bias in ‘narrative policy making’
This is example of what I have called ‘narrative policy making’ in which civil servants pay very close attention to the narrative they are presenting in their policy documents and to ensuring that the narrative is congruent with the expressed wishes of specialist advisors and ministers.
This approach reveals deep bias. First, because people are fitting the evidence to preconceived notions of how they think the world works. Secondly, it is biased because it supports rather than challenges inequality. This goes against the supposed ethos of evidence-based policy. Social scientists – perhaps naively – hope that increasing the role of evidence in policy will support democracy because evidence is perceived as being politically neutral. But the people who are making the choices about what evidence will be translated into policy come from the groups in society that already have unequal access to resources, money and power. So the process I have observed of ‘narrative filtering’ of the evidence tends to preserve existing inequalities and power structures rather than challenging them.
You can see how this works when looking at drugs policy. We could ask, for example, why the possession of alcohol and tobacco is legal whereas the possession of cannabis and ecstasy is illegal. This distinction is strange, since we know from recent controversies, including the sacking of Professor David Nutt, that there is not a scientific consensus that cannabis and ecstasy are more harmful than alcohol and tobacco. Yet, despite such uncertainty, the users of ecstasy and cannabis face greater punishment than the users of alcohol and tobacco. This is an example of an inequality that is supported by the narrative filtering of evidence. In this field, as in others, the use of evidence in policy making is characterised by systemically distorted communication. We do not see the rational exchange of views that we might expect. The communication of evidence is used strategically to bolster positions of power, rather than openly to improve policy.
Missed opportunities to experiment
To see what we have lost in process, it is worth going back to one of the founding documents of evidence-based policy making by Donald Campbell. ‘Reforms as Experiments’, a paper Campbell published in 1969, argues that policy should be considered a great opportunity for experimentation. So, by failing to pursue rigorous evidence-based policy, in for example drugs and crime policy, civil servants are missing opportunities to create better knowledge for policy.
Sadly, practice indicates that Campbell’s optimism has also proved to be misplaced – it naively relies on politicians using evidence as though they are academic peer reviewers. We have placed too much faith in the concept that research can solve social problems. Such problems are, in fact, always laden with values and moral choices, not just soluble questions about numbers and outcomes. We’re fooling ourselves if we think otherwise.
Flaws in ‘Transforming Rehabilitation’ policy
The development of new policy for managing ex-offenders is now showing many of the problems caused by failing to be honest about the role of evidence. For a start, the case for transforming probation services in the first place was flawed because it was based on high re-offending rates of prisoners sentenced to less than 12 months in jail. This group is not subject to post-release management by probation services – so their higher rates of re-offending can hardly be blamed on the probation services.
Nevertheless, reoffending by this group underpins a massive untested privatisation of probation. The mechanism being chosen is Payment By Results (PBR). This was being tested by National Offender Management Service (NOMS). However, the pilots were truncated in order to allow the national roll-out of the privatisation programme – the ‘Transforming Rehabilitation’ programme. Like a number of other policies in the criminal justice field, pilots were scaled up before lesson could be learned from them. Civil servants might contend that the pilots were testing the implementation process rather than the long-term success of the pilots. Nevertheless, there is understandable suspicion that pilots were really designed just to soften people up to the idea of privatised probation services.
A related example of flawed ‘evidence-based policy making’ is the Justice Data Lab. It is a Ministry of Justice initiative to give organisations working with offenders access to data on the reoffending of their clients, compared to a matched sample of other offenders.
It sounds like a good idea. But the process for matching is inevitably imperfect. In practice, organisations that have less complex case loads will tend to show good results and organisations with more complex case loads are likely to show poorer results. So, for example, organisations working with homeless people and those with drug and alcohol problems won’t come out looking successful. This failure really matters. There are already incentives to ‘cherry pick’ ex-offenders in PBR and the approach of the Justice Data Lab will make that worse.
What happens if we add all this together? We get poorer policy-making. The overall narrative for Payment By Results and the Justice Data Lab is very persuasive. It is seems very attractive for the Government to pay only when good outcomes are achieved and to give organisations the ability to track those outcomes for their own clients. But these persuasive narratives have taken precedence over finding evidence of whether PBR or the Justice Data Lab will, in practice, have the intended outcomes. So what seems like a smart piece of policy-making might well turn out to be a rather foolish decision, based on flimsy evidence.
How to improve evidence-based policy-making
What can we do to improve ‘evidence-based’ policy making? For a start, we should be increasing accountability for poor decision-making. At the moment ministers and policy-making civil servants tend to move around departments quite quickly. So it is very difficult to make them accountable for any problems with the policies that they have designed.
The Institute for Government has come up with a possible solution – an accountable officer within each department, tasked with telling ministers if they are traducing or ignoring the evidence. I am rather sceptical of this working well because of the power relationships involved. I would not bank on the career prospects of those officers if they stuck well to that brief – just look at what tends to happen to whistleblowers.
A more sustainable solution would rely on developing methods and opportunities for the wider public to be involved and put pressure on politicians when they misuse the evidence. This approach would push academics such as myself into more engagement with the public, more open access publishing and providing more research publications that are useful in making policy and which are available and comprehensive to the public at large. This approach needs academics to do more to inform public discussion of policy.
There are examples in practice. We have recently seen push back on some of welfare policies thanks to informed critics providing quick and useful data-driven analysis of social security policy. We need such bodies to challenge social policy, organisations that are as independent and intellectually robust as, for example, the Institute for Fiscal Studies has become with respect to economic policy.
Fundamentally, we need to recognise that the choice of evidence used in policy-making is likely to be a reflection of who has the power to make the decisions. If power is not democratised or equally distributed then we should not expect the use of evidence to be even-handed. The wise person should be wary of policy-makers bearing evidence.
Alex Stevens is Professor in Criminal Justice and Deputy Head of the University of Kent’s School of Social Policy, Sociology and Social Research. He has worked on issues of drugs, crime and health in the voluntary sector, as an academic researcher and as an adviser to the UK government.
Professor Stevens will present a special SPHR@L Seminar at the London School of Hygiene & Tropical Medicine on Tuesday, 19th Nov 2013, 12.45-2pm: LSHTM, Room Jerry Morris A, 15-17 Tavistock Place, London WC1H 9SH. The title is: ‘Telling Policy Stories: An Ethnographic Study of the Use of Evidence in Policy-making in the UK.’