by mark petticrew
Interventions that succeed in some instances may or may not work in other circumstances. You have to consider whether the contextual differences really are ‘significant’, says Mark Petticrew
How important is the particular context of a policy intervention in deciding whether that intervention can work elsewhere? The answer must lie in the significance of the context. Every place is different. Every time is different. Everybody is different. The important question must be: which differences really matter, which are actually significant? We should avoid mistakenly thinking that the inherent uniqueness of everything means that a particular intervention will never work elsewhere. It might still be generalisable and transferable elsewhere.
Similarity and uniqueness
It is, of course, highly implausible that interventions work the same way across different contexts. Nevertheless, it is equally implausible that that evidence collected in one context has no value for another. These polar positions are unhelpful because neither is true. (‘We are all individuals,’ shouted the crowd to Brian in the Monty Python movie. ‘I’m not,’ said a lone dissenter). Clearly, all individual study contexts are different, but there may be similarities.
Similarity and portability across apparently very different contexts were aptly illustrated to me when I was involved in housing research. The earliest controlled trial of a housing improvement intervention was done in Stockton on Tees in 1929. Families were moved out of the slums, which were then demolished, and moved into new housing. Unexpectedly, many people’s health deteriorated.
This type of intervention is common today. Urban improvement accompanied by large-scale housing regeneration occurs frequently. However, the context is very different from 1929. In those days, poverty was probably more widespread, as was slum housing. Yet, more recently the same unanticipated adverse effect has been found in one study, with a minority of people’s health deteriorating when their housing improves. Although the context looks very different, the underlying mechanisms seem to be the same, namely that, when the housing is improved, rents rise and so people scrimp on their diets and their health gets worse.
Another field where the same mechanisms apparently work across different contexts is smoke-free legislation which aims to restrict the impact of second-hand smoke in work and public places. This has been evaluated at least 11 times in very different contexts. When the issue reached the UK, critics, often in the hospitality industry, said this might have worked in these other countries but it wasn’t going to work in pubs in Glasgow, say, or in London. The same arguments were raised around the implementation of smoke-free legislation in Ireland, that these are very different contexts, that people’s drinking and smoking were wedded. Yet, in fact, the success of implementation has been broadly similar across many different states and countries.
Aspects of context that matter
In short, predicting the generalisability of an intervention is all about understanding the significance of context. So the first step must be to reflect on which aspects of contexts might really matter. A lot of checklists to help this task have been put together. Dr Helen Burchett from the London School of Hygiene and Tropical Medicine has reviewed dozens of these frameworks which are used to help users to judge whether evidence collected in one setting might be applicable in another context. Her study found that there are 19 categories of context that might be important and a few more can probably be added.
Some of the work that we have been doing as part of the NIHR School for Public Health Research has been particularly enlightening around economic contexts. Local practitioners tell us that the current economic climate has been a big constraint not only on the use of evidence by, for example, local government, but also on evaluation itself, which is often seen as a luxury.
However, as I have tried to show, context always varies and simply pointing out the differences is not sufficient. You have to determine – or sometimes make assumptions – about which of these variations actually matter – which are likely to be clinically, or socially significant. How do you do this? This assessment should be informed by at least three considerations. First, there is knowledge of the existing evidence, which helps one discover whether and how the intervention has worked in other settings. Second, understanding the underlying theory and assumptions about how the intervention works and is moderated can be helpful. Finally, one can draw on the judgement of experts, practitioners and policy makers who might have insights into whether one context is significantly different from another.
There is a lot more scope for research in this field. For example, there may be classes of interventions that are less context-dependent than others. Smoke-free legislation with its 11 evaluations would be a case in point, and suggests that perhaps regulatory interventions are less affected by context than interventions that require more individual behavioural change.
Context and interventions intertwined
We may also need to revise our sometimes simplistic view of the relationship between context and intervention. There is a tendency to see context merely as a moderator, something that interferes with an intervention in some way. Yet there are many situations and policies where the intervention is the context. The intervention changes the nature of the system in some way so that the intervention and the context are, in effect, the same thing. This makes defining the start and the end of an intervention and its boundaries – and thinking about how you evaluate it – hugely challenging.
The significance of context in generalisability also places question marks against the culture of systematic reviews. During such reviews, researchers aim to put all the evidence together from interventions and attempt to discern a single effect based on everything that is known about an issue. It is an attempt to separate the ‘things that work’ from the ‘things that don’t work’ and identify an overall effect size. This may be problematic because, during this process, the context that produces that effect usually gets stripped away. As a result, in the process of producing evidence, we lose the context.
As researchers we also have a tendency to see the world in terms of studies of ‘magic bullets’ which tell us that, if things work, then they work everywhere. However, at least in public health, we are increasingly putting together assemblages of evidence from different contexts that show what happened when those interventions were implemented in different places to guide future decision makers. This is very different from saying simply that something always ‘works’. It might be more helpful to see the wider goal of collecting evidence as being to inform decisions, rather than to simply test hypotheses. This may be one way forward to make proper sense of context, rather than trying either to eradicate it or allowing its uniqueness to rule out the possibility that an intervention can be transferred across time and space.
Dr Mark Petticrew is Professor of Public Health Evaluation at the London School of Hygiene and Tropical Medicine and a member of PIRU. He is also a co-director the NIHR School for Public Health Research at LSHTM. (mark.petticrew@lshtm.ac.uk)