By Zaid Chalabi
Mathematical models can test multiple variables cheaply and quickly, giving early indications of what really matters. That helps in designing pilots and understanding how context can affect a policy’s success, argues Dr Zaid Chalabi.
I was part of a team that evaluated the implementation and cost-effectiveness of the government’s Cold Weather Plan (CWP) for England. The CWP is a guidance document which aims to reduce the thousands of additional deaths that typically occur in England when temperatures plummet. The fall in temperature can increase risks for elderly people as well as those with heart or breathing problems and other chronic conditions.
The plan’s principle is that, when cold weather is expected, the authorities are alerted and they can enact various measures, suggested in the CWP. They might, for example, contact or visit vulnerable patients, check that they have medication, that they are warm and have enough food to last a cold snap. The CWP guidance is quite general and it is up to each local authority to implement the plan in its own way.
Assessing the Cold Weather Plan
There is a lot that we do not know in assessing the CWP’s cost-effectiveness. How fully will each health authority implement the plan? Which aspects will they focus on and what impacts are made by each particular action?
Somehow, we need to know how the CWP, implemented in its various ways, might impact on the health of the population and also avoid hospital admissions to save on costs to the NHS. The costs of elective admissions – postponed if there are weather-induced surges in emergency admissions – also must be assessed. We must evaluate the quality of life, and the years of life extended by the CWP, as part of the final cost-effectiveness calculation.
Clearly all of this is hard to evaluate because there are so many variables. Also the circumstances may not occur often: a decade of mild winters might pass before there is a harsh year. Because the CWP has been running for only a few years, we do not have much data, yet it is also a life and death scenario, so policy makers need advice on how best to implement the CWP in order to deploy limited resources well.
Options for evaluators
What should evaluators do? A meaningful analysis of the outcomes of the CWP would require the ability to compare vigilant health authorities with those less engaged, and to evaluate the CWP over several harsh winters. We cannot afford to wait that long, and it may be unwise to stop some authorities making preparations to protect their populations.
Modelling helps when data is missing
We adopted mathematical modelling as the best approach in such circumstances, where lots of important data are not yet available. It is possible to search the literature or seek expert opinion to make estimates for almost every scenario and action. We can estimate, for example, the benefit to patients’ health of being contacted during cold weather. Theoretical costs can also be factored in for the different options – be it a phone call or a more expensive actual visit. These figures can never be absolutely accurate. But, by building in as much data as possible, the model begins to reveal which variables are significant and which ones do not matter much.
Modelling also provides indications of where the extra costs of the CWP might occur (probably in social services) – and which areas are likely to enjoy savings (probably acute hospitals). In health and social care, implementation of cost-effective innovations is often held up because silo working means there are both losers and gainers. Because the losers may not be compensated, the policy may not be implemented, thus depriving the system of a net gain overall. Modelling can identify who wins and who loses, giving policy makers the chance to equalise the outcome between the silos.
This approach to evaluation, in identifying the many variables and assumptions, also helps us to see gaps in knowledge so we can focus research on gathering any important missing data.
Modelling and other public health interventions
The value of modelling goes beyond policies such as the Cold Weather Plan that can take years to bear fruit. Most public health interventions occur in complex environments and involve multiple variables. It is important to understand the role that context plays in their success.
Models provide an opportunity to vary contexts and see what matters and what does not. They are the obvious precursor to setting up a pilot that can then be designed around key factors that seem to be important, regardless of context. Inserting modelling into the process of early evaluation, before piloting, could be vital if we are to escape the scourge of successful pilots that are not rolled out, because they seem to work only in one place.
Dr Zaid Chalabi is Associate Professor in Mathematical Modelling at the London School of Hygiene and Tropical Medicine.