Japan highlights innovative Asia Pacific model for Social Impact Bonds

BY CHIH HOONG SIN & ICHIRO TSUKAMOTO

SIBs are emerging as a way to reconfigure relationships between the state and civil society in ways that their Western inventors may not have anticipated but may wish to replicate.

Japan’s experience of Social Impact Bonds (SIBs) offers some important insights. First, it challenges the notion – held by some – that SIBs should be seen chiefly as a way to cut costs: in Japan we see a typically Asia-Pacific framing of SIBs that’s more focussed on developing civic society. It’s an approach that may catch on more widely around the world.

Secondly, the Japanese experience demonstrates the importance of institutional infrastructure and culture – and not only availability of finance – to make SIBs work well. SIBs require a complicated culture in which to thrive: Japan shows how it can take time for the various elements to be established.

Japan expressed an early interest in SIBs, becoming part of the OECD Social Impact Investment Taskforce. A five year nationally-funded research project, based at Meiji University in Tokyo, looking into developing SIBs for Japan is almost complete.

On the face of it, Japan might look to be in the same boat as, for example, the UK. The country’s national debt at the end of the last fiscal year was 254% of GDP, compared with the UK’s 88%. So, one might think, Japan would use SIBs to drive down spending and increase efficiency.

In fact, although Japanese rhetoric initially focussed on fiscal savings, interest swiftly shifted to SIBs steering improvements in societal wellbeing that might require more, rather than less, public spending. Japan has also been much more like other Asia-Pacific countries where SIBs have been seen as a means to reconfigure relationships between the state and civil society and focus on long-term issues of public health.

Past and existing projects have looked at, for example, how to create learning supports in deprived areas for NEET young people – those not in education, employment or training. A nationwide health SIB has just been launched focussing on better managing or preventing long-term conditions such as dementia, diabetes, cancer, cardio vascular disease.

Japan’s government has shown a high level of interest in SIBs. It’s seen its role largely as being to signal interest and legislating to make money available. In December 2016 Japan harnessed dormant bank accounts, like the UK has done, to set up the equivalent of Big Society capital, which is envisaged to go live from 2019. In addition, the Ministry of Economy, Trade and Industry (METI) and the Ministry of Health, Labour and Welfare have provided pilots around the country with funding for operations.

So pilot programmes have tended to suffer not from shortages of capital. Their problems have, rather, concerned lack of measurement or been because service providers are short on thinking or capacity to deliver outcomes-based contracts.

Institutional weaknesses in Japan
Japan has been institutionally handicapped in developing SIBs in a number of ways. The country has a weak public management culture: transparency, a focus on efficiency and on prioritising clients’ views of outcomes are less developed than in the UK.

The country is only slowly creating the building blocks for outcomes-based markets. The UK government has supported the creation of outcome payers: SIB investment funds such as the Innovation Fund, the Fair Chance Fund, Commissioning Better Outcomes Fund, and the Life Chances Fund. Britain has invested in developing an evidence base that can support outcomes-based contracting. There is also legislation that directly or indirectly supports SIBs, including the Public Services Social Value Act and provision of social investment tax relief. Japan has had a much less developed basic infrastructure.

In comparison, the UK’s voluntary – not-for-profit – sector is also more robust, diverse and heterogeneous, including organisations of different sizes. There are strong umbrella bodies such as the National Council for Voluntary Organisations. Japan has a long history of social sector organisations but the flowering of the not-for-profit sector is more recent, encouraged in the past 20 years by legislation liberalising regulation of the sector.

Many Japanese organisations remain small and hyper-local, which raises questions about their capacity to engage with SIBs. The country’s not-for-profit umbrella bodies are also relatively weak in terms of their lobbying capacities and the extent of their networks across the sector.

There are fewer charitable foundations than in the UK. So, social sector organisations in Japan largely rely largely on government funding, whereas UK social sector organisations tend to have more diverse funding. This financial dependence makes it harder for social sector organisations to question government.

Problems measuring impact
These days, the UK social sector is familiar with measuring and demonstrating impact. Japan’s social sector organisations are less experienced in measurement and, where they are, their focus tends largely to be on activities and inputs.

Lack of diversity in markets – in terms of service providers, investors and intermediaries – also hampers the growth of SIBs in Japan. Until 2017, when the Yokohama SIB was developed, all pilots were driven by the Nippon Foundation or its subsidiaries. So a single, large player was trying to nudge the market along.

There are significant differences between the UK and Japan in the maturity of intermediary markets. The UK has moved from a virtual monopoly among a few intermediaries in the early days of SIBs to a growing, diverse market, bringing in many different skills. Intermediaries no longer offer only financial backgrounds. There are people with commissioning support experience as well as good provider engagement experience. The same cannot be said quite of Japan, where the emergent intermediary market remains in its infancy.

From the commissioner side in Japan there remains a poor understanding of SIBs so it is difficult to collaborate with providers about constructing SIB models. As a result, commissioners may not know what to specify. They tend to identify general outcomes which may lack meaning and be unachievable within the timescales.

Building underdeveloped civil society
Nonetheless, Japan’s vision for SIBs is innovative as it tries to use them to place individuals at the heart of public services that have tended to be dominated by centralised bureaucracies. As in China, SIBs offer Japan a model that facilitates an underdeveloped civil society sector to grow and play a more visible role. SIBs allow responsibility to be devolved downwards while retaining central control through outcome-based funding. SIBs help square the circle for centralised governmental structures that wish to devolve responsibility while retaining policy direction.

Western countries such as the UK, where SIBs originated, would be wise to take stock of this evolving emphasis. Just as football played in Brazil and Spain has turned out to be more exciting than the game played by its inventers, SIBs, in their cultural adaptation, may prove to be a more transformative model than originally envisaged in Britain.

Attractions of Asia Pacific model
This adapted model could turn out to be an attractive export even to its inventors. Austerity – the ground into which SIBs were originally planted – will not last forever. SIBs may, in the longer run, offer opportunities for a more ambitious and hopeful agenda of improved well-being that may be politically realistic in more prosperous times. Such reinvention and rebranding of SIBs might also make them more attractive to their Western critics. The Left in Britain and US has tended to dismiss SIBs, suspicious of them as vehicles for commodisation and privatisation of public services. A revisioning of SIBs might suit a more left-wing agenda.

This SIBs model with a different emphasis might also offer an approach to reform in Britain where big providers such as Serco and Capita – focussed on driving down costs rather than on quality delivery to the person – are edging out smaller local players. Since 2010, many small civil society organisations have collapsed in the UK and others have had to merge as size has become the key to survival. Might SIBs eventually offer a way to address this trend and recognise financially the value of local deliverers with smaller caseloads but higher quality?

As the SIBs story unfolds, the purposes and future of this approach to service delivery remain in flux. Analysing the Japanese – and wider Asia Pacific – approach is a reminder that we may not have scratched the surface of SIB’s potential for achieving a wide range of agenda.

Dr Chih Hoong Sin is Director of Innovation at the Office of Public Management, London, UK

Ichiro Tsukamoto is Professor of Public Management at the School of Business Administration, Meiji University, in Tokyo, Japan.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

We should ask three big questions about SIBs

BY ELEANOR CARTER & CLARE FITZGERALD

Evaluation should test key SIB promises to government: greater collaboration, prevention and innovation, argue two researchers from Oxford University’s Government Outcomes Lab.

To discover whether Social Impact Bonds work, we would be wise, first of all, to ask: “Work for whom?” Our response is: “For public service commissioners.” This answer acknowledges a reality: if SIBs don’t make sense to government, they are unlikely to be more than a passing fashion.

This standpoint cuts through a myriad of promises that typically surround SIBs and other payment by results programmes. It also helps focus how to evaluate them. We should measure SIBs by looking at what, if anything, they can deliver that public service commissioners seek.

That’s what we’re doing in the Government Outcomes Lab – GO Lab – at the Blavatnik School of Government, where our work is currently co-sponsored by the School and the UK government.

The dominant narrative around SIBs, born in an age of austerity, is that they can deliver cashable savings to government. Of course, every government seeks such dividends. However, this claim, far from proven – at least at scale – may or may not be true. Either way, we believe SIBs go well beyond promising the efficiency and entrepreneurialism claimed by conventional outsourcing. SIBs make a more fundamental offer that should be rigorously tested: that they provide an opportunity to remake the public sector itself.

Public sector reform has sat at the heart of every government’s agenda – Conservative and Labour – for nearly 40 years, since the election of Margaret Thatcher in 1979. Its goals – aside from making savings – have included three principles that SIBs directly claim to address.

SIBs address 3 principles of public sector reform
The first principle is to improve collaboration in the commissioning and provision of services so that the collective effort is better focused on the needs of the service user.

The second principle is to shift activity to a more preventive footing, so that service provision is moved upstream of social challenges and towards earlier identification of risk.

The third principle is to make innovation less risky for commissioners so they move to evidence-based procurement, decommissioning underperforming services and becoming better at reaching those, often experiencing the most complex mix of social issues, who may be poorest served by existing provision.

We should, therefore, question, in our evaluations, whether SIB models are improving collaboration, shifting services towards prevention and creating spaces in which public service commissioners and delivery organizations can innovate.

Principles largely untested by evaluations
Our review of evaluations suggests that they rarely explicitly or rigorously test any these three principles by comparing a SIB commissioning approach with a grant, fee-for-service or even in-house delivery for a given population. Evaluations also face the “doughnut challenge”. That’s the difficulty, particularly with impact evaluations, in teasing apart the effects of the commissioning architecture (the dough) from the effects of frontline services operating in different ways (the jam). Indeed, most evaluation material is quite descriptive and exploratory. It doesn’t typically offer or test a mechanism by which a particular SIB is expected to work in practice.

Collaboration is seen
However, where we have tried to read these three principles into the evaluations that have been produced, we see glimmers of each logic being present in SIBs. So Peterborough’s SIB – focused on reducing ex-offender recidivism – saw front-line organisations pull together in cohesive ways, wrapping around the service user.

We’ve also seen multiple commissioners joining forces as, for example, where the Department for Education, Cabinet Office and the Department for Work and Pensions came together for a fund-based approach to support young people vulnerable to disengaging from school and work. We see top-up funding approaches from the Cabinet Office to acknowledge that there might be wider public service benefits springing from a SIB that are not captured within the outcome payer’s domain.

“Step-down services”, more than prevention
As far as prevention is concerned, very early intervention programmes are rare in the SIB field, but there is a focus on “step-down” services. The Birmingham Foster Care SIB is a good example that is already demonstrating cashable savings.

A growing number of SIBs try to de-escalate situations of family conflict or address unstable conditions for children and young people. Typically, for example, the Department for Work and Pensions has worked with people only once they are employed. Now, through SIB programmes, the Department, through the Young Engagement Fund, is becoming involved in programmes that try to understand and anticipate future needs. It is working with school children to prepare them for a smooth transition into work.

Space for innovation
There are also examples of SIBs supporting innovation and helping to dissipate cultural inertia. Typically commissioning has a cycle – there’s the planning stage, the doing phase and then there’s the evaluation. A SIB can help reduce risk at any or all of these stages, allowing SIB commissioners flexibility to think differently about their work. So, for example, with an innovative programme such as the GLA Rough Sleeping project, a SIB allowed commissioners to ask of themselves: ‘Do I know how to be the best contract holder, or would I benefit from bringing in a third party that might have a better appreciation or knowledge base about that side of things?’

SIBs can also enable a certain amount of experimentation by providing quicker feedback through performance management. It’s this feedback that allows frontline organisations to change the way they are doing things to better meet the needs of their clients. A well-designed SIB also creates impetus for high quality impact evaluation within a payment mechanism. That inclusion can, in the long run, only help to improve the quality of future commissioning.

Time to test public sector reform potential
In conclusion, we cannot demonstrate, on the basis of existing evaluation evidence, that SIBs fulfil the many and varied promises that their champions make for them. But observing SIBs through the lens of public sector reform – and their possible role in it – we offer an approach to evaluating whether they might be a tool for remaking the public sector.

Eleanor Carter and Clare FitzGerald are research fellows at the Government Outcomes Lab (GO Lab)at Oxford University’s Blavatnik School of Government.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Investors need rigorous assessments of Social Impact Bonds

BY KATY PILLAI

A major investor highlights the vital role that research and evaluation should play in developing this form of outcomes funding.

Evaluation and research into Social Impact Bonds (SIBs) is a hot topic for Big Issue Invest. We are one of the UK’s leading social impact investment firms, having invested in approximately 350 charities and social enterprises since 2005, in our quest to dismantle poverty and create opportunity.

Our investments support areas such as access to housing, financial and social inclusion, mental and physical health and well-being, and employment, education and training.

We have made several SIB investments since 2012 and have watched the market develop. Our first-hand experience is that there is more work to be done to refine the model, but there have been impressive outcomes for programme participants and the charities delivering the contracts in which we have invested.

How we use research and evaluation in SIBs
We aim to address the structural challenges of SIBs and maximise their individual and collective social impact. Research and evaluation can help in this goal. Many commissioners and service delivery providers are unfamiliar with SIBs and I often direct them to impartial, well-informed research to build awareness and understanding.

Discussing Big Issue Invest’s learnings and experiences from individual programmes with evaluators helps us to contextualise the situation and identify emerging trends in this rapidly-developing field. These partners can develop the tools and models to test and critique our theories and insights ‘from the field’ and evaluate the wider market, whereas our frame of reference is often limited to our specific investments.

Research in social investment and SIBs is at a nascent stage. It is crucial that the right foundations are laid today to enable good-quality ‘market’ level analysis in the future. Consistency and rigour of approach will pave the way for systematic reviews and meta-analyses, essential if outcomes-based approaches are to become commonplace in the commissioner and policymaker toolkit.

We also welcome more quantitative approaches and ambitious evaluations that compare SIBs to traditional fee-for-service mechanisms or payment by results more broadly. SIBs are often conflated with outcomes-based approaches in general, which makes it very hard for investors to assess – and improve – their social impact.

Funding is, of course, needed to allow this work to take place. At the moment, evaluations are too often the balancing item in a very limited budget, constraining their ambition. We wholeheartedly support the calls for a ring-fenced fund for SIB evaluations, recognising the value of the output to commissioners, central government and potentially also philanthropic funders who might seed the fund.

Understanding what, why, how
We are an impact investor: outcomes are our reason for being, not just a by-product of our investment activity. Social impact due diligence underpins every investment decision we make.

We look closely at the theory of change for each SIB. There needs to be a coherent and credible hypothesis for how outcomes will be improved for the programme participants and – beyond that – how the programme could help to tackle the underlying issue through, for example, earlier intervention. We interrogate potential perverse incentives in order to mitigate them. Research and evaluation from previous programmes helps us to do this: we rely on it to validate the causal link between inputs, outputs and outcomes and complete our due diligence of the intervention.

It’s important that everyone involved reviews the theory of change periodically after the contract launches. One of the strengths of SIBs is that they shine a light on what works and what doesn’t, enabling real-time improvements and sharing of learnings for future contracts. If we scrimp on monitoring and evaluation, we undermine the programme and indeed the SIB model.

Evaluations therefore need to be robust, relevant and timely. We want to understand not only the results achieved but also why they are (or are not) achieved and how we can replicate and improve on them. That might be a programme evaluation or an impact evaluation, or qualitative or quantitative approach, depending on the context but we certainly need more than outcomes verification.

We seek insights into the drivers of success so they can be reflected in future projects. The GLA Rough Sleeping SIB in 2012 was divided into two lots awarded to separate providers, one funded by Big Issue Invest. We know the absolute outcomes achieved by the programme but would like to dig deeper into whether different operational or investment approaches had a bearing on success.

We are keen to work with the evaluator community to design the evaluations and contribute to them. It’s important to be confident that Big Issue Invest’s loan has achieved its social objectives – and those of our investors in turn – so we are a consumer of evaluations as well as contributors to them.

We are one of very few organisations that has worked across several SIBs in different regions and policy areas. We can contribute data, insight and practical experience and welcome the opportunity to do so. At a practical level, we can coordinate with the evaluator to minimise the data collection burden on the service provider’s staff and the programme participants. If we can bring the evaluator in to the design phase early, we can also incorporate their evaluation into the delivery model early to avoid duplication or complication down the line.

Using data and analysis to target interventions
Reliable data and analysis is essential to high-quality SIB design. For example, we are involved in an ‘edge of care’ SIB for young children where it is unfeasible to roll out an intensive (in other words, expensive) intervention to all children on social services’ radar. Rather than only work with children on the very cusp of care – when it is often too late to reverse their trajectory or the trauma they have suffered – a researcher is working with commissioner data to identify early risk factors that increase the child’s propensity to enter care. The programme will be targeted towards these high risk children as well as those on the very cusp of care. This allows the commissioner to fund an early intervention service that is also cost-effective, often challenging in outcomes-based commissioning. There is huge potential to harness data and analysis in this way to design preventative services.

The value of timely feedback
Speed of evaluations is a challenge. Evaluations are valuable to SIB stakeholders when developing follow-up programmes and carrying out due diligence. If investors have a good level of confidence in the achievable outcomes, the cost of their risk capital should be lower. That is in everyone’s interest. It doesn’t encourage evidence-based commissioning if the evidence is released after the next programme is launched!

Midline and end-line reviews as part of a formal evaluation are, of course, extremely important but they are not enough. Outcomes-based programmes also need shorter, informal feedback loops, preferably involving the evaluator. Early results and findings can be used to improve programme delivery – but not if they are shared in an end-of-programme evaluation that takes a year to publish. Ideally, we’d like a quarterly or a six monthly check-in with the evaluators that can identify and unpick performance and its drivers.

We recognise the tension between this approach and concerns that the evaluation will influence the programme outcomes. A balance needs to be struck. SIBs support people with complex needs who deserve the best possible chance of better life outcomes, so although evaluation rigour is crucial, we owe it to them to make the intervention as effective as it can be.

Importance of counterfactuals
Three inputs are usually needed to assign a value to a SIB outcome: (1) the projected costs to deliver a programme (preferably validated through a competitive procurement process); (2) the costs per outcome achieved under comparable programmes, if known; and (3) the savings case (the projected benefits for the commissioner). If you don’t have an understanding of what would have happened anyway, at least one of these calculations will be flawed. That’s why we can’t afford to disregard the counterfactual.

That doesn’t mean that every SIB needs to link payments to performance compared to a counterfactual, measured by an RCT. There are lots of factors to consider when designing the payment mechanism and there is no single ‘right’ approach. However the counterfactual can always be taken into account. Under a rate card approach, the rates should be set after considering deadweight – even if the assessment is imperfect, it is better than ignoring it completely. The counterfactual can then be assessed in the programme evaluation and used to inform the pricing of future contracts.

I am not saying SIBs should be commissioned only if there is perfect data to value the counterfactual. Rather, I am emphasising the need for new approaches that measure outcomes and cost-per-outcome to allow commissioners to make evidence-based decisions in future. Big Issue Invest is trialling approaches that allow an outcomes-based contract to be launched with imperfect information, while ensuring checks and balances limit windfall gains and losses and include mechanisms to tackle the information gap.

One option is to run an initial ‘discovery phase’ of the contract for one to two years. The discovery phase outcomes pricing estimates the counterfactual, but sets parameters to ensure that no party makes excess gains or losses. In this way, the partners have the opportunity to implement the SIB delivery model. During this time, outcomes and the counterfactual are measured rigorously. The data and analysis is then incorporated into a revised payment mechanism for the rest of the contract, after which point it operates like a ‘standard’ SIB. This approach bridges the knowledge gap without delaying a potentially high-impact programme or risking inequitable risk and return.
Where next?

SIBs bring together different worlds. The success of SIBs is dependent on partnerships where the whole is greater than the sum of the parts. They require new ways of working for everyone involved – for investors, providers and commissioners. I expect they can seem strange to evaluators as well. Forging new links and understanding the perspectives of others is crucial.

We are starting to see these worlds come together and collaborate for better outcomes. There is more interaction and understanding between researchers and evaluators, policymakers and budget holders, delivery organisations, and investors. It is early days but the outlook is promising.

Katy Pillai is Investment Director, Big Issue Invest: www.bigissueinvest.com @katyjones | @bigissueinvest

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Academics can show governments how to evaluate SIBs more rigorously

BY CHRIS FOX

A wide range of approaches can help identify causality and effectiveness even in complex environments.

We can – and we should – improve our evaluations of SIB and Payment By Results (PBR) programmes. They should focus more on causality, rather than simply contract compliance or implementation.

If we don’t focus on attribution, it will become hard to demonstrate that SIBs are more than a series of interesting pilots. We’ll miss the chance to test an alluring proposition – that SIBs could transform the large scale commissioning and delivery of health and social welfare programmes.

Getting evaluation right in this field is not – as some might suggest – intrinsically challenging. SIBs and PBR projects do not create unusual difficulties for evaluation techniques. We have the knowhow – sophisticated, diverse tools are well-developed that could settle most questions thrown up by SIBs. The real issue is: will those who champion SIBs expose such initiatives to the full rigour of the evaluative tools that exist?

Academic responsibility
The academic community can help ensure that rigour. Management consultants, contracted to perform evaluations, tend to provide what governments specify, which, so far, has been limited and fallen short of what’s required. Academics could set out a wider, more exacting range of evaluation options that are more suitable. We should show policy makers clearly how better evaluations could be achieved, particularly if the case for widespread adoption of SIBs is to be made.

This difficulty in properly assessing the impact of SIBs seems to be a particularly British problem. In the United States, most SIBs have been accompanied by fairly rigorous counterfactual evaluations, including randomised control trials (RCTs). There, the credibility of the SIB model among commissioners and investors has required demonstration of its ability to deliver tangible outcomes. This may be because, in the US, more funding has come from wealthy individuals or private foundations, with an investment ethos. In Britain, funding tends to spring from philanthropic organisations which seem interested in testing concepts over categorical outcomes.

Evaluations are too based on performance management
Whatever the reasons, SIB pay-outs in the UK typically rely more on performance management information to demonstrate the achievement of outputs. Supporters of this approach say that complicated counterfactual evaluations add to the already high transaction costs associated with SIBs. That’s understandable for individual SIBs. However, cumulatively, this approach hinders the quest to find out whether SIBs really work. It undermines the case for wider roll-out.

Evaluations can and should answer two major questions about SIBs. There’s “attribution”: whether SIBs actually achieve the outcomes desired. Second, we need to understand SIBs as a mechanism and establish how effective they are compared with other models of commissioning. This is important because there are less expensive, less complicated methods than SIBs for commissioning services in this field.

The attribution issue has become unnecessarily mired in a polarised debate about whether RCTs are suitable for SIBs projects. Opponents contend that RCTs are not particularly useful in this field because SIBs interventions tend to take place in highly complex environments. While it’s true that these interventions often occur amid complexity, that actually strengthens the case for RCTs. It becomes even more important to understand whether an intervention is indeed responsible for any of the impacts being observed.

Testing theories of change
Good RCTs would strengthen SIBs evaluations because they would be theoretically informed. They would start with a theory of change setting out the potential causal mechanisms that are of interest. In contrast, many SIBs evaluations rely on contractual frameworks and demonstrating whether they have worked, rather than testing hypothesised causality. Most good RCTs today are also accompanied by high quality implementation evaluation. So they have a dual strategy.

Well organised RCTs avoid “one-shot” design. They are actually a sequence of evaluations that build by testing, at a granular level, particular moderators of change, rather than simply focussing on the overall social outcome and trying to come to a one-shot conclusion. This is how, in reality, even medical research works. You don’t do a single RCT. You build from small scale studies through to larger scale studies.

Sequences of evaluations are good
The wider evaluation world is focussing more on sequencing evaluations and ensuring that tools employed are appropriate to the point of a programme’s development. This avoids problems that one shot evaluations can create: that you evaluate too early; that the throughputs you were promised never arrive; that you end up developing an evaluation design which is underpowered to identify the changes that you’re looking for; there are inconclusive findings that have cost a lot of money but don’t provide the hoped for insights.

I advise against the one shot model. Instead, we like to start evaluations early without diving straight in with an RCT. We focus on developing a sequence. That’s the strength of the Education Endowment Foundation evaluation model. It begins with small scale pilot studies that focus on theory of change and early implementation, then efficacy trials that are more like a small RCT, leading up finally to effectiveness trials. Only at that point – when causalities have been established – is control finally handed over to implementers.

Building commissioner confidence
This sequential approach gives commissioners confidence. You’re saying to them that this isn’t a “one shot, put all your money on the table up-front” model. It’s about gradually building knowledge and providing gate-keeping points where a commissioner can ultimately say: “This isn’t working, we need to rethink. We may need to reinvest or, even disinvest.” That’s helpful to commissioners, especially if they are being asked to back innovation that feels risky.

Small ‘n’ designs
In some cases, RCTs are not possible, but there are many alternative models of impact evaluation that could be considered for SIBs. “Small n” designs provide ways to think about causal attribution where a programme does not have sufficient numbers to allow a traditional impact evaluation design. Process tracing is an example of “small n” design, where one uses theory to identify critical points in a change process that need to be tested. Then one selects cases to test these critical points, using interviews and observations of what’s going on. This Popperian approach acknowledges that there is no absolute objective knowledge. However, it can find ‘smoking gun’ evidence that strongly suggests causality, even if that may not amount to absolute proof.

These process approaches that search out causality would be an improvement on current tests of some SIB or PBR programmes which, if they can’t do an RCT, tend to opt for process/implementation evaluations that are less demanding – usually interviewing stakeholders and writing a report, but lacking a more theory-driven approach.

More rigour is needed
I’ve set out ways in which SIB and PBR evaluations could be improved by RCTs or hybrids that avoid the unnecessarily polarised debate between the pro- and anti-RCT lobbies. Beyond RCTs, there are other approaches to evaluating causality, suitable in instances where there are small numbers of cases. We should learn from this wider discussion of evaluation techniques. Academics owe it to those investing and working in SIBs to ensure that policy makers adopt a rigorous approach to evaluation. We need to know what works and what doesn’t if SIBs are ever to be widely adopted.

Chris Fox is Professor of Evaluation and Policy Analysis and Director of the Policy Evaluation and Research Unit at Manchester Metropolitan University. He is co-author of “Payment by results and social impact bonds: Outcome-based payment systems in the UK and US”, published by Policy Press in February 2018.
https://policypress.co.uk/payment-by-results-and-social-impact-bonds

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

Impact bonds could offer a paradigm shift towards more effective public services

BY EMILY GUSTAFSSON-WRIGHT

Social and Development Impact Bonds require enormous effort for the partners involved, but they have a potential to transform the financing and delivery of social services across the globe.

In winter 2015, Courtney arrived at Frontline Services, a not-for-profit US organisation that helps citizens in Cleveland, Ohio. She was 28, living in a shelter for homeless women, struggling with mental health and substance abuse issues and parenting three young children who were in the custody of the county.

Courtney had just about given up hope that she would ever be able to care for her children on her own. Until that point, the county caseworker assigned to her family had little incentive to reunite Courtney and her children because the caseworker’s primary job was to protect the children.

Living with a birth parent is almost always better for a child’s development than foster care, provided the home environment is safe and healthy. Nevertheless, before entering Frontline Services, Courtney had few ways to change the trajectory of her children’s lives. Fortunately for her children, she was walking that day into a social services experiment, one of only seven similar experiments in the US at the time. In this experiment, a social impact bond (SIB) – designed to “pay for success” – the county’s government had pledged to repay private investors for successful reductions in out-of-home placements for children whose primary caregivers were homeless.

This incentive meant that Courtney was assigned a caseworker dedicated to her – someone who would look at her particular circumstances and tailor a plan to help her turn her life around and unite her with her children. Courtney’s caseworker could work across county service providers to identify the right mix of services for her.

The SIB meant that a dedicated group of stakeholders was meeting regularly across government and non-government entities to focus on one thing – reuniting Courtney with her children, and doing the same for other families in similar circumstances.

SIB contracts focus on outcomes, so service providers tailor their services to what works for the target population. They helped Courtney to address her debts with classes in financial management and offered family counselling.

As a result, Courtney was able to reunite with her children, enrol them in supportive school environments and stop the cycle of dependency on the foster care system. The result was not only a better family outcome, but also a reduction in the enormous costs to the county, had Courtney’s children remained in the county’s care.

Impact bonds are changing developing countries
Meanwhile, nearly 12,000 km away, in a village in rural Rajasthan, India, lives a 13-year-old girl, named Punam. She comes from a poor family – her parents are labourers. Although Punam started school at age seven, she became one of India’s three million out-of-school girls, when she was forced to drop out to tend to her family’s goats.

In the same year as Courtney arrived at Frontline Services in Cleveland, Ohio, a field co-ordinator, working for an organisation called “Educate Girls”, arrived in Punam’s home in Rajasthan. He spoke with her parents, explained the benefits of educating Punam and tried to convince them to send her back to school.

Even after multiple attempts, the parents didn’t agree to send Punam back to school. The Educate Girls caseworker returned some weeks later with a volunteer from the community but again failed to persuade the parents. “What benefit will it give her or us?” they asked. “She will eventually marry and her responsibilities will revolve around doing household chores, assisting in farming life, raising children and taking care of her family.”

Nevertheless, Educate Girls made a further attempt to encourage her parents to let Punam attend school. This time, they asked the school’s principal to join them in a final visit to her home. With this added influence, Punam’s parents agreed finally to sending their daughter to school.

Why was an impact bond so important in this case? Because the contract was based on the achievement of outputs and outcomes, Educate Girls field-staff were empowered to innovate at the field level, trying to find solutions for getting Punam into school. Now, two years later, Punam, and many girls like her, are enrolled in and enjoying school thanks to Educate Girls and this Development Impact Bond (DIB), based on the same principle as a SIB, but with a third-party outcome funder, instead of the government.

These two stories capture the real human benefit that can emerge from outcome-based contracts such as SIBs or DIBs.

How impact bonds work
Let’s just re-cap for a moment how impact bonds actually work. In an impact bond, private investors supply upfront capital to service providers to deliver an intervention or program to a population in need. Upon the achievement of a set of agreed-upon results, the investors are then repaid by an outcome funder. With a SIB, this outcome funder is the government. With a DIB, outcomes are financed by a third-party organisation, such as a foundation or donor.

Since the launch of the first SIB in the UK in 2010, the impact bond market has grown exponentially. Last year, some 32 new contracts were signed. As of January, 2018, there were 108 contracted impact bonds (103 of them SIBs, 5 of them DIBs) across 25 countries, along with many more in design. All but one of the 103 SIBs were in high income countries: last year marked the contracting of the first SIB in a low- or middle-income country, the Workforce Development SIB in Colombia.

Most (42) SIBs are in the UK, the country that pioneered the impact bond model with the Peterborough SIB in 2010. The results of that SIB – aimed at rehabilitating ex-offenders – were released last year: reoffending of short-sentenced offenders dropped by 9 percent and the investors were repaid in full. The US has also established itself as a player in the field, coming in second with 19 impact bonds.

The five DIBs include Educate Girls in India, one for coffee and cocoa production in Peru, as well as one for physical rehabilitation across three countries in West Africa, a poverty graduation program in Kenya and Uganda, and the recently launched Utkrisht impact bond for Maternal and Newborn Health in Rajasthan, India.

Characteristics of SIBs
Most SIBs contracted globally are in the employment field, followed by the social welfare sector, which includes programmes to reduce homeless – “rough” – sleeping, or reduce out-of-home placements as in the case of Courtney and her children. Other areas for SIBs are health, criminal justice, education, the environment and agriculture.

Probably about 30 or 40 impact bonds are in design in high-income countries while more than 20 are being designed in low and middle-income countries. We see some difference comparing high income and low or middle-income countries. The majority of impact bonds in the latter are in the health sector, followed by employment and, then, agriculture.

What do these impact bonds look like in terms of size? The smallest one, in terms of beneficiaries, reaches 22 individuals – that’s in Canada. The largest one reaches 650,000 individuals in Washington DC, which is an environmental impact bond focusing on developing infrastructure. It is perhaps a little bit unfair to compare that one in terms of size with the rest of them because it’s a city-wide programme. The next largest in size is the Maternal and Newborn Health DIB in Rajasthan, India, with 600,000 potential beneficiaries.
However, the median impact bonds are reaching about 565 individuals, so they are quite small. Capital commitment of bonds ranges from $80,000 to $25 million. Again, that $25 million is the one in DC. The average is about $4 million and the total upfront capital invested across the impact bonds is over $300 million.

Who is benefiting?
The bonds mostly target marginalised populations, including women affected by violence, young migrants, single mothers, with a few for ex-convicts, vulnerable and young people, people diagnosed with mental health conditions, refugees and individuals with physical disabilities.
What do we know, eight years in about their performance? There have been some outcomes achieved and payments made, such as in the case of the Peterborough SIB, mentioned earlier. In an Australian SIB, 203 children were reunited with their families and the return to investors was nearly 16.5 per cent over the four years of the scheme.

Shifts in public programme behaviour
However, perhaps the more interesting observation is a real shift seen in government and service providers to thinking about outcomes as opposed to paying for inputs. Impact bonds are also driving performance management so service providers are introducing or improving systems of performance management in their programmes.

Impact bonds are incentivising collaboration, between the public and private sectors, but also across government, vertically and horizontally. They are building a culture of evaluation because outcomes must be measured and monitored. Most impact bonds so far have been focused on investment in preventive interventions. There has also been some reduction in risk for governments, which have not paid for outcomes that weren’t achieved.

What are we not seeing so far? It had been hoped that impact bonds would lead to an influx of additional private funding. However, given that government or outcome funders ultimately repay the investors, then that’s not really more money for a particular social service. Impact bonds have also yet to achieve change at scale: the majority are reaching very few individuals and are fairly small in terms of investment.

Many thought that impact bonds would focus on experimental interventions. So far, we haven’t seen that: investors have been unwilling to take that risk. We’re seeing SIBs used in the middle phase of development of interventions, rather than at the “seed” or “at scale” ends of the process. However, the flexibility that service providers are allowed in terms of their service delivery has the potential to encourage innovation. It’s also probably too soon to say whether or not impact bonds can achieve sustainable outcomes in the long run through the systematic change that’s happening but it does appear that the partners currently involved have indeed shifted their thinking.

Challenges of impact bonds
What are the challenges? This is a new form of government contracting, a new way to do business. Co-ordination of all of the stakeholders is difficult: sometimes they don’t understand each other well; just getting all those people around the table can be really difficult. There can be some political constraints and legal barriers.

Key questions remain. Can impact bonds be used at scale? Are they more effective than input-based based financing or traditional payment-by-results? Do the actors in social service provision have the capacity to adapt to the demands of financing tied to results? Can they manage the rigorous focus on performance management that this is likely to entail?

Next steps
It is worth considering what would be needed to expand the use of impact bonds or, more generally, payment by results. The evidence base needs to grow and there is a need to collect more information on services that work and on their costs. Also, potential outcome funders and investors need to be educated in not only the potential of an impact bond approach but also the challenges. There needs to be supporting legislation and regulations to facilitate paying for outcomes both at a country and local level but also within organisations.

To achieve scale, countries could establish outcome funds for particular sectors. The UK government has launched seven outcomes funds and efforts are underway in the US to develop outcomes rate cards. There are several outcome funds being developed to tackle tough social issues in the developing world as well. These would allow the outcome funder to set prices for desired social outcomes and then to contract with service providers to deliver interventions to achieve those results. Global investment funds would also benefit from contributing to this new financing mode.

In the US, $800 billion are spent annually on social services. Only one per cent of that spending is evaluated for effectiveness. In the UK, £220bn was spent on social and health services (2015/2016), yet we know very little about the effectiveness of that expenditure. Thus we need more empirical research which asks: “What do impact bonds achieve, compared with input based financing?” It is also important to know how well impact bonds perform compared with traditional results-based financing. These are both hard questions to answer rigorously and will take some time.

Impact bonds and global problems
The social and environmental problems that we face at a global level are enormous. It’s estimated that $1.4 trillion will be needed annually to achieve the Sustainable Development Goals by 2030. There is little evidence that such complex problems can be solved by continuing the same old failed approaches.

Investing in preventive measures can avoid higher costs down the road and make the public and civil society sectors more efficient. By paying for outputs and outcomes rather than paying for inputs that have unknown outcomes, spending should be more effective.

Impact bonds may not be the right solution to every problem. However, they do represent a long-overdue, paradigm shift. They’re a means to an end, an opportunity to think about, and hopefully produce, systematic change. At the very least, they may be the stepping stone to establishing the monitoring and evaluation performance standards and output planning that can ensure every individual receives the services that they need to live safe, healthy and productive lives.

Dr Emily Gustafsson-Wright is a Fellow in the Global Economy and Development Program at the Center for Universal Education, Brookings Institution, Washington DC.

If you wish to receive our weekly blog on SIBs, please email Paula.Fry@lshtm.ac.uk and we will add you to our subscriber list.

“Rethink questions to patients in general practice and focus more on improving primary care”

By Tommaso Manacorda

The “Friends and Family Test”, seeking patients’ views, has created anxiety among practices but shed little light on patients’ concerns. It’s time for a rethink, suggests our study.

The many ways in which patients can now feedback to GPs should offer a rich source of information for those keen to improve their services. However, when we evaluated the latest addition to patient consultation in England – the “Friends and Family Test” of patient satisfaction – we found that “more” did not necessarily equate with “better”.

We found that general mistrust of the FFT process, combined with inappropriate framing of its core question is undermining the initiative. But we also identified ways both to bolster trust and make patient feedback more useful for improving primary care.

Two important pieces of learning emerge from examining this initiative. First, the issues raised by the “Friends and Family Test” could usefully prompt a rethink about how views are gathered from patients about their primary care experiences. Our main recommendation is that the FFT should be revised, although discontinuing it should also be considered, as its reception within general practice has been largely negative.

Second, GPs need more encouragement and guidance to tackle inadequacies revealed by patients’ experience. We found that commitment to quality improvement is uneven across primary care and remains a low priority in some, often in those practices where patient participation groups (PPGs) meet rarely and patient surveys are infrequently conducted. Nurturing a rigorous improvement culture is at least as important as getting right the questions posed to patients.

Wording of the FFT
The “Friends and Family Test” puts a single question to patients about their general practice: “We would like you to think about your recent experience of service. How likely are you to recommend our GP practice to friends and family if they needed similar care and treatment?” Answers are recorded on a five point scale from “extremely likely” to “extremely unlikely”. Additionally, patients may be asked to comment on their reasons for the score they have given.

General practices across the NHS are required to make the FFT available to patients after every contact, collecting the data with the method that suits them best. The majority of them use handwritten cards, but tablet kiosks and online apps are also used.

Our evaluation, involving 42 practices and 118 interviews with clinicians, practice staff and patients’ representatives, found two sets of problems with the FFT. The first concerned the usefulness of the information it produced. The second problem lay with how general practices understood the FFT’s purpose and how they engaged with it.

The experiences of hospitals, where the FFT was first introduced, showed that the quantitative scores were statistically unreliable. So the metrics could not be used to compare providers for quality. That’s because the FFT does not involve a representative sample and is vulnerable to selection bias. One can’t tell whether the scores are representative of all patients. So, when the FFT question was subsequently rolled out in 2014 into primary care, the widespread awareness of these statistical limitations contributed to unease in general practices about the FFT approach.

In our study, the FFT question was deemed inappropriate by most interviewees. Many, particularly in rural areas, lacked choice of general practice, so the question of where they might send friends or family lacked reality. Also, patients found it difficult to compare their personal care with what someone else might receive, because that depends on individual factors such as age, sex and existing health conditions. Most interviewees suggested the use of a more straightforward question.

The additional space provided for further comment did potentially offer some useful feedback. However, staff in general practices felt that the anonymity of patients made it difficult to act on these comments. That’s because patients were often unclear which service they had received, from whom and the precise nature of issue that concerned them. For example, a patient complained about a “terrible phone service”. But staff said it was difficult to respond to this statement because they didn’t know who the complainant was, whom the patient had spoken to or what they needed.

Most patient feedback collected by practices was positive, but anonymity and vagueness made it difficult to identify and reward good practice. Positive feedback would often lift staff morale. However, the inability to act on complaints was often reported to be very frustrating for staff.

When additional comments gathered by the FFT pointed at some specific issues, their contribution was still considered to be of little value because the issues were already known to staff from other sources, such as practice surveys and patient participation groups.

Professionals mistrust the process
The second issue with the FFT concerned the negative reception that it received from primary care professionals. Some feared that it left them vulnerable to hostile patients. Someone might, for example, be correctly denied an antibiotic for sound clinical reasons but then score down the practice unfairly. The practice would have no way to question the scoring.

Monthly reporting of scores to the DH and NHSE added to professional concerns that the FFT process was, as some claimed, “a stick with which to beat General Practice”. Even though the Government had stated that it would not use FFT scores to rank practices, that fear of unfairness remained unassuaged, which disheartened hard-working professionals.

As a result, although all the practices in our evaluation had made the FFT available to patients, few practices felt committed to, or “owned”, the process. Even though practices had been assured that the test was intended for local quality improvement, not regulation (or criticism) by national bodies, they remained doubtful, particularly because of the monthly reporting. One GP, not realising that the scores were for meant for practice consumption, even asked their practice manager during a joint interview: “Do we have to open the box?”

Shift to reporting quality improvements
We recommend that monthly reporting of FFT scores should be stopped, and maybe replaced by a qualitative report on local quality improvements, perhaps submitted annually. That would help restore trust among practices that central bodies are not “spying” on them.

Particularly valuable aspects of the FFT are the chance for patients to comment briefly and quickly about their experiences, and the opportunity for practices to collect such feedback rapidly and easily. This comment facility could, for example, be kept within an FFT that had a less confusing core question, and patients could be encouraged to be more specific about particular services (e.g. phone consultations, clinics for chronic diseases, immunisation, etc.), making it easier to identify and address problems on aspects of care that they want to be improved.

The key issue in the long-term will be whether, and to what extent, patients’ views will contribute to making services better. Commitment to quality improvement was found to be uneven across practices. A revised FFT might play an useful role in addressing this problem, being easy to implement and thus a feasible option particularly for smaller practices with less capacity for data collection. But extra detailed guidance is needed on how to ensure that patient feedback leads to service improvement. It will be important to make clear that the Government’s priority is aligned with that of GPs in being focussed on securing higher quality primary care.

Tommaso Manacorda is a Research Fellow at the London School of Hygiene and Tropical Medicine. His report “Implementation and use of the Friends and Family as a tool for local service improvement in NHS general practice in England” is published by PIRU and co-authored by Dr Bob Erens, Professor Sir Nick Black and Professor Nicholas Mays.

This commentary summarizes an independent report commissioned and funded by the Policy Research Programme of the Department of Health for England, via its core support for the Policy Innovation Research Unit, with additional funding provided for data collection from the main sample of general practices. The views expressed are those of the authors and not necessarily those of the Department.

Social Impact Bonds offer challenges and opportunities in health and social care

By Alec Fraser, Stefanie Tan & Nicholas Mays

SIBs bring promises of extra cash and reduced fragmentation but also concerns about private sector interests, practicality and governance.

What does research tell us about Social Impact Bonds (SIBs) and their applicability to health and social care? That’s our big question as we review the issues raised by our experts in their thoughtful commentaries over the past few weeks.

It’s clear how the mood music around SIBs has appeal, notably concerning two issues that bedevil health and social care – funding shortfalls and fragmentation. Ring-fencing funds and staff to achieve specific outcomes is a key SIB feature, particularly highlighted in the contribution from Ben Jupp of Social Finance. Those in health and social care who feel frustrated by juggling cash for competing tasks – having to “rob Peter to pay Paul” – would like to get on with their jobs, unencumbered by unexpected resource cuts.

Likewise, part of SIBs’ appeal is the feeling that they address chronic fragmentation of service delivery, particularly between health and social care. In principle, SIBs incentivise everyone to work in harmony on behalf of the patient/user.

Three questions
Some of our commentators have cast doubt over these possible benefits. Ring-fencing of spending and staffing can, for example, be achieved by other means, such as minimum nurse staffing ratios. Our contributors also highlight that SIBs can’t cure the structural problem of health and social care fragmentation, though they can be helpful on a case by case basis.

We would like to draw attention to three further issues raised by our commentators: the public’s general discomfort with the idea of private interests in NHS provision; the practicality of outcome-based, debt-funding in health and social care compared with traditional forms of payment; and the governance challenges created by SIBs as they reintegrate the purchasing and providing roles that have been separated in the English NHS since the 1990s.

On the other hand, we suggest that SIBs can contribute to mitigating some longstanding issues facing health and social care. These include the need for a better focus on prevention and outcome delivery; personalising services around the user/patient; and better partnership in health and social care planning and delivery between purchasers and providers.

1. Discomfort with private financing of healthcare
The early UK SIB-funded programmes have typically dealt with issues outside the mainstream of health and social care provision, such as youth unemployment and probation services. SIBs in health and social care have mainly focused on preventive interventions to improve health (e.g. social prescribing). We’ve yet to see proposed applications that address core NHS treatment activities such as funding hospitals and primary care. Such proposals might be highly controversial to the public.

The NHS retains a unique status for voters, unlike, for example, tertiary education, where private fees are widely considered to be acceptable, or social care, which is means-tested. The NHS is a universal service. There is a general belief that it should be paid for publicly. There is also less pressure to find alternative sources than there is, for example, in the case of some local authority social services because the NHS has been better protected from financial austerity than other public services.

2. Important practical and effectiveness questions
Our second theme concerns practical questions about the effectiveness of SIBs. Alex Nicholls and Mildred Warner have drawn attention to the weakness of the evidence in this area. There is a lack of evidence to demonstrate that SIBs provide better services that are more cost-effective or more likely to save money than those delivered using conventional funding methods. A further issue is how to make cashable any notional savings generated by improving user outcomes: for example, being able to close wards and sack nurses as a result of a successful SIB would not be a popular approach in such a highly politicised public service. Unlike many social services, such as problematic families, the public generally regards spending more on the NHS as a positive thing, so there may be less political support to pursue “savings” in health.

Healthcare commissioners in the UK are well versed in scrutinising cost-effectiveness evidence but healthcare lacks the interoperability of information systems that aids oversight of the outcomes and costs arising from interventions. So, for example, it’s currently easier at the Department for Work and Pensions to assess a SIB’s impact on employment rates among NEETs (Not in Employment, Education or Training) than for the NHS to track whether a preventive health SIB has reduced use of primary and hospital care.

Healthcare commissioners may also be concerned that SIBs typically have high set-up/transaction costs and quite opaque management costs. They may be reluctant to pay for outcomes when the sources of money needed to pay intermediaries and other actors lack transparency. This caution about SIBs is well founded given the continuing costs of previous PFI projects, which created new facilities quickly but left the NHS with debt and concern among some that providers may have secured poor value for money.

3. SIBs and governance issues in healthcare
Thirdly, there is a governance challenge that may accompany the adoption of SIB-funding for health and social care services. SIBs are attractive because they are designed to bring stakeholders into partnership in pursuit of a set of goals around a particular client group. This may imply, for healthcare, a degree of reintegration of purchasing and providing. This may be problematic as it appears that a genuine market in intermediaries is currently lacking.

We also do not yet have a recognised governance framework setting out how intermediaries and the other parties should operate at particular stages of the SIB development process. This is of crucial importance for policymakers as SIBs currently develop in a space that lacks agreed rules and processes. It will be important to resolve these governance issues and create an explicit framework for managing potential conflicts of interest.

Digging a little deeper, the SIBs debate should remind us of the need to maintain a healthy scepticism about all providers, even if they are from the Third/not-for-profit sector. They should be subject to due scrutiny, like any other providers of contracted-out services.

Conclusion
There are important questions and doubts about the wisdom of widespread application of SIBs to health and social care. Nevertheless, we should be careful and avoid prematurely rejecting a still largely untested innovation that could have niche applications for entrenched social issues. It is also worth remembering that traditional health and social care procurement models are far from perfect.

SIBs are rightly taken seriously because they help to question services that have, through bureaucracy, size and provider-domination, sometimes failed to meet the needs of patients or the requirement for cost-effectiveness. But there are important questions to be asked about the transparency, costs and accountability of SIBs. Our commentators have shown that SIBs are not a panacea and that they bring unanswered questions with them. More robust and more plentiful evidence would be needed to justify widespread roll-out of SIBs in UK health and social care.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.
Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

Acknowledgement and disclaimer:
This is an independent commentary commissioned and funded by the Policy Research Programme in the Department of Health. The views expressed are not necessarily those of the Department.

Just how bad is the NHS at innovation? The answer – we don’t know

BY James Barlow

“There is strong encouragement for the NHS to make better use of innovation, but we’ve not known how our system performs internationally. Now we are developing a potential measure.”

The Accelerated Access Review (AAR) – which aims to help speed the introduction of health innovations into the NHS – was published last October. Readers with long memories may recall a string of previous reports all tackling the need to improve the UK’s health innovation system: the House of Commons Health Committee’s inquiry into the use of new medical technologies (2005)1, Best Research for Best Health (2006)2, the Cooksey review (2006)3, Innovation for Health (2007)4, Our NHS Our Future (2012)5, Innovation Health and Wealth (2012)6.

The AAR calls for a broader remit for NICE, to include more medtech and digital health innovations. Other recommendations are for closer alignment between regulatory requirements and processes, and for once-only assessment by NHS England of innovations not referred to NICE. Simpler and swifter procurement processes are part of the future agenda. And in an implicit nod towards the work of Clayton Christensen, the AAR suggests there should be some funding to support the commercialisation of disruptive innovations that have the potential to significantly improve care pathways.

These are all sound recommendations, but they are only the beginning. Recommendations now need to be turned into specific actions with budgets, owners, milestones and deliverables.

The good news from the AAR is that consensus seems to have broken out, with a genuine belief by government and NHS leaders in the potential of innovative technologies to improve patient care. The consensus extends to a call for more flexible approaches to pricing and reimbursement which can support the adoption of innovations.

The bad news is that the NHS is in deep financial trouble. For all the Churchillian rhetoric of healthcare leaders – “we have run out of money, now we must think” – and exhortations that a crisis is a great opportunity for innovation, the distinct tendency in the NHS is to hunker down, deal with the immediate concerns and shuffle innovation into the “too difficult” pile.

Goals, targets or crisis can of course concentrate the mind and, when coupled with the right kind of support and incentives, generate useful new ideas – a good example was Scotland’s Unscheduled Care Collaborative Programme in the mid-2000s7. However, the magnitude of the challenge of modernising the NHS, whilst continuing to run services with shrinking budgets, seems overwhelming. We know we need innovation, we know what kinds of innovation we need and we have a pretty good idea what is the potential impact of many innovations on improving care – the problem is introducing and establishing them in the NHS.

The AAR, if its recommendations are implemented, might help to speed up the flow of some new technologies into everyday practice, but it will not tackle the other layers of innovation that are needed. These were nicely described by the Health Foundation in 20158 as five interlinked components of a framework for change, all of which require innovative thinking – population health management, ways of delivering care, process improvement, active cost management, and scientific discovery, technology and skills. The report argued that all these are capable of delivering improvement over different timescales. Interestingly, the last of these – science and technology – is felt to have the most limited potential impact.

But just how bad is the NHS at innovation? There is much anecdote and received wisdom – “the UK is great at generating innovations but poor at adopting them”, “the NHS is always a late adopter”, “developing countries health systems are where the really interesting new ideas are”, “the USA over-adopts healthcare technology”.

Much seems to depend on perception. The 2016 Global Diffusion of Healthcare Innovation study surveyed 1,356 frontline health workers and healthcare leaders in six countries and found that the USA and UK were ranked as the two most important sources of innovation, but there were variations partly according to location – India was perceived as by far the most important by interviewees in some countries.

The volume of research on healthcare innovation processes has grown over the last fifteen years or so. We know what constrains the adoption of innovation and we know what innovators need to support them from research to commercialisation.

What we lack, though, is any kind of assessment of where different countries stand in relation to each other in their healthcare innovation capacity – how good they are at adopting innovative solutions developed elsewhere and originating innovations themselves. None of the reports on the state of healthcare innovation in the UK described above provide any real insight into our performance compared to other countries. Apart from limited work by Deloitte9, PWC10 and Accenture11, there has been no attempt to create an international comparative healthcare innovation index (the Global Diffusion of Healthcare Innovation is not an index because it teases out perceptions of innovativeness).

There are many indices or composite indicators measuring aspects of national performance in innovation in general. Indicators include measures of entrepreneurship, technology development and scientific research, innovation in general, and innovation in public sector organisations. However, these remain underdeveloped in relation to health.

A carefully constructed global health innovation index would be valuable in two ways:

• It would focus attention on the relative performance of the UK’s (or any other country’s) health system, pinpointing its strengths and weaknesses, identifying bottlenecks and issues for attention. This would enable policy makers and the healthcare sector to identify and prioritise the levers that can have the greatest impact on innovation.

• It would help medical technology and life sciences companies by providing an understanding of the attractiveness of each country’s health system as a market for products, as a source of innovations, and as a location for R&D.

A project funded by PIRU has carried out the groundwork needed to develop a global health innovation index12. We reviewed existing global innovation indices and explored the conceptual, methodological and practical issues that must be addressed. We draw three conclusions from our review.

First, one should proceed with caution when deriving policy, research or other implications from composite indicators. Before we can be confident of their implications for the comparative performance of countries or regions and establish benchmarks to underpin policy or other decisions, it is essential to understand how innovation indicators impact on innovation processes. This in turn requires indices to be underpinned by a clear and strong theoretical framework.

Second, there are a number of data issues which have to be addressed. To explore the dynamics and evolution of health innovation across countries through an index requires panel data (collected over time). While the availability of cross-sectional data (between countries) is fraught with difficulties, the use of panel data is even more so. Another issue is variation at the regional or local level. Policy makers are interested not only in their national scores, but in whether different regions in their country perform differently.

Finally, policy makers should be careful not to draw conclusions about the relationship between composite innovation indicators and other indicators of policy interest such as health outcomes. While it may be of interest to correlate innovation index scores with those of an index of health outcomes, we must always remember that correlation does not mean causality – introducing a new drug or device may well lead to better health outcomes, but an index will not tell us this.

We are now hoping to create a prototype index for a few countries. This involves investigating data sources and more work on what the index is measuring, for whom and for what purpose.

Anyone interested in finding out more should contact James Barlow at Imperial College Business School, j.barlow@imperial.ac.uk

James Barlow is a Professor of Technology and Innovation Management (Healthcare) at Imperial College Business School and a member of PIRU. His latest book, Managing Innovation in Healthcare, was published by World Scientific in January 2017.

1House of Commons Health Committee (2005) The Use of New Medical Technologies Within the NHS: Fifth Report of Session 2004–2005.
2Department of Health (2006) Best Research for Best Health: A New National Health Research Strategy: The NHS Contribution to Health Research in England.
3HM Treasury (2006) A Review of UK Health Research Funding.
4Strategic Implementation Group & Healthcare Industries Task Force (2007) Innovation for Health: Making a Difference.
5Department of Health (2007) Our NHS Our Future: NHS Next Stage Review – Interim Report.
6Department of Health (2011) Innovation Health and Wealth, accelerating adoption and diffusion in the NHS.
7Dattée B, Barlow J (in press) Multilevel organizational adaptation: Scale invariance in the Scottish healthcare system. Organization Science; Dattée B, Barlow J (2010) Complexity and whole-system change programmes. Journal of Health Services Research & Policy 15, S2, 12-18.
8The Health Foundation (2015) Shaping the Future. A Strategic Framework for a Successful NHS.
9Deloitte (2012) Innovation Indicators for Healthcare in Emerging Countries. Understanding and Promoting Innovation in Emerging Markets.
10PWC (2011) Medical Technology Innovation Scorecard. The race for global leadership.
11Accenture (2013) Measuring the Unmeasurable. A New Framework for Assessing Healthcare Innovation.
12Cravo Oliveira T, Barrenho E, Vernet A, Autio E, Barlow J (2017) Developing a Global Healthcare Innovation Index. PIRU Reports 2017-20. http://www.piru.ac.uk/publications/piru-publications.html

Next step: develop Social Investment Partnerships

by Ben Jupp

The experience of Social Impact Bonds can inform a wider set of relationships to help address social needs, argues a key innovator in the field.

The health and social care system I work in has rarely been in greater need of transformative investment. As demographic and financial pressures threaten to pull it apart, better responses to people’s needs are urgently required.

In my experience, developing more community focused and preventative health and care services will rest on approaches which combine both innovation and rigour. It will rely on strengthening the capacity of services to understand needs, learn from others, adapt and implement change. It will take an approach that looks beyond the annual budgeting of the NHS.

Many Social Impact Bonds seek to change systems
To me, and my colleagues at Social Finance, Social Impact Bonds (SIBs) have always been one tool to support such transformative responses. Take, for example, the Reconnections Service in Worcestershire. It aims to address chronic loneliness, responding to a need that was highlighted by older people in the county themselves.

Establishing Reconnections has involved designing a new service, pioneering long-term cost-benefit analysis on the impact of chronic loneliness, and helping draw together a partnership with a network of half a dozen voluntary and community sector organisations. For the range of commissioners involved, an outcome-based contract was attractive: making payments only if and when the new approach is effective. For the service itself, an outcome-based contract has also enabled flexibility and innovation.

The first year of Reconnections has been hard. Service providers, volunteers and investors have all had to work in ways they had not expected. We have had to learn and adapt. But, slowly, the six charities and community organisations are developing better responses to needs that have often been neglected for far too long, to the detriment of individuals and the health and care system as a whole.

The value of Social Investment Partnerships
Our experience of developing the Reconnections Social Impact Bond and other transformative programmes has also highlighted the value of creating broader Social Investment Partnerships (SIPs) in health and care.

Such Social Investment Partnerships encompass approaches to address social needs which include Social Impact Bonds but also wider forms of collaboration where commissioners, providers and investors share risk and pool knowledge and experience in the design and delivery of services. Highlighting partnerships reflects the need for commissioners, investors and providers to work collaboratively and flexibility, rather than the more adversarial separation between ”purchasers” and ”providers” of services that has characterised the NHS in England for the last 30 years.

For example, in a Social Investment Partnership to improve community-based end of life care, we have been working collaboratively with commissioners and providers for six months to scope out the need jointly. The investors and commissioners have then jointly considered which organisation would be best placed to develop the service; sometimes through a competitive process, sometimes by just building on an existing service.

Alternatives to outcome-based contracts
In such partnerships, we have also found that an outcomes-based contract is only one way to transform services. A focus on the underlying outcomes enabled by a service is important: an outcome-based contract can have significant advantages in terms of allowing service flexibility and providing focus. But for other areas, it’s impossible to capture the full range of people’s needs with a focus on a small number of objectives.

That is why, for example, in our partnerships to develop better Shared Lives social care (the Shared Lives Incubator), we have invested directly in the provider without an outcomes-based contract. Creating the conditions in which someone wants to share their home with another person in need of care is difficult to stipulate in a contract which prioritises one or two impacts. So the investors and commissioners jointly identify the provider, based on their overall qualities and experience. The investor then provides the upfront funding for developing the local service and repayment is made simply as proportion of the service revenue, if and when it grows. In other partnerships, risks are likely to be shared on the basis of a capitated budget – a set amount of funding for the population.

Drawing in investment from a range of sources
Finally, I have found that the concept of a “Social Investment Partnership” draws attention to the need for new resources and capacity and a long-term approach when tackling social challenges, rather than focusing on a particular type of financing. In some of the partnerships which we are building, such as those that develop better employment support for people with health conditions, the source of the funding is a mixture of public sector, charitable and external socially motivated investment. In nearly all such partnerships, we are combining financial investment with access to clinical and operational support. For example, because Health and Employment Partnerships operates across multiple areas, it has the scale to able to employ clinicians and service managers with deep experience of managing similar services before and is helping to manage NHS innovation funding as well as social investment.

I’m proud of the pioneering approaches that my team and colleagues across the sector are engaged in through Social Impact Bonds such as Reconnections. By broadening the approach to developing Social Investment Partnerships, we now have the opportunity to enable a wider range of contractual forms, further blur traditional adversarial relationships, and draw in resources from a number of sources.

These partnerships, whether traditional Social Impact Bonds or a not, are united by a passion for supporting communities and the most vulnerable in particular. They also recognise the importance of both innovation and rigour to achieve social change and ensure that organisations and systems have the capacity to adapt.

Ben Jupp is a Director at Social Finance. He was Director of Public Services Strategy at the Cabinet Office. He has also been Director of the Office of the Third Sector in the Cabinet Office.

Council information brokers could help technology to play vital role in dementia care

By Martin Knapp and James Barlow

There’s no shortage of gadgets. But users and carers require help to make the most of what is available. Industry also should focus more on understanding what people really need from assisted living technologies, argues a new PIRU report.

In 25 years’ time, when many of today’s middle-aged population can expect to be living with dementia, technology will play a big part in their care. It will help them to stay in their homes longer and to avoid being institutionalized. It will also support their carers.

But right now, this isn’t happening enough, even though there is no shortage of gadgets developed by industry. “Why is this?” we ask in our report entitled “The Case for Investment in Technology to Manage the Global Costs of Dementia”. What can be done to accelerate the adoption of assisted living technologies, given the high levels of need among those who have dementia, the stress and isolation of their carers, and the urgency with which the cash-strapped health and social care systems need to control demand for costly treatment and support?

Slow adoption is partly explained by the particular characteristics of those whom we expect eventually to adopt these technologies – people with dementia themselves, their carers – and the local authorities and NHS bodies that might facilitate change. More on these issues in a moment.

Supply-side industry issues
However, responsibility also sits with those who develop the technologies. Research programmes which have looked into the use of assisted living devices continue to find that industry fails to optimise the user interfaces of technology, whether for health and care professionals or the general public.

That said, industry ought not to take all of the blame. It does, given the right circumstances, create and sell low-cost, user-friendly health-related technologies such as Fitbits, which many older people feel comfortable adopting. And most older people have mobile phones. So why doesn’t the large but niche market for people living with dementia show a similar level of growth and development?

Partly, it is because industry has not been as sensitive to this market as it has been to more able-bodied and -minded consumers. It can produce some really helpful, easy-to-use innovations, such as weight-sensitive mats that tell carers remotely whether someone has got out of bed and is walking about. However, as far as more sophisticated devices are concerned, research shows that suppliers often still tend to blame users when technology doesn’t work well. They should instead ask themselves more searching questions that would identify often simple design difficulties in their products.

Suppliers are not necessarily fully cognisant of the need to match technology to the sensory and tactile skills that older people have. Major companies are sometimes naïve about both the basics of what ageing means for individual abilities and the markets for a product. Developers in the ICT industry are still seduced by bigger, more obvious markets in which less trouble has to be taken in show people how to use the technology.

Despite all the effort put into user-led design, there is also still a big gap between research and development and engaging users to ensure that products are right from the outset. There are also plenty of examples where researchers and companies appear to develop a solution that’s in search of a problem.

Lack of support for users and carers
Nevertheless, despite these supply-side concerns, perhaps the biggest issues with regard to dementia care and assisted living technologies are on the consumer or demand side. We can’t be scientific, but perhaps about 80 per cent of the challenge is human, organisational and funding, compared with 20 per cent of the problem being about developing the technologies in the first place.

Funding issues are acute in the UK, where, like in most countries, the health and social care systems are fragmented. That means the costs and benefits of assisted living technologies fall into different silos, which can block take-up of innovations.

From previous research we reckon that an eighth of the costs of dementia care fall on the NHS, a quarter on social care services and the rest on informal carers. As a result, it is sometimes difficult to persuade one system to fund technology when the cost savings might accrue elsewhere. This sharing of cost is particularly difficult in the UK because social care is means-tested while healthcare is not.

An underlying issue, which our report also highlights, is that these technologies can be challenging for everyone, but particularly for older people, many of whom are not competent or confident with what for them are new technologies. That is particularly true for people with cognitive difficulties arising through dementia.

Carers will generally not have the cognition issues – although many carers are quite old themselves, and may have their own health issues – but they may not be familiar enough with the technologies to take on new challenges and tasks within the care setting. Older carers in particular might not have spent much of their working lives using ICT. They may perceive broadband to be expensive, even though the actual cost is not particularly high.

People need reliable advice and support
Often advice and support that’s available from the local authority is neither confident nor competent. The Care Act 2014 requires support for carers, but this duty has coincided with a sharp reduction in social care funding and availability, so that implementation has not matched the ambition.

Our report finds that local authorities are not good at sign-posting technologies to carers or at providing other associated supports. This concerns not just the monetary costs of purchasing technology but the costs of learning new skills. Such help would require fresh investment by councils.

However, this support could be vital to carers who need sound advice around technologies that they themselves might buy. They are not alone in being in the dark: a whole range of people in the system – including commissioners and care homes – lack knowledge about what is available, and what it might achieve.

That is why we argue that the public sector should take on an information broker role, offering a source of unbiased advice on assisted living technologies. A council might even use its purchasing power to negotiate attractive deals with suppliers, so that people feel assured that they are buying the right product at the best possible price.

There are examples of healthcare professionals – including dentists, GPs and pharmacists – providing this kind of advice to vulnerable patients and carers, even if their own service cannot foot the actual bill. The Third Sector has also long provided important consumer support, going right back to the founding of the Consumers’ Association. Some of the charities that support older people are well-positioned to take on such a role.

It may well be that industry – keen to expand its own markets – has a part to play in easing this problem. It should consider how it can support delivery of unbiased information and support. Carers and patients need all the assisted living help that they can get. They can’t wait for another 25 years.

Martin Knapp is Professor of Social Policy, Director of the Personal Social Services Research Unit at the LSE, and Director of the NIHR School for Social Care Research.

James Barlow is Professor of Technology and Innovation Management (Healthcare) at Imperial College Business School.

Both authors are members of the Department of Health’s Policy Innovation Research Unit (PIRU). Their report, written with several other colleagues, “The Case for Investment in Technology to Manage the Global Costs of Dementia”, is published by PIRU.