Author Archives: Paula Fry

Shining a light on the dark side of coproduction

by Kathryn Oliver, Anita Kothari and Nicholas Mays

Advocacy of co-produced research

Many, if not most applied research projects, are undertaken with some degree of collaboration between researchers and ‘stakeholders’, who include research funders, policymakers or practitioners, members of the public and civil society, or other actors such as patients in health studies. This collaboration takes various forms (e.g. consulting on research topics, or working iteratively throughout the research process), and in general, people feel positively about it. Funders often require it, or at least support it, and researchers have argued that it is the most effective way to ensure that evidence is used or translated into practice. Today, there are many forms of collaborative research practice, including co-production, co-design, co-creation, stakeholder and public engagement, participation/involvement and integrated knowledge translation which sit under this umbrella.  They reflect a very diverse set of motivations, activities, identities and discourses about how research interacts with the rest of society. Here we use ‘co-production’ as shorthand to discuss these practices, but acknowledge the diversity and differences in the above.

People argue that collaborative and co-productive research

Facing up to the challenges and costs of co-production

Undoubtedly, making research more relevant and used is the aim for many applied researchers, and their attempts to do so and document this are laudable. There is a need to help uncover if, when, and how collaboration is the best way to achieve this (normative) aim.

Firstly, there is very limited empirical evidence about whether collaborative research does lead to improved uptake of findings – even if we could agree what that looks like in reality.

Secondly, collaborative research brings significant challenges. Collaborative research may be more uncertain, slower, or less innovative than non-collaborative research. These challenges and related costs may be experienced throughout the research process, as the lists below indicates.

Challenges of coproduction

Developing mixed research teams

  • Stakeholders not homogeneous, and can disagree
  • ‘Usual suspects’ can take over, where coproductive discussions are dominated by certain individuals

Framing research questions

  • Stakeholders and researchers may have different priorities and values
  • Useful research can lack originality
  • Research can be co-opted by partners, for example, to justify status quo or historical decisions

Collecting data

  • Researchers may pressure stakeholders to allow their organisational resources to be used to facilitate data collection – e.g. using staff time or applying pressure for site access

Analysing and interpreting data

  • Stakeholders may want to know which participant agreed to participate or what they contributed to the dataset
  • Stakeholders may want to help analyse the data

Formulating recommendations

  • May be little agreement about the importance of research
  • Researchers may be pressed to frame findings in particular ways

Disseminating research

  • Researchers or stakeholders may be prevented from sharing unwanted findings
  • Stakeholders may want to share findings before researchers are ready

Implementing change

  • Tension between advocating for research, or advocating for policy/practice changes
  • Researchers show little interest in providing assistance with implementation efforts

Costs of coproduction

Developing mixed research teams

  • The research process may take more time compared to a traditional research process
  • Shared decision-making is threatened when process dominated by certain voices or interests

Framing research questions

  • Damage to interpersonal or organisational relationships
  • Damage to research careers
  • Damage to researcher independence and credibility

Collecting data

  • Damage to interpersonal or organisational relationships, particularly with more powerful stakeholders

Analysing and interpreting data

  • Violation of research ethics obligations
  • Researcher needs to train stakeholders and format data in an appropriate way to conform with research ethics obligations

Formulating recommendations

  • Findings are misrepresented
  • Damage to researcher independence and credibility

Disseminating research

  • Damage to researcher independence and credibility
  • Damage to the credibility of the research process

Implementing change

  • Can damage relationship with practice or policy colleagues
  • Implementation of research findings fail

Finally, there are other very significant potential costs of co-production, which, in our experience are often unequally borne by junior, untenured, female members of staff.

  • Practical and administrative resources such as the time required to arrange meetings with busy people for whom research is not their primary activity. Managing group dynamics in such meetings can require interpersonal skills with which researchers may not be endowed.
  • Stressful interpersonal interactions: these can be dismissed as mere relational difficulties. We must take the potential for disagreement, conflict, reputational risk and power imbalances seriously as part of the research process. The consequences of mismanaging these are severe.
  • Individual researchers already have to balance their teaching, research and administrative responsibilities. Developing another set of professional skills and networks to create collaborative research projects with real world impact is simply too great a leap for some. The implication of this is that some or all of the co-production activities related to a research project could be led by a specialist in knowledge transfer and exchange rather than by members of the research team. This approach has been advocated and implemented by some.
  • Some feel that taking part in applied, highly collaborative research can lead to researchers becoming, or seeming partisan and biased, or as academic “lightweights” producing little of substance
  • The research outputs themselves may be co-opted to serve the political agendas of others. Co-produced research findings (possibly no different from any research) can be appropriated and used to serve the self-interest of more powerful groups. Some groups lack the skills to engage in the use and promotion of research findings so lose out to more skilled and better connected groups. There is also the risk that co-produced research is more likely than other forms of research to produce findings biased in favour of prevailing norms of what is ‘correct’. This last type of research leads to repetitious, ‘safe’ research

What do we need to do differently?

We may be able to avoid some of these costs, or they may be an intrinsic part of collaborative processes, in which case we need to work out the best balance between costs and benefits.

Armed with a better understanding of the costs and benefits of co-production, those planning new projects should be much better placed consistently to ask themselves:

  1. What is everyone bringing to the table? For example, policy-makers and funders bring money, knowledge of the political context, pressure for answers; researchers bring topic and methodological expertise; the public and patients bring their experiences.
  2. Under which circumstances are these resources needed, for what purpose, and at which stages of the research process? For example, when is it better to have patient representatives articulate the user perspective rather than derive understanding from a systematic review of patient experiences?
  3. What are the costs, and how will they be borne and defrayed by those involved?
  4. How will decisions about the direction of the research be taken, and how will responsibility and accountability for decisions be shared? Will group dynamics, market forces, formal authority or some other basis be used? In turn, how will this be governed?

In parallel, research organisations and funders also need to consider:

  1. How to create (co-create) and support the infrastructure and leadership for co-production
  2. How to provide training in co-production, and help interested researchers and funders take this seriously as a necessary skill
  3. How to reward good practice, and to recognise the work co-production may take even if it does not lead to research impact
  4. How to evaluate the potential impact(s) of co-production
  5. How to ensure that co-production supports diversity and quality in research and policy


Co-production is an exciting approach to research that can, with care, generate truly novel, unexpected findings and impacts. Yet it takes time and investment, and there is still little evidence about how co-production changes research, policy or practice, or how it compares to alternatives. We think more reflection about how, why and when we do co-production would be helpful, as would more discussion about how co-production influences the process of research, and the roles and responsibilities of everyone involved in collaborations. Ultimately, we need to be clear about how and why exactly we work with each other, to transform evidence for society.

This blog grows from a paper by the authors published in Health Research Policy and Systems 2019: 17: 33

Kathryn Oliver is Associate Professor of Sociology and Public Health, Department of Public Health, Environments and Society, London School of Hygiene and Tropical Medicine, UK

Anita Kothari is Associate Professor, School of Health Studies, University of Western Ontario, Canada

Nicholas Mays is Director of the Policy Innovation and Evaluation Research Unit (PIRU) and Professor of Health Policy, Department of Health Services Research & Policy, London School of Hygiene and Tropical Medicine, UK


SIBs may provide components for outcome-based contracting, but they are not a panacea


Public service reform can benefit from outcome-focussed approaches to commissioning. However, we should avoid assuming that Social Impact Bonds are likely to have wide application, explain PIRU’s research team.

Social Impact Bonds (SIBs) are likely to become just one part of the search among policy makers, practitioners and researchers for viable approaches to outcomes-based commissioning of public services. That’s what we conclude in our assessment of the SIBs Trailblazers in health and social care in England, set out in our recent report. Here we reflect on a number of issues raised in our own research and by previous blogs in the PIRU series on SIBs.

Testing, as yet unproven, outcome-based models of commissioning is important for public services, which are often clear about their goals but hamstrung by inadequate implementation including unsupportive funding mechanisms. In health policy, for example, there is a goal to develop healthier citizens – not just to treat more sick people. Reaching that goal could reconcile the ambition to provide better health care with the need to control budgets. However, how can health and social care – and the other parts of the public sector – work together to achieve such goals? We lack a tried and trusted mechanism.

Focus on outcomes-based funding
In this light, the focus that SIBs provide on developing outcomes-based approaches to commissioning services is welcome. SIBs in health and social care have the potential to break through institutional inertia and generate innovations in public service delivery. At best, they can free and motivate front-line service provider staff to focus strongly on the needs and desired outcomes of their clients.

One of the issues holding back public service innovation, particularly in the last decade, has been a lack of funding for prevention as the demands of those currently in need of support have become ever more pressing. Amid austerity and lack of time for contract management, commissioners have found it difficult to think strategically. In this context, SIBs have been welcomed by policy-makers, commissioners and providers, drawn by the promise of upfront funds to develop services designed to improve outcomes in the longer term.

Issues with SIB upfront funds
What’s not to like about that promise? We have two caveats. Some Third Sector organisations lack the capacity to monitor user outcomes well and thus to demonstrate accountability for the upfront funding that they receive. Thus, upfront funding needs to be used not just for service delivery, but also for developing capacity at the provider level in areas such as outcomes measurement and data management.

A second – and perhaps more fundamental – concern about SIBs’ provision of upfront funds is whether this is an efficient way to generate extra cash. The injection of more funds can be seen as a good thing per se, but, it brings with it two further problems.

First, investors are often risk-averse, which can get in the way of innovation, a raison d’être for SIBs and for outcomes-based funding in general. There may also be clashes between the values of private investors and those of the ultimate payer for results – the public commissioner on behalf of the public/tax payer. Additionally, there may be adverse impacts on Third Sector staff’s motivation if the pursuit of specific outcomes is prioritised ahead of meeting the specific needs of individual clients.

Second, many SIBs are set up with the help of intermediary organisations which can add to their cost. As a result, these costs have to be set against the value of any client outcome improvements. The activities of such intermediaries may also inhibit capacity building at the provider level if they assume long-term project management and monitoring roles.

Need for experiments to stretch the SIB model
Ultimately, the public purse foots the bill for SIBs, even if this only takes place after the successful delivery of agreed outcome improvements. Yet, there may well be other more efficient approaches to outcomes funding. We may need to have experiments that stretch the already flexible SIB model and compare it with other approaches. So, for example, Big Society Capital, the Big Lottery Fund, or other socially minded investors could experiment with conditional grants, rooted in outcomes-based contracts, to spur innovation in service delivery, and compare these with SIBs. In her contribution to this blog series, Katy Pillai, Investment Director at Big Issue Invest, signalled that investors are keen to support research into what works best in this regard.

In our experience, most up-front SIB finance does not come from business investors. It comes from philanthropic investors, alongside central government sources such as the Life Chances Fund. For example, the eventual outcomes payments made to investors in the Trailblazer SIBs which we studied were mostly paid for by tax-funded public commissioners upon delivery of the service. Upfront philanthropic funds (investors) were useful and the Big Lottery (which provides a quasi-governmental subsidy for outcome payments) helped to fund the additional effort needed for commissioners to consider and experiment with outcomes payments.

Given how SIBs are largely working this way in practice, would it be wise to experiment with a different SIB model that removes the commercial imperative altogether? This SIB model is, in ways, a variant on the experiments suggested above for socially-minded investors such as Big Society Capital and the Big Lottery Fund. It could amount essentially to organising forgivable loans to Third Sector providers from philanthropic agents, alongside dedicated contract and performance management support. Such an approach would remove the financial architecture of ‘investment’ that surrounds SIBs at the moment but could reduce their transaction costs and accelerate the innovation process.

Clarify values of outcome-based contracting
A big issue when discussing SIBs – or any form of outcome-based funding – is to define their purpose. Are they simply about driving efficiency and making savings to the public purse or are they about transforming the social value of services? Are they a tool to create better monitoring and evaluation cultures in non-profit organisations, so improving accountability and value for money? What are the values being prioritised in such contracts? Thinking and practice in the Asia Pacific region, discussed in the blog by Chih Hoong Sin and Ichiro Tsukamoto, focussed, for example, on a further role for SIBs in transforming the relationship between civic society and the State.

What PIRU and other research has found
So far at least, it has not generally been possible to show that SIBs have produced cashable savings to the public purse. Globally, the main demonstrable success of SIB projects has been in helping marginalised groups that had previously been badly neglected by public services. Here, there was often a baseline of little or no provision, so it was highly likely that anything would have helped in such circumstances. In areas with some previous provision, it is less clear that SIB-funded services, for example, for people with chronic health conditions, have led to superior improvements in health.

Next Steps
The general argument, in principle, for developing outcome-based commissioning and contracting is strong, given the difficulties that policy makers have in matching their goals with the limited funding mechanisms currently available to them. However, we are some distance from resolving all the difficulties in making an outcomes-focus work in public service commissioning and in establishing the case for applying these approaches at scale. So, it is wise to experiment widely and to keep learning.

In this light, policy makers should focus on the components within SIBs that show promise in strengthening outcome-based contracting, while avoiding the notion that the whole package offered by a SIB is necessarily the way forward. SIBs, as currently conceived, may have some dedicated roles, but these are likely to be in particular circumstances (for instance, when outcomes are relatively easy to measure and relatively uncontroversial). The greater value of SIBs and related thinking, in the long run, may be in informing the wider development of other models of outcome-based contracting for mainstream service delivery rather than in replacing these approaches.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.

Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

Japan highlights innovative Asia Pacific model for Social Impact Bonds


SIBs are emerging as a way to reconfigure relationships between the state and civil society in ways that their Western inventors may not have anticipated but may wish to replicate.

Japan’s experience of Social Impact Bonds (SIBs) offers some important insights. First, it challenges the notion – held by some – that SIBs should be seen chiefly as a way to cut costs: in Japan we see a typically Asia-Pacific framing of SIBs that’s more focussed on developing civic society. It’s an approach that may catch on more widely around the world.

Secondly, the Japanese experience demonstrates the importance of institutional infrastructure and culture – and not only availability of finance – to make SIBs work well. SIBs require a complicated culture in which to thrive: Japan shows how it can take time for the various elements to be established.

Japan expressed an early interest in SIBs, becoming part of the OECD Social Impact Investment Taskforce. A five year nationally-funded research project, based at Meiji University in Tokyo, looking into developing SIBs for Japan is almost complete.

On the face of it, Japan might look to be in the same boat as, for example, the UK. The country’s national debt at the end of the last fiscal year was 254% of GDP, compared with the UK’s 88%. So, one might think, Japan would use SIBs to drive down spending and increase efficiency.

In fact, although Japanese rhetoric initially focussed on fiscal savings, interest swiftly shifted to SIBs steering improvements in societal wellbeing that might require more, rather than less, public spending. Japan has also been much more like other Asia-Pacific countries where SIBs have been seen as a means to reconfigure relationships between the state and civil society and focus on long-term issues of public health.

Past and existing projects have looked at, for example, how to create learning supports in deprived areas for NEET young people – those not in education, employment or training. A nationwide health SIB has just been launched focussing on better managing or preventing long-term conditions such as dementia, diabetes, cancer, cardio vascular disease.

Japan’s government has shown a high level of interest in SIBs. It’s seen its role largely as being to signal interest and legislating to make money available. In December 2016 Japan harnessed dormant bank accounts, like the UK has done, to set up the equivalent of Big Society capital, which is envisaged to go live from 2019. In addition, the Ministry of Economy, Trade and Industry (METI) and the Ministry of Health, Labour and Welfare have provided pilots around the country with funding for operations.

So pilot programmes have tended to suffer not from shortages of capital. Their problems have, rather, concerned lack of measurement or been because service providers are short on thinking or capacity to deliver outcomes-based contracts.

Institutional weaknesses in Japan
Japan has been institutionally handicapped in developing SIBs in a number of ways. The country has a weak public management culture: transparency, a focus on efficiency and on prioritising clients’ views of outcomes are less developed than in the UK.

The country is only slowly creating the building blocks for outcomes-based markets. The UK government has supported the creation of outcome payers: SIB investment funds such as the Innovation Fund, the Fair Chance Fund, Commissioning Better Outcomes Fund, and the Life Chances Fund. Britain has invested in developing an evidence base that can support outcomes-based contracting. There is also legislation that directly or indirectly supports SIBs, including the Public Services Social Value Act and provision of social investment tax relief. Japan has had a much less developed basic infrastructure.

In comparison, the UK’s voluntary – not-for-profit – sector is also more robust, diverse and heterogeneous, including organisations of different sizes. There are strong umbrella bodies such as the National Council for Voluntary Organisations. Japan has a long history of social sector organisations but the flowering of the not-for-profit sector is more recent, encouraged in the past 20 years by legislation liberalising regulation of the sector.

Many Japanese organisations remain small and hyper-local, which raises questions about their capacity to engage with SIBs. The country’s not-for-profit umbrella bodies are also relatively weak in terms of their lobbying capacities and the extent of their networks across the sector.

There are fewer charitable foundations than in the UK. So, social sector organisations in Japan largely rely largely on government funding, whereas UK social sector organisations tend to have more diverse funding. This financial dependence makes it harder for social sector organisations to question government.

Problems measuring impact
These days, the UK social sector is familiar with measuring and demonstrating impact. Japan’s social sector organisations are less experienced in measurement and, where they are, their focus tends largely to be on activities and inputs.

Lack of diversity in markets – in terms of service providers, investors and intermediaries – also hampers the growth of SIBs in Japan. Until 2017, when the Yokohama SIB was developed, all pilots were driven by the Nippon Foundation or its subsidiaries. So a single, large player was trying to nudge the market along.

There are significant differences between the UK and Japan in the maturity of intermediary markets. The UK has moved from a virtual monopoly among a few intermediaries in the early days of SIBs to a growing, diverse market, bringing in many different skills. Intermediaries no longer offer only financial backgrounds. There are people with commissioning support experience as well as good provider engagement experience. The same cannot be said quite of Japan, where the emergent intermediary market remains in its infancy.

From the commissioner side in Japan there remains a poor understanding of SIBs so it is difficult to collaborate with providers about constructing SIB models. As a result, commissioners may not know what to specify. They tend to identify general outcomes which may lack meaning and be unachievable within the timescales.

Building underdeveloped civil society
Nonetheless, Japan’s vision for SIBs is innovative as it tries to use them to place individuals at the heart of public services that have tended to be dominated by centralised bureaucracies. As in China, SIBs offer Japan a model that facilitates an underdeveloped civil society sector to grow and play a more visible role. SIBs allow responsibility to be devolved downwards while retaining central control through outcome-based funding. SIBs help square the circle for centralised governmental structures that wish to devolve responsibility while retaining policy direction.

Western countries such as the UK, where SIBs originated, would be wise to take stock of this evolving emphasis. Just as football played in Brazil and Spain has turned out to be more exciting than the game played by its inventers, SIBs, in their cultural adaptation, may prove to be a more transformative model than originally envisaged in Britain.

Attractions of Asia Pacific model
This adapted model could turn out to be an attractive export even to its inventors. Austerity – the ground into which SIBs were originally planted – will not last forever. SIBs may, in the longer run, offer opportunities for a more ambitious and hopeful agenda of improved well-being that may be politically realistic in more prosperous times. Such reinvention and rebranding of SIBs might also make them more attractive to their Western critics. The Left in Britain and US has tended to dismiss SIBs, suspicious of them as vehicles for commodisation and privatisation of public services. A revisioning of SIBs might suit a more left-wing agenda.

This SIBs model with a different emphasis might also offer an approach to reform in Britain where big providers such as Serco and Capita – focussed on driving down costs rather than on quality delivery to the person – are edging out smaller local players. Since 2010, many small civil society organisations have collapsed in the UK and others have had to merge as size has become the key to survival. Might SIBs eventually offer a way to address this trend and recognise financially the value of local deliverers with smaller caseloads but higher quality?

As the SIBs story unfolds, the purposes and future of this approach to service delivery remain in flux. Analysing the Japanese – and wider Asia Pacific – approach is a reminder that we may not have scratched the surface of SIB’s potential for achieving a wide range of agenda.

Dr Chih Hoong Sin is Director of Innovation at the Office of Public Management, London, UK

Ichiro Tsukamoto is Professor of Public Management at the School of Business Administration, Meiji University, in Tokyo, Japan.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

We should ask three big questions about SIBs


Evaluation should test key SIB promises to government: greater collaboration, prevention and innovation, argue two researchers from Oxford University’s Government Outcomes Lab.

To discover whether Social Impact Bonds work, we would be wise, first of all, to ask: “Work for whom?” Our response is: “For public service commissioners.” This answer acknowledges a reality: if SIBs don’t make sense to government, they are unlikely to be more than a passing fashion.

This standpoint cuts through a myriad of promises that typically surround SIBs and other payment by results programmes. It also helps focus how to evaluate them. We should measure SIBs by looking at what, if anything, they can deliver that public service commissioners seek.

That’s what we’re doing in the Government Outcomes Lab – GO Lab – at the Blavatnik School of Government, where our work is currently co-sponsored by the School and the UK government.

The dominant narrative around SIBs, born in an age of austerity, is that they can deliver cashable savings to government. Of course, every government seeks such dividends. However, this claim, far from proven – at least at scale – may or may not be true. Either way, we believe SIBs go well beyond promising the efficiency and entrepreneurialism claimed by conventional outsourcing. SIBs make a more fundamental offer that should be rigorously tested: that they provide an opportunity to remake the public sector itself.

Public sector reform has sat at the heart of every government’s agenda – Conservative and Labour – for nearly 40 years, since the election of Margaret Thatcher in 1979. Its goals – aside from making savings – have included three principles that SIBs directly claim to address.

SIBs address 3 principles of public sector reform
The first principle is to improve collaboration in the commissioning and provision of services so that the collective effort is better focused on the needs of the service user.

The second principle is to shift activity to a more preventive footing, so that service provision is moved upstream of social challenges and towards earlier identification of risk.

The third principle is to make innovation less risky for commissioners so they move to evidence-based procurement, decommissioning underperforming services and becoming better at reaching those, often experiencing the most complex mix of social issues, who may be poorest served by existing provision.

We should, therefore, question, in our evaluations, whether SIB models are improving collaboration, shifting services towards prevention and creating spaces in which public service commissioners and delivery organizations can innovate.

Principles largely untested by evaluations
Our review of evaluations suggests that they rarely explicitly or rigorously test any these three principles by comparing a SIB commissioning approach with a grant, fee-for-service or even in-house delivery for a given population. Evaluations also face the “doughnut challenge”. That’s the difficulty, particularly with impact evaluations, in teasing apart the effects of the commissioning architecture (the dough) from the effects of frontline services operating in different ways (the jam). Indeed, most evaluation material is quite descriptive and exploratory. It doesn’t typically offer or test a mechanism by which a particular SIB is expected to work in practice.

Collaboration is seen
However, where we have tried to read these three principles into the evaluations that have been produced, we see glimmers of each logic being present in SIBs. So Peterborough’s SIB – focused on reducing ex-offender recidivism – saw front-line organisations pull together in cohesive ways, wrapping around the service user.

We’ve also seen multiple commissioners joining forces as, for example, where the Department for Education, Cabinet Office and the Department for Work and Pensions came together for a fund-based approach to support young people vulnerable to disengaging from school and work. We see top-up funding approaches from the Cabinet Office to acknowledge that there might be wider public service benefits springing from a SIB that are not captured within the outcome payer’s domain.

“Step-down services”, more than prevention
As far as prevention is concerned, very early intervention programmes are rare in the SIB field, but there is a focus on “step-down” services. The Birmingham Foster Care SIB is a good example that is already demonstrating cashable savings.

A growing number of SIBs try to de-escalate situations of family conflict or address unstable conditions for children and young people. Typically, for example, the Department for Work and Pensions has worked with people only once they are employed. Now, through SIB programmes, the Department, through the Young Engagement Fund, is becoming involved in programmes that try to understand and anticipate future needs. It is working with school children to prepare them for a smooth transition into work.

Space for innovation
There are also examples of SIBs supporting innovation and helping to dissipate cultural inertia. Typically commissioning has a cycle – there’s the planning stage, the doing phase and then there’s the evaluation. A SIB can help reduce risk at any or all of these stages, allowing SIB commissioners flexibility to think differently about their work. So, for example, with an innovative programme such as the GLA Rough Sleeping project, a SIB allowed commissioners to ask of themselves: ‘Do I know how to be the best contract holder, or would I benefit from bringing in a third party that might have a better appreciation or knowledge base about that side of things?’

SIBs can also enable a certain amount of experimentation by providing quicker feedback through performance management. It’s this feedback that allows frontline organisations to change the way they are doing things to better meet the needs of their clients. A well-designed SIB also creates impetus for high quality impact evaluation within a payment mechanism. That inclusion can, in the long run, only help to improve the quality of future commissioning.

Time to test public sector reform potential
In conclusion, we cannot demonstrate, on the basis of existing evaluation evidence, that SIBs fulfil the many and varied promises that their champions make for them. But observing SIBs through the lens of public sector reform – and their possible role in it – we offer an approach to evaluating whether they might be a tool for remaking the public sector.

Eleanor Carter and Clare FitzGerald are research fellows at the Government Outcomes Lab (GO Lab)at Oxford University’s Blavatnik School of Government.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

Investors need rigorous assessments of Social Impact Bonds


A major investor highlights the vital role that research and evaluation should play in developing this form of outcomes funding.

Evaluation and research into Social Impact Bonds (SIBs) is a hot topic for Big Issue Invest. We are one of the UK’s leading social impact investment firms, having invested in approximately 350 charities and social enterprises since 2005, in our quest to dismantle poverty and create opportunity.

Our investments support areas such as access to housing, financial and social inclusion, mental and physical health and well-being, and employment, education and training.

We have made several SIB investments since 2012 and have watched the market develop. Our first-hand experience is that there is more work to be done to refine the model, but there have been impressive outcomes for programme participants and the charities delivering the contracts in which we have invested.

How we use research and evaluation in SIBs
We aim to address the structural challenges of SIBs and maximise their individual and collective social impact. Research and evaluation can help in this goal. Many commissioners and service delivery providers are unfamiliar with SIBs and I often direct them to impartial, well-informed research to build awareness and understanding.

Discussing Big Issue Invest’s learnings and experiences from individual programmes with evaluators helps us to contextualise the situation and identify emerging trends in this rapidly-developing field. These partners can develop the tools and models to test and critique our theories and insights ‘from the field’ and evaluate the wider market, whereas our frame of reference is often limited to our specific investments.

Research in social investment and SIBs is at a nascent stage. It is crucial that the right foundations are laid today to enable good-quality ‘market’ level analysis in the future. Consistency and rigour of approach will pave the way for systematic reviews and meta-analyses, essential if outcomes-based approaches are to become commonplace in the commissioner and policymaker toolkit.

We also welcome more quantitative approaches and ambitious evaluations that compare SIBs to traditional fee-for-service mechanisms or payment by results more broadly. SIBs are often conflated with outcomes-based approaches in general, which makes it very hard for investors to assess – and improve – their social impact.

Funding is, of course, needed to allow this work to take place. At the moment, evaluations are too often the balancing item in a very limited budget, constraining their ambition. We wholeheartedly support the calls for a ring-fenced fund for SIB evaluations, recognising the value of the output to commissioners, central government and potentially also philanthropic funders who might seed the fund.

Understanding what, why, how
We are an impact investor: outcomes are our reason for being, not just a by-product of our investment activity. Social impact due diligence underpins every investment decision we make.

We look closely at the theory of change for each SIB. There needs to be a coherent and credible hypothesis for how outcomes will be improved for the programme participants and – beyond that – how the programme could help to tackle the underlying issue through, for example, earlier intervention. We interrogate potential perverse incentives in order to mitigate them. Research and evaluation from previous programmes helps us to do this: we rely on it to validate the causal link between inputs, outputs and outcomes and complete our due diligence of the intervention.

It’s important that everyone involved reviews the theory of change periodically after the contract launches. One of the strengths of SIBs is that they shine a light on what works and what doesn’t, enabling real-time improvements and sharing of learnings for future contracts. If we scrimp on monitoring and evaluation, we undermine the programme and indeed the SIB model.

Evaluations therefore need to be robust, relevant and timely. We want to understand not only the results achieved but also why they are (or are not) achieved and how we can replicate and improve on them. That might be a programme evaluation or an impact evaluation, or qualitative or quantitative approach, depending on the context but we certainly need more than outcomes verification.

We seek insights into the drivers of success so they can be reflected in future projects. The GLA Rough Sleeping SIB in 2012 was divided into two lots awarded to separate providers, one funded by Big Issue Invest. We know the absolute outcomes achieved by the programme but would like to dig deeper into whether different operational or investment approaches had a bearing on success.

We are keen to work with the evaluator community to design the evaluations and contribute to them. It’s important to be confident that Big Issue Invest’s loan has achieved its social objectives – and those of our investors in turn – so we are a consumer of evaluations as well as contributors to them.

We are one of very few organisations that has worked across several SIBs in different regions and policy areas. We can contribute data, insight and practical experience and welcome the opportunity to do so. At a practical level, we can coordinate with the evaluator to minimise the data collection burden on the service provider’s staff and the programme participants. If we can bring the evaluator in to the design phase early, we can also incorporate their evaluation into the delivery model early to avoid duplication or complication down the line.

Using data and analysis to target interventions
Reliable data and analysis is essential to high-quality SIB design. For example, we are involved in an ‘edge of care’ SIB for young children where it is unfeasible to roll out an intensive (in other words, expensive) intervention to all children on social services’ radar. Rather than only work with children on the very cusp of care – when it is often too late to reverse their trajectory or the trauma they have suffered – a researcher is working with commissioner data to identify early risk factors that increase the child’s propensity to enter care. The programme will be targeted towards these high risk children as well as those on the very cusp of care. This allows the commissioner to fund an early intervention service that is also cost-effective, often challenging in outcomes-based commissioning. There is huge potential to harness data and analysis in this way to design preventative services.

The value of timely feedback
Speed of evaluations is a challenge. Evaluations are valuable to SIB stakeholders when developing follow-up programmes and carrying out due diligence. If investors have a good level of confidence in the achievable outcomes, the cost of their risk capital should be lower. That is in everyone’s interest. It doesn’t encourage evidence-based commissioning if the evidence is released after the next programme is launched!

Midline and end-line reviews as part of a formal evaluation are, of course, extremely important but they are not enough. Outcomes-based programmes also need shorter, informal feedback loops, preferably involving the evaluator. Early results and findings can be used to improve programme delivery – but not if they are shared in an end-of-programme evaluation that takes a year to publish. Ideally, we’d like a quarterly or a six monthly check-in with the evaluators that can identify and unpick performance and its drivers.

We recognise the tension between this approach and concerns that the evaluation will influence the programme outcomes. A balance needs to be struck. SIBs support people with complex needs who deserve the best possible chance of better life outcomes, so although evaluation rigour is crucial, we owe it to them to make the intervention as effective as it can be.

Importance of counterfactuals
Three inputs are usually needed to assign a value to a SIB outcome: (1) the projected costs to deliver a programme (preferably validated through a competitive procurement process); (2) the costs per outcome achieved under comparable programmes, if known; and (3) the savings case (the projected benefits for the commissioner). If you don’t have an understanding of what would have happened anyway, at least one of these calculations will be flawed. That’s why we can’t afford to disregard the counterfactual.

That doesn’t mean that every SIB needs to link payments to performance compared to a counterfactual, measured by an RCT. There are lots of factors to consider when designing the payment mechanism and there is no single ‘right’ approach. However the counterfactual can always be taken into account. Under a rate card approach, the rates should be set after considering deadweight – even if the assessment is imperfect, it is better than ignoring it completely. The counterfactual can then be assessed in the programme evaluation and used to inform the pricing of future contracts.

I am not saying SIBs should be commissioned only if there is perfect data to value the counterfactual. Rather, I am emphasising the need for new approaches that measure outcomes and cost-per-outcome to allow commissioners to make evidence-based decisions in future. Big Issue Invest is trialling approaches that allow an outcomes-based contract to be launched with imperfect information, while ensuring checks and balances limit windfall gains and losses and include mechanisms to tackle the information gap.

One option is to run an initial ‘discovery phase’ of the contract for one to two years. The discovery phase outcomes pricing estimates the counterfactual, but sets parameters to ensure that no party makes excess gains or losses. In this way, the partners have the opportunity to implement the SIB delivery model. During this time, outcomes and the counterfactual are measured rigorously. The data and analysis is then incorporated into a revised payment mechanism for the rest of the contract, after which point it operates like a ‘standard’ SIB. This approach bridges the knowledge gap without delaying a potentially high-impact programme or risking inequitable risk and return.
Where next?

SIBs bring together different worlds. The success of SIBs is dependent on partnerships where the whole is greater than the sum of the parts. They require new ways of working for everyone involved – for investors, providers and commissioners. I expect they can seem strange to evaluators as well. Forging new links and understanding the perspectives of others is crucial.

We are starting to see these worlds come together and collaborate for better outcomes. There is more interaction and understanding between researchers and evaluators, policymakers and budget holders, delivery organisations, and investors. It is early days but the outlook is promising.

Katy Pillai is Investment Director, Big Issue Invest: @katyjones | @bigissueinvest

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

Academics can show governments how to evaluate SIBs more rigorously


A wide range of approaches can help identify causality and effectiveness even in complex environments.

We can – and we should – improve our evaluations of SIB and Payment By Results (PBR) programmes. They should focus more on causality, rather than simply contract compliance or implementation.

If we don’t focus on attribution, it will become hard to demonstrate that SIBs are more than a series of interesting pilots. We’ll miss the chance to test an alluring proposition – that SIBs could transform the large scale commissioning and delivery of health and social welfare programmes.

Getting evaluation right in this field is not – as some might suggest – intrinsically challenging. SIBs and PBR projects do not create unusual difficulties for evaluation techniques. We have the knowhow – sophisticated, diverse tools are well-developed that could settle most questions thrown up by SIBs. The real issue is: will those who champion SIBs expose such initiatives to the full rigour of the evaluative tools that exist?

Academic responsibility
The academic community can help ensure that rigour. Management consultants, contracted to perform evaluations, tend to provide what governments specify, which, so far, has been limited and fallen short of what’s required. Academics could set out a wider, more exacting range of evaluation options that are more suitable. We should show policy makers clearly how better evaluations could be achieved, particularly if the case for widespread adoption of SIBs is to be made.

This difficulty in properly assessing the impact of SIBs seems to be a particularly British problem. In the United States, most SIBs have been accompanied by fairly rigorous counterfactual evaluations, including randomised control trials (RCTs). There, the credibility of the SIB model among commissioners and investors has required demonstration of its ability to deliver tangible outcomes. This may be because, in the US, more funding has come from wealthy individuals or private foundations, with an investment ethos. In Britain, funding tends to spring from philanthropic organisations which seem interested in testing concepts over categorical outcomes.

Evaluations are too based on performance management
Whatever the reasons, SIB pay-outs in the UK typically rely more on performance management information to demonstrate the achievement of outputs. Supporters of this approach say that complicated counterfactual evaluations add to the already high transaction costs associated with SIBs. That’s understandable for individual SIBs. However, cumulatively, this approach hinders the quest to find out whether SIBs really work. It undermines the case for wider roll-out.

Evaluations can and should answer two major questions about SIBs. There’s “attribution”: whether SIBs actually achieve the outcomes desired. Second, we need to understand SIBs as a mechanism and establish how effective they are compared with other models of commissioning. This is important because there are less expensive, less complicated methods than SIBs for commissioning services in this field.

The attribution issue has become unnecessarily mired in a polarised debate about whether RCTs are suitable for SIBs projects. Opponents contend that RCTs are not particularly useful in this field because SIBs interventions tend to take place in highly complex environments. While it’s true that these interventions often occur amid complexity, that actually strengthens the case for RCTs. It becomes even more important to understand whether an intervention is indeed responsible for any of the impacts being observed.

Testing theories of change
Good RCTs would strengthen SIBs evaluations because they would be theoretically informed. They would start with a theory of change setting out the potential causal mechanisms that are of interest. In contrast, many SIBs evaluations rely on contractual frameworks and demonstrating whether they have worked, rather than testing hypothesised causality. Most good RCTs today are also accompanied by high quality implementation evaluation. So they have a dual strategy.

Well organised RCTs avoid “one-shot” design. They are actually a sequence of evaluations that build by testing, at a granular level, particular moderators of change, rather than simply focussing on the overall social outcome and trying to come to a one-shot conclusion. This is how, in reality, even medical research works. You don’t do a single RCT. You build from small scale studies through to larger scale studies.

Sequences of evaluations are good
The wider evaluation world is focussing more on sequencing evaluations and ensuring that tools employed are appropriate to the point of a programme’s development. This avoids problems that one shot evaluations can create: that you evaluate too early; that the throughputs you were promised never arrive; that you end up developing an evaluation design which is underpowered to identify the changes that you’re looking for; there are inconclusive findings that have cost a lot of money but don’t provide the hoped for insights.

I advise against the one shot model. Instead, we like to start evaluations early without diving straight in with an RCT. We focus on developing a sequence. That’s the strength of the Education Endowment Foundation evaluation model. It begins with small scale pilot studies that focus on theory of change and early implementation, then efficacy trials that are more like a small RCT, leading up finally to effectiveness trials. Only at that point – when causalities have been established – is control finally handed over to implementers.

Building commissioner confidence
This sequential approach gives commissioners confidence. You’re saying to them that this isn’t a “one shot, put all your money on the table up-front” model. It’s about gradually building knowledge and providing gate-keeping points where a commissioner can ultimately say: “This isn’t working, we need to rethink. We may need to reinvest or, even disinvest.” That’s helpful to commissioners, especially if they are being asked to back innovation that feels risky.

Small ‘n’ designs
In some cases, RCTs are not possible, but there are many alternative models of impact evaluation that could be considered for SIBs. “Small n” designs provide ways to think about causal attribution where a programme does not have sufficient numbers to allow a traditional impact evaluation design. Process tracing is an example of “small n” design, where one uses theory to identify critical points in a change process that need to be tested. Then one selects cases to test these critical points, using interviews and observations of what’s going on. This Popperian approach acknowledges that there is no absolute objective knowledge. However, it can find ‘smoking gun’ evidence that strongly suggests causality, even if that may not amount to absolute proof.

These process approaches that search out causality would be an improvement on current tests of some SIB or PBR programmes which, if they can’t do an RCT, tend to opt for process/implementation evaluations that are less demanding – usually interviewing stakeholders and writing a report, but lacking a more theory-driven approach.

More rigour is needed
I’ve set out ways in which SIB and PBR evaluations could be improved by RCTs or hybrids that avoid the unnecessarily polarised debate between the pro- and anti-RCT lobbies. Beyond RCTs, there are other approaches to evaluating causality, suitable in instances where there are small numbers of cases. We should learn from this wider discussion of evaluation techniques. Academics owe it to those investing and working in SIBs to ensure that policy makers adopt a rigorous approach to evaluation. We need to know what works and what doesn’t if SIBs are ever to be widely adopted.

Chris Fox is Professor of Evaluation and Policy Analysis and Director of the Policy Evaluation and Research Unit at Manchester Metropolitan University. He is co-author of “Payment by results and social impact bonds: Outcome-based payment systems in the UK and US”, published by Policy Press in February 2018.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

Impact bonds could offer a paradigm shift towards more effective public services


Social and Development Impact Bonds require enormous effort for the partners involved, but they have a potential to transform the financing and delivery of social services across the globe.

In winter 2015, Courtney arrived at Frontline Services, a not-for-profit US organisation that helps citizens in Cleveland, Ohio. She was 28, living in a shelter for homeless women, struggling with mental health and substance abuse issues and parenting three young children who were in the custody of the county.

Courtney had just about given up hope that she would ever be able to care for her children on her own. Until that point, the county caseworker assigned to her family had little incentive to reunite Courtney and her children because the caseworker’s primary job was to protect the children.

Living with a birth parent is almost always better for a child’s development than foster care, provided the home environment is safe and healthy. Nevertheless, before entering Frontline Services, Courtney had few ways to change the trajectory of her children’s lives. Fortunately for her children, she was walking that day into a social services experiment, one of only seven similar experiments in the US at the time. In this experiment, a social impact bond (SIB) – designed to “pay for success” – the county’s government had pledged to repay private investors for successful reductions in out-of-home placements for children whose primary caregivers were homeless.

This incentive meant that Courtney was assigned a caseworker dedicated to her – someone who would look at her particular circumstances and tailor a plan to help her turn her life around and unite her with her children. Courtney’s caseworker could work across county service providers to identify the right mix of services for her.

The SIB meant that a dedicated group of stakeholders was meeting regularly across government and non-government entities to focus on one thing – reuniting Courtney with her children, and doing the same for other families in similar circumstances.

SIB contracts focus on outcomes, so service providers tailor their services to what works for the target population. They helped Courtney to address her debts with classes in financial management and offered family counselling.

As a result, Courtney was able to reunite with her children, enrol them in supportive school environments and stop the cycle of dependency on the foster care system. The result was not only a better family outcome, but also a reduction in the enormous costs to the county, had Courtney’s children remained in the county’s care.

Impact bonds are changing developing countries
Meanwhile, nearly 12,000 km away, in a village in rural Rajasthan, India, lives a 13-year-old girl, named Punam. She comes from a poor family – her parents are labourers. Although Punam started school at age seven, she became one of India’s three million out-of-school girls, when she was forced to drop out to tend to her family’s goats.

In the same year as Courtney arrived at Frontline Services in Cleveland, Ohio, a field co-ordinator, working for an organisation called “Educate Girls”, arrived in Punam’s home in Rajasthan. He spoke with her parents, explained the benefits of educating Punam and tried to convince them to send her back to school.

Even after multiple attempts, the parents didn’t agree to send Punam back to school. The Educate Girls caseworker returned some weeks later with a volunteer from the community but again failed to persuade the parents. “What benefit will it give her or us?” they asked. “She will eventually marry and her responsibilities will revolve around doing household chores, assisting in farming life, raising children and taking care of her family.”

Nevertheless, Educate Girls made a further attempt to encourage her parents to let Punam attend school. This time, they asked the school’s principal to join them in a final visit to her home. With this added influence, Punam’s parents agreed finally to sending their daughter to school.

Why was an impact bond so important in this case? Because the contract was based on the achievement of outputs and outcomes, Educate Girls field-staff were empowered to innovate at the field level, trying to find solutions for getting Punam into school. Now, two years later, Punam, and many girls like her, are enrolled in and enjoying school thanks to Educate Girls and this Development Impact Bond (DIB), based on the same principle as a SIB, but with a third-party outcome funder, instead of the government.

These two stories capture the real human benefit that can emerge from outcome-based contracts such as SIBs or DIBs.

How impact bonds work
Let’s just re-cap for a moment how impact bonds actually work. In an impact bond, private investors supply upfront capital to service providers to deliver an intervention or program to a population in need. Upon the achievement of a set of agreed-upon results, the investors are then repaid by an outcome funder. With a SIB, this outcome funder is the government. With a DIB, outcomes are financed by a third-party organisation, such as a foundation or donor.

Since the launch of the first SIB in the UK in 2010, the impact bond market has grown exponentially. Last year, some 32 new contracts were signed. As of January, 2018, there were 108 contracted impact bonds (103 of them SIBs, 5 of them DIBs) across 25 countries, along with many more in design. All but one of the 103 SIBs were in high income countries: last year marked the contracting of the first SIB in a low- or middle-income country, the Workforce Development SIB in Colombia.

Most (42) SIBs are in the UK, the country that pioneered the impact bond model with the Peterborough SIB in 2010. The results of that SIB – aimed at rehabilitating ex-offenders – were released last year: reoffending of short-sentenced offenders dropped by 9 percent and the investors were repaid in full. The US has also established itself as a player in the field, coming in second with 19 impact bonds.

The five DIBs include Educate Girls in India, one for coffee and cocoa production in Peru, as well as one for physical rehabilitation across three countries in West Africa, a poverty graduation program in Kenya and Uganda, and the recently launched Utkrisht impact bond for Maternal and Newborn Health in Rajasthan, India.

Characteristics of SIBs
Most SIBs contracted globally are in the employment field, followed by the social welfare sector, which includes programmes to reduce homeless – “rough” – sleeping, or reduce out-of-home placements as in the case of Courtney and her children. Other areas for SIBs are health, criminal justice, education, the environment and agriculture.

Probably about 30 or 40 impact bonds are in design in high-income countries while more than 20 are being designed in low and middle-income countries. We see some difference comparing high income and low or middle-income countries. The majority of impact bonds in the latter are in the health sector, followed by employment and, then, agriculture.

What do these impact bonds look like in terms of size? The smallest one, in terms of beneficiaries, reaches 22 individuals – that’s in Canada. The largest one reaches 650,000 individuals in Washington DC, which is an environmental impact bond focusing on developing infrastructure. It is perhaps a little bit unfair to compare that one in terms of size with the rest of them because it’s a city-wide programme. The next largest in size is the Maternal and Newborn Health DIB in Rajasthan, India, with 600,000 potential beneficiaries.
However, the median impact bonds are reaching about 565 individuals, so they are quite small. Capital commitment of bonds ranges from $80,000 to $25 million. Again, that $25 million is the one in DC. The average is about $4 million and the total upfront capital invested across the impact bonds is over $300 million.

Who is benefiting?
The bonds mostly target marginalised populations, including women affected by violence, young migrants, single mothers, with a few for ex-convicts, vulnerable and young people, people diagnosed with mental health conditions, refugees and individuals with physical disabilities.
What do we know, eight years in about their performance? There have been some outcomes achieved and payments made, such as in the case of the Peterborough SIB, mentioned earlier. In an Australian SIB, 203 children were reunited with their families and the return to investors was nearly 16.5 per cent over the four years of the scheme.

Shifts in public programme behaviour
However, perhaps the more interesting observation is a real shift seen in government and service providers to thinking about outcomes as opposed to paying for inputs. Impact bonds are also driving performance management so service providers are introducing or improving systems of performance management in their programmes.

Impact bonds are incentivising collaboration, between the public and private sectors, but also across government, vertically and horizontally. They are building a culture of evaluation because outcomes must be measured and monitored. Most impact bonds so far have been focused on investment in preventive interventions. There has also been some reduction in risk for governments, which have not paid for outcomes that weren’t achieved.

What are we not seeing so far? It had been hoped that impact bonds would lead to an influx of additional private funding. However, given that government or outcome funders ultimately repay the investors, then that’s not really more money for a particular social service. Impact bonds have also yet to achieve change at scale: the majority are reaching very few individuals and are fairly small in terms of investment.

Many thought that impact bonds would focus on experimental interventions. So far, we haven’t seen that: investors have been unwilling to take that risk. We’re seeing SIBs used in the middle phase of development of interventions, rather than at the “seed” or “at scale” ends of the process. However, the flexibility that service providers are allowed in terms of their service delivery has the potential to encourage innovation. It’s also probably too soon to say whether or not impact bonds can achieve sustainable outcomes in the long run through the systematic change that’s happening but it does appear that the partners currently involved have indeed shifted their thinking.

Challenges of impact bonds
What are the challenges? This is a new form of government contracting, a new way to do business. Co-ordination of all of the stakeholders is difficult: sometimes they don’t understand each other well; just getting all those people around the table can be really difficult. There can be some political constraints and legal barriers.

Key questions remain. Can impact bonds be used at scale? Are they more effective than input-based based financing or traditional payment-by-results? Do the actors in social service provision have the capacity to adapt to the demands of financing tied to results? Can they manage the rigorous focus on performance management that this is likely to entail?

Next steps
It is worth considering what would be needed to expand the use of impact bonds or, more generally, payment by results. The evidence base needs to grow and there is a need to collect more information on services that work and on their costs. Also, potential outcome funders and investors need to be educated in not only the potential of an impact bond approach but also the challenges. There needs to be supporting legislation and regulations to facilitate paying for outcomes both at a country and local level but also within organisations.

To achieve scale, countries could establish outcome funds for particular sectors. The UK government has launched seven outcomes funds and efforts are underway in the US to develop outcomes rate cards. There are several outcome funds being developed to tackle tough social issues in the developing world as well. These would allow the outcome funder to set prices for desired social outcomes and then to contract with service providers to deliver interventions to achieve those results. Global investment funds would also benefit from contributing to this new financing mode.

In the US, $800 billion are spent annually on social services. Only one per cent of that spending is evaluated for effectiveness. In the UK, £220bn was spent on social and health services (2015/2016), yet we know very little about the effectiveness of that expenditure. Thus we need more empirical research which asks: “What do impact bonds achieve, compared with input based financing?” It is also important to know how well impact bonds perform compared with traditional results-based financing. These are both hard questions to answer rigorously and will take some time.

Impact bonds and global problems
The social and environmental problems that we face at a global level are enormous. It’s estimated that $1.4 trillion will be needed annually to achieve the Sustainable Development Goals by 2030. There is little evidence that such complex problems can be solved by continuing the same old failed approaches.

Investing in preventive measures can avoid higher costs down the road and make the public and civil society sectors more efficient. By paying for outputs and outcomes rather than paying for inputs that have unknown outcomes, spending should be more effective.

Impact bonds may not be the right solution to every problem. However, they do represent a long-overdue, paradigm shift. They’re a means to an end, an opportunity to think about, and hopefully produce, systematic change. At the very least, they may be the stepping stone to establishing the monitoring and evaluation performance standards and output planning that can ensure every individual receives the services that they need to live safe, healthy and productive lives.

Dr Emily Gustafsson-Wright is a Fellow in the Global Economy and Development Program at the Center for Universal Education, Brookings Institution, Washington DC.

If you wish to receive our weekly blog on SIBs, please email and we will add you to our subscriber list.

“Rethink questions to patients in general practice and focus more on improving primary care”

By Tommaso Manacorda

The “Friends and Family Test”, seeking patients’ views, has created anxiety among practices but shed little light on patients’ concerns. It’s time for a rethink, suggests our study.

The many ways in which patients can now feedback to GPs should offer a rich source of information for those keen to improve their services. However, when we evaluated the latest addition to patient consultation in England – the “Friends and Family Test” of patient satisfaction – we found that “more” did not necessarily equate with “better”.

We found that general mistrust of the FFT process, combined with inappropriate framing of its core question is undermining the initiative. But we also identified ways both to bolster trust and make patient feedback more useful for improving primary care.

Two important pieces of learning emerge from examining this initiative. First, the issues raised by the “Friends and Family Test” could usefully prompt a rethink about how views are gathered from patients about their primary care experiences. Our main recommendation is that the FFT should be revised, although discontinuing it should also be considered, as its reception within general practice has been largely negative.

Second, GPs need more encouragement and guidance to tackle inadequacies revealed by patients’ experience. We found that commitment to quality improvement is uneven across primary care and remains a low priority in some, often in those practices where patient participation groups (PPGs) meet rarely and patient surveys are infrequently conducted. Nurturing a rigorous improvement culture is at least as important as getting right the questions posed to patients.

Wording of the FFT
The “Friends and Family Test” puts a single question to patients about their general practice: “We would like you to think about your recent experience of service. How likely are you to recommend our GP practice to friends and family if they needed similar care and treatment?” Answers are recorded on a five point scale from “extremely likely” to “extremely unlikely”. Additionally, patients may be asked to comment on their reasons for the score they have given.

General practices across the NHS are required to make the FFT available to patients after every contact, collecting the data with the method that suits them best. The majority of them use handwritten cards, but tablet kiosks and online apps are also used.

Our evaluation, involving 42 practices and 118 interviews with clinicians, practice staff and patients’ representatives, found two sets of problems with the FFT. The first concerned the usefulness of the information it produced. The second problem lay with how general practices understood the FFT’s purpose and how they engaged with it.

The experiences of hospitals, where the FFT was first introduced, showed that the quantitative scores were statistically unreliable. So the metrics could not be used to compare providers for quality. That’s because the FFT does not involve a representative sample and is vulnerable to selection bias. One can’t tell whether the scores are representative of all patients. So, when the FFT question was subsequently rolled out in 2014 into primary care, the widespread awareness of these statistical limitations contributed to unease in general practices about the FFT approach.

In our study, the FFT question was deemed inappropriate by most interviewees. Many, particularly in rural areas, lacked choice of general practice, so the question of where they might send friends or family lacked reality. Also, patients found it difficult to compare their personal care with what someone else might receive, because that depends on individual factors such as age, sex and existing health conditions. Most interviewees suggested the use of a more straightforward question.

The additional space provided for further comment did potentially offer some useful feedback. However, staff in general practices felt that the anonymity of patients made it difficult to act on these comments. That’s because patients were often unclear which service they had received, from whom and the precise nature of issue that concerned them. For example, a patient complained about a “terrible phone service”. But staff said it was difficult to respond to this statement because they didn’t know who the complainant was, whom the patient had spoken to or what they needed.

Most patient feedback collected by practices was positive, but anonymity and vagueness made it difficult to identify and reward good practice. Positive feedback would often lift staff morale. However, the inability to act on complaints was often reported to be very frustrating for staff.

When additional comments gathered by the FFT pointed at some specific issues, their contribution was still considered to be of little value because the issues were already known to staff from other sources, such as practice surveys and patient participation groups.

Professionals mistrust the process
The second issue with the FFT concerned the negative reception that it received from primary care professionals. Some feared that it left them vulnerable to hostile patients. Someone might, for example, be correctly denied an antibiotic for sound clinical reasons but then score down the practice unfairly. The practice would have no way to question the scoring.

Monthly reporting of scores to the DH and NHSE added to professional concerns that the FFT process was, as some claimed, “a stick with which to beat General Practice”. Even though the Government had stated that it would not use FFT scores to rank practices, that fear of unfairness remained unassuaged, which disheartened hard-working professionals.

As a result, although all the practices in our evaluation had made the FFT available to patients, few practices felt committed to, or “owned”, the process. Even though practices had been assured that the test was intended for local quality improvement, not regulation (or criticism) by national bodies, they remained doubtful, particularly because of the monthly reporting. One GP, not realising that the scores were for meant for practice consumption, even asked their practice manager during a joint interview: “Do we have to open the box?”

Shift to reporting quality improvements
We recommend that monthly reporting of FFT scores should be stopped, and maybe replaced by a qualitative report on local quality improvements, perhaps submitted annually. That would help restore trust among practices that central bodies are not “spying” on them.

Particularly valuable aspects of the FFT are the chance for patients to comment briefly and quickly about their experiences, and the opportunity for practices to collect such feedback rapidly and easily. This comment facility could, for example, be kept within an FFT that had a less confusing core question, and patients could be encouraged to be more specific about particular services (e.g. phone consultations, clinics for chronic diseases, immunisation, etc.), making it easier to identify and address problems on aspects of care that they want to be improved.

The key issue in the long-term will be whether, and to what extent, patients’ views will contribute to making services better. Commitment to quality improvement was found to be uneven across practices. A revised FFT might play an useful role in addressing this problem, being easy to implement and thus a feasible option particularly for smaller practices with less capacity for data collection. But extra detailed guidance is needed on how to ensure that patient feedback leads to service improvement. It will be important to make clear that the Government’s priority is aligned with that of GPs in being focussed on securing higher quality primary care.

Tommaso Manacorda is a Research Fellow at the London School of Hygiene and Tropical Medicine. His report “Implementation and use of the Friends and Family as a tool for local service improvement in NHS general practice in England” is published by PIRU and co-authored by Dr Bob Erens, Professor Sir Nick Black and Professor Nicholas Mays.

This commentary summarizes an independent report commissioned and funded by the Policy Research Programme of the Department of Health for England, via its core support for the Policy Innovation Research Unit, with additional funding provided for data collection from the main sample of general practices. The views expressed are those of the authors and not necessarily those of the Department.

Social Impact Bonds offer challenges and opportunities in health and social care

By Alec Fraser, Stefanie Tan & Nicholas Mays

SIBs bring promises of extra cash and reduced fragmentation but also concerns about private sector interests, practicality and governance.

What does research tell us about Social Impact Bonds (SIBs) and their applicability to health and social care? That’s our big question as we review the issues raised by our experts in their thoughtful commentaries over the past few weeks.

It’s clear how the mood music around SIBs has appeal, notably concerning two issues that bedevil health and social care – funding shortfalls and fragmentation. Ring-fencing funds and staff to achieve specific outcomes is a key SIB feature, particularly highlighted in the contribution from Ben Jupp of Social Finance. Those in health and social care who feel frustrated by juggling cash for competing tasks – having to “rob Peter to pay Paul” – would like to get on with their jobs, unencumbered by unexpected resource cuts.

Likewise, part of SIBs’ appeal is the feeling that they address chronic fragmentation of service delivery, particularly between health and social care. In principle, SIBs incentivise everyone to work in harmony on behalf of the patient/user.

Three questions
Some of our commentators have cast doubt over these possible benefits. Ring-fencing of spending and staffing can, for example, be achieved by other means, such as minimum nurse staffing ratios. Our contributors also highlight that SIBs can’t cure the structural problem of health and social care fragmentation, though they can be helpful on a case by case basis.

We would like to draw attention to three further issues raised by our commentators: the public’s general discomfort with the idea of private interests in NHS provision; the practicality of outcome-based, debt-funding in health and social care compared with traditional forms of payment; and the governance challenges created by SIBs as they reintegrate the purchasing and providing roles that have been separated in the English NHS since the 1990s.

On the other hand, we suggest that SIBs can contribute to mitigating some longstanding issues facing health and social care. These include the need for a better focus on prevention and outcome delivery; personalising services around the user/patient; and better partnership in health and social care planning and delivery between purchasers and providers.

1. Discomfort with private financing of healthcare
The early UK SIB-funded programmes have typically dealt with issues outside the mainstream of health and social care provision, such as youth unemployment and probation services. SIBs in health and social care have mainly focused on preventive interventions to improve health (e.g. social prescribing). We’ve yet to see proposed applications that address core NHS treatment activities such as funding hospitals and primary care. Such proposals might be highly controversial to the public.

The NHS retains a unique status for voters, unlike, for example, tertiary education, where private fees are widely considered to be acceptable, or social care, which is means-tested. The NHS is a universal service. There is a general belief that it should be paid for publicly. There is also less pressure to find alternative sources than there is, for example, in the case of some local authority social services because the NHS has been better protected from financial austerity than other public services.

2. Important practical and effectiveness questions
Our second theme concerns practical questions about the effectiveness of SIBs. Alex Nicholls and Mildred Warner have drawn attention to the weakness of the evidence in this area. There is a lack of evidence to demonstrate that SIBs provide better services that are more cost-effective or more likely to save money than those delivered using conventional funding methods. A further issue is how to make cashable any notional savings generated by improving user outcomes: for example, being able to close wards and sack nurses as a result of a successful SIB would not be a popular approach in such a highly politicised public service. Unlike many social services, such as problematic families, the public generally regards spending more on the NHS as a positive thing, so there may be less political support to pursue “savings” in health.

Healthcare commissioners in the UK are well versed in scrutinising cost-effectiveness evidence but healthcare lacks the interoperability of information systems that aids oversight of the outcomes and costs arising from interventions. So, for example, it’s currently easier at the Department for Work and Pensions to assess a SIB’s impact on employment rates among NEETs (Not in Employment, Education or Training) than for the NHS to track whether a preventive health SIB has reduced use of primary and hospital care.

Healthcare commissioners may also be concerned that SIBs typically have high set-up/transaction costs and quite opaque management costs. They may be reluctant to pay for outcomes when the sources of money needed to pay intermediaries and other actors lack transparency. This caution about SIBs is well founded given the continuing costs of previous PFI projects, which created new facilities quickly but left the NHS with debt and concern among some that providers may have secured poor value for money.

3. SIBs and governance issues in healthcare
Thirdly, there is a governance challenge that may accompany the adoption of SIB-funding for health and social care services. SIBs are attractive because they are designed to bring stakeholders into partnership in pursuit of a set of goals around a particular client group. This may imply, for healthcare, a degree of reintegration of purchasing and providing. This may be problematic as it appears that a genuine market in intermediaries is currently lacking.

We also do not yet have a recognised governance framework setting out how intermediaries and the other parties should operate at particular stages of the SIB development process. This is of crucial importance for policymakers as SIBs currently develop in a space that lacks agreed rules and processes. It will be important to resolve these governance issues and create an explicit framework for managing potential conflicts of interest.

Digging a little deeper, the SIBs debate should remind us of the need to maintain a healthy scepticism about all providers, even if they are from the Third/not-for-profit sector. They should be subject to due scrutiny, like any other providers of contracted-out services.

There are important questions and doubts about the wisdom of widespread application of SIBs to health and social care. Nevertheless, we should be careful and avoid prematurely rejecting a still largely untested innovation that could have niche applications for entrenched social issues. It is also worth remembering that traditional health and social care procurement models are far from perfect.

SIBs are rightly taken seriously because they help to question services that have, through bureaucracy, size and provider-domination, sometimes failed to meet the needs of patients or the requirement for cost-effectiveness. But there are important questions to be asked about the transparency, costs and accountability of SIBs. Our commentators have shown that SIBs are not a panacea and that they bring unanswered questions with them. More robust and more plentiful evidence would be needed to justify widespread roll-out of SIBs in UK health and social care.

Stefanie Tan and Alec Fraser are Research Fellows at the London School of Hygiene & Tropical Medicine and part of the Policy Innovation Research Unit (PIRU) at LSHTM.
Nicholas Mays is Professor of Health Policy at LSHTM and Director of PIRU.

Acknowledgement and disclaimer:
This is an independent commentary commissioned and funded by the Policy Research Programme in the Department of Health. The views expressed are not necessarily those of the Department.

Just how bad is the NHS at innovation? The answer – we don’t know

BY James Barlow

“There is strong encouragement for the NHS to make better use of innovation, but we’ve not known how our system performs internationally. Now we are developing a potential measure.”

The Accelerated Access Review (AAR) – which aims to help speed the introduction of health innovations into the NHS – was published last October. Readers with long memories may recall a string of previous reports all tackling the need to improve the UK’s health innovation system: the House of Commons Health Committee’s inquiry into the use of new medical technologies (2005)1, Best Research for Best Health (2006)2, the Cooksey review (2006)3, Innovation for Health (2007)4, Our NHS Our Future (2012)5, Innovation Health and Wealth (2012)6.

The AAR calls for a broader remit for NICE, to include more medtech and digital health innovations. Other recommendations are for closer alignment between regulatory requirements and processes, and for once-only assessment by NHS England of innovations not referred to NICE. Simpler and swifter procurement processes are part of the future agenda. And in an implicit nod towards the work of Clayton Christensen, the AAR suggests there should be some funding to support the commercialisation of disruptive innovations that have the potential to significantly improve care pathways.

These are all sound recommendations, but they are only the beginning. Recommendations now need to be turned into specific actions with budgets, owners, milestones and deliverables.

The good news from the AAR is that consensus seems to have broken out, with a genuine belief by government and NHS leaders in the potential of innovative technologies to improve patient care. The consensus extends to a call for more flexible approaches to pricing and reimbursement which can support the adoption of innovations.

The bad news is that the NHS is in deep financial trouble. For all the Churchillian rhetoric of healthcare leaders – “we have run out of money, now we must think” – and exhortations that a crisis is a great opportunity for innovation, the distinct tendency in the NHS is to hunker down, deal with the immediate concerns and shuffle innovation into the “too difficult” pile.

Goals, targets or crisis can of course concentrate the mind and, when coupled with the right kind of support and incentives, generate useful new ideas – a good example was Scotland’s Unscheduled Care Collaborative Programme in the mid-2000s7. However, the magnitude of the challenge of modernising the NHS, whilst continuing to run services with shrinking budgets, seems overwhelming. We know we need innovation, we know what kinds of innovation we need and we have a pretty good idea what is the potential impact of many innovations on improving care – the problem is introducing and establishing them in the NHS.

The AAR, if its recommendations are implemented, might help to speed up the flow of some new technologies into everyday practice, but it will not tackle the other layers of innovation that are needed. These were nicely described by the Health Foundation in 20158 as five interlinked components of a framework for change, all of which require innovative thinking – population health management, ways of delivering care, process improvement, active cost management, and scientific discovery, technology and skills. The report argued that all these are capable of delivering improvement over different timescales. Interestingly, the last of these – science and technology – is felt to have the most limited potential impact.

But just how bad is the NHS at innovation? There is much anecdote and received wisdom – “the UK is great at generating innovations but poor at adopting them”, “the NHS is always a late adopter”, “developing countries health systems are where the really interesting new ideas are”, “the USA over-adopts healthcare technology”.

Much seems to depend on perception. The 2016 Global Diffusion of Healthcare Innovation study surveyed 1,356 frontline health workers and healthcare leaders in six countries and found that the USA and UK were ranked as the two most important sources of innovation, but there were variations partly according to location – India was perceived as by far the most important by interviewees in some countries.

The volume of research on healthcare innovation processes has grown over the last fifteen years or so. We know what constrains the adoption of innovation and we know what innovators need to support them from research to commercialisation.

What we lack, though, is any kind of assessment of where different countries stand in relation to each other in their healthcare innovation capacity – how good they are at adopting innovative solutions developed elsewhere and originating innovations themselves. None of the reports on the state of healthcare innovation in the UK described above provide any real insight into our performance compared to other countries. Apart from limited work by Deloitte9, PWC10 and Accenture11, there has been no attempt to create an international comparative healthcare innovation index (the Global Diffusion of Healthcare Innovation is not an index because it teases out perceptions of innovativeness).

There are many indices or composite indicators measuring aspects of national performance in innovation in general. Indicators include measures of entrepreneurship, technology development and scientific research, innovation in general, and innovation in public sector organisations. However, these remain underdeveloped in relation to health.

A carefully constructed global health innovation index would be valuable in two ways:

• It would focus attention on the relative performance of the UK’s (or any other country’s) health system, pinpointing its strengths and weaknesses, identifying bottlenecks and issues for attention. This would enable policy makers and the healthcare sector to identify and prioritise the levers that can have the greatest impact on innovation.

• It would help medical technology and life sciences companies by providing an understanding of the attractiveness of each country’s health system as a market for products, as a source of innovations, and as a location for R&D.

A project funded by PIRU has carried out the groundwork needed to develop a global health innovation index12. We reviewed existing global innovation indices and explored the conceptual, methodological and practical issues that must be addressed. We draw three conclusions from our review.

First, one should proceed with caution when deriving policy, research or other implications from composite indicators. Before we can be confident of their implications for the comparative performance of countries or regions and establish benchmarks to underpin policy or other decisions, it is essential to understand how innovation indicators impact on innovation processes. This in turn requires indices to be underpinned by a clear and strong theoretical framework.

Second, there are a number of data issues which have to be addressed. To explore the dynamics and evolution of health innovation across countries through an index requires panel data (collected over time). While the availability of cross-sectional data (between countries) is fraught with difficulties, the use of panel data is even more so. Another issue is variation at the regional or local level. Policy makers are interested not only in their national scores, but in whether different regions in their country perform differently.

Finally, policy makers should be careful not to draw conclusions about the relationship between composite innovation indicators and other indicators of policy interest such as health outcomes. While it may be of interest to correlate innovation index scores with those of an index of health outcomes, we must always remember that correlation does not mean causality – introducing a new drug or device may well lead to better health outcomes, but an index will not tell us this.

We are now hoping to create a prototype index for a few countries. This involves investigating data sources and more work on what the index is measuring, for whom and for what purpose.

Anyone interested in finding out more should contact James Barlow at Imperial College Business School,

James Barlow is a Professor of Technology and Innovation Management (Healthcare) at Imperial College Business School and a member of PIRU. His latest book, Managing Innovation in Healthcare, was published by World Scientific in January 2017.

1House of Commons Health Committee (2005) The Use of New Medical Technologies Within the NHS: Fifth Report of Session 2004–2005.
2Department of Health (2006) Best Research for Best Health: A New National Health Research Strategy: The NHS Contribution to Health Research in England.
3HM Treasury (2006) A Review of UK Health Research Funding.
4Strategic Implementation Group & Healthcare Industries Task Force (2007) Innovation for Health: Making a Difference.
5Department of Health (2007) Our NHS Our Future: NHS Next Stage Review – Interim Report.
6Department of Health (2011) Innovation Health and Wealth, accelerating adoption and diffusion in the NHS.
7Dattée B, Barlow J (in press) Multilevel organizational adaptation: Scale invariance in the Scottish healthcare system. Organization Science; Dattée B, Barlow J (2010) Complexity and whole-system change programmes. Journal of Health Services Research & Policy 15, S2, 12-18.
8The Health Foundation (2015) Shaping the Future. A Strategic Framework for a Successful NHS.
9Deloitte (2012) Innovation Indicators for Healthcare in Emerging Countries. Understanding and Promoting Innovation in Emerging Markets.
10PWC (2011) Medical Technology Innovation Scorecard. The race for global leadership.
11Accenture (2013) Measuring the Unmeasurable. A New Framework for Assessing Healthcare Innovation.
12Cravo Oliveira T, Barrenho E, Vernet A, Autio E, Barlow J (2017) Developing a Global Healthcare Innovation Index. PIRU Reports 2017-20.