Setting the scope for the Global Systematic Review on Outcomes-Based Contracting

Embarking on an ambitious evidence review

Using outcomes – the positive results that programmes and services produce in the lives of service users and citizens – to steer contracts with non-government providers is not new. Across a range of policy areas and geographies governments and philanthropists are increasingly looking to adopt this approach. Although promising, evidence suggests that there are risks (especially around managing perverse incentives) and policymakers are keen to adopt outcomes-based contracting only where appropriate and informed by best practice.

GO Lab and Ecorys have partnered to undertake a global systematic review to explore whether, when, and where (and if possible, how) outcomes-based contracting approaches deliver improved impact when compared to more conventional funding arrangements.

To aid with the mission of the research, we established a Policy Advisory Group (PAG) to help steer the study and ensure that it produces the insights and outputs that are of most value to policymakers. The PAG is open to representatives from organisations responsible for funding social programmes which have either i) expended significant resource on outcomes-based contracts; ii) intend to; or iii) have considered this. We contacted organisations that we knew met these criteria via an initial rapid review. We also launched a call for participants. We were pleased with the response: our first PAG meeting included 36 members, covered all continents, and included representatives from national and regional governments, multilateral organisations and NGOs.

The PAG will provide crucial insights across three sessions:

1) Refining scope and key areas of focus (26 November 2020)

2) Unearthing evidence and developing a framework for synthesis (January 2021) 

3) Ensuring policy-relevant outputs and informing future activity (June 2021)

Outcomes-based contracting: What’s in a name?

Crucially, for this research to be meaningful to decision-makers the review scope and approach needs to match the questions that leading policymakers are asking.

The ambition of the first PAG meeting was to refine the scope of the review. We used three exercises to identify a) the relevant terminology, b) justifications and c) evidence gaps as described by policymakers. We provide headline findings from these sessions below.

Diverse – and sometimes puzzling – terminology is used to describe ’outcomes-based contracting’ and related funding tools. It is important for the review team to identify and incorporate all key terms at the outset, in order to ensure that the search terms used are comprehensive. To assist with this, PAG members spent time suggesting search terms in English and, where possible, in other languages. Participants also provided descriptions of the terms and began to draw out clusters of related terms.

The search terms identified through this activity will be used by the review team to design a sensitivity maximising search strategy. In total, the PAG suggested 89 English search terms (with 34 of these also provided in other languages). We have grouped together the terms into different categories – as shown in the table below. This shows that terminology linked to Payment-by-Results (PbR) and Impact Bonds (a sub-set of PbR) was particularly popular amongst the participants. This is an indicative typology which we will refine during the review – if you have any feedback on these categories (or how we have divided the specific terms in the annex) we would be delighted to hear it.

What purposes or ‘effects’ should the review look for?

A range of justifications and purposes for the adoption of outcomes-based contracts were proposed by the advisory group. These justifications and areas of application were listed against each of the terms identified in exercise 1. Key themes include improved efficiency, effectiveness, accountability and space for innovation or experimentation. The full list of justifications is provided in the annex. In many instances, the justifications used for different terms were strikingly similar, yet in others there were explicit reasons for choosing one approach over another (e.g. one participant noted that a standard PbR contract may be simpler and less costly than a SIB). In addition to proposing and justifying terms for individual funding instruments, some participants also highlighted the importance of considering Outcome Funds, which can be used to aggregate multiple contracts under a common financing framework.

Where is the thirst for evidence strongest?

The Policy Advisory Group shared, in aggregate, almost 100 questions they wished the review to contribute to. The great majority cluster under four categories. In order of frequency, these categories comprise:

1) Prescriptive: How to tackle known difficulties in designing outcomes-based approaches? Questions often centred on best practices (for example, in the use of metrics, incentives, validation, evaluation) and capacity building

2)  Normative: An overview of what the different instruments are intended to achieve, supported by empirical evidence of what has worked where/when

3)  Descriptive: A census of existing practices i.e., an illustration of where and when different varieties of outcomes-based contracting have been adopted

4)  Special interest: A better understanding of risk profiles and the role of (impact) investors.

The research team will use this information to develop the search terms and research questions. We will meet with the PAG again in January to discuss how we will review the evidence in-depth.

The review team are grateful to PAG participants for their energetic engagement at online workshops and ongoing support for the systematic review.