About the SyROCCo machine learning tool

The SyROCCo Machine Learning Tool is the product of a collaboration between researchers from the Government Outcomes Lab and machine learning experts from Warwick University/Alan Turing Institute. We officially launched the tool at our 2022 Social Outcomes Conference. If you want to learn more about the tool, how it works, and our plans for the future, watch the session's recording here.

If you want to cite this work, please refer to this academic paper: Fang, Z., Arana-Catania, M., van Lier, F.-A., Outes Velarde, J., Bregazzi, H., Airoldi, M., Carter, E. & Procter, R. (2024). SyROCCo: enhancing systematic reviews using machine learning. Data & Policy, 6, e39. DOI: 10.1017/dap.2024.33

The Aim of the Project

The aim of the project is to make use of the latest machine learning technology to increase the scope and accessibility of the GO Lab’s Global Systematic Review of Outcomes Based Contracting by: (1) creating and maintaining the world’s most comprehensive dataset on outcomes-based contracting; (2) creating exploration tools that will make the data accessible both for researchers and policymakers.

The broadest and deepest Systematic Review on Outcomes Based Contracting to date

The project builds on the deepest and broadest systematic review of outcomes-based contracts to date, conducted by the Government Outcomes Lab and Ecorys. The aim of systematic reviews is to summarise and condense all the available published research in response to a particular research question. To do so, systematic reviews use reproducible and transparent methods to “identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review” (NHS Centre for Reviews and Dissemination, 2001) [1]

Our systematic review seeks to be exhaustive in terms of capturing all empirical studies of outcomes-based interventions published between 1990 and 2020. That means that our systematic review is extraordinarily broad (see here for the Systematic Review Protocol). After an initial search according to the search criteria, our team double-reviewed more than 11,000 abstracts and titles, which reduced the number of potentially relevant articles to about 2000. To identify the literature that matched more detailed inclusion criteria (e.g. specific requirements regarding empirical content and contract details), the remaining 2000 articles underwent a full-text review, resulting in a set of 300 empirical studies that matched the more narrow inclusion criteria. Finally, researchers extracted data and synthesised findings from the 300 empirical studies according to specific themes, such as education or climate change (see here for a more detailed description of the process).

The Data

The SyROCCo Machine Learning Tool taps into the wealth of knowledge that lies in the “middle” of our systematic review, focussing on the roughly 2000 articles that made it through the title and abstract screening process. While these studies do not match the very narrow inclusion criteria of empirical content or contract details, we can be confident that this literature entails rich and meaningful information on outcomes-based contracting.

For example, this literature contains important high-level data on key policy areas in which outcomes-based financing tools have been employed; on the different types of financing tools that have been employed; on key organisations active in this field; or on relevant laws and regulations. Extracting such data via state-of-the-art machine learning tools enables us to gain rich insights about global trends in outcomes-based financing and potential areas of policy engagement that would otherwise not be available.

Additionally, we combine high-level information extracted from the Systematic Review with data from our human-curated INDIGO Impact Bond Dataset, the most complete and granular dataset on Social Impact Bonds available to date. This allows users of our tool to explore the broadest range of available data on outcomes based contracting at ease - ranging from high-level and global to granular, project-level information.

We hope that this data and its accompanying tools will assist researchers seeking to push the boundaries of academic knowledge on outcomes-based-contracting, and offer crucial insights to policy makers and other relevant stakeholders to make evidence-informed decisions on the most appropriate form of outcome contract or financing model in different contexts.

The dataset behind Syrocco can be found on Zenodo.

[1] Undertaking Systematic Reviews of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition). NHS Centre for Reviews and Dissemination, University of York. March 2001.

We thank these funders for their contribution to this work: Department for Digital, Culture Media & Sports; Children's Investment Fund Foundation; UK Research and Innovation; the UK Foreign, Commonwealth and Development Office; and the John Fell Fund.