The Hitchhiker's Guide to SyROCCo

In September, the Government Outcomes Lab and our partners at the University of Warwick and The Alan Turing Institute launched the Systematic Review of Outcomes Contracts – Collaboration (SyROCCo). Unveiled during the 2022 Social Outcomes Conference, and building on the GO Lab and Ecorys' global systematic review of social outcomes contracting, this pioneering tool uses machine learning to support practitioners, policymakers and researchers to find the right evidence and resources to support their projects related to outcomes-based contracting.

What is SyROCCo?

SyROCCo is a prototype tool that allows the user to quickly navigate through the GO Lab’s global systematic review of outcomes based contracting: the deepest and broadest dataset to date of all the empirical academic publications, evaluations and policy reports on outcomes-based contracting that were published between 1990 and 2020.

Our global systematic review currently has around 2000 resources with empirical data and insights about outcomes-based contracting. This type of knowledge bank is already a remarkable instrument in itself. But manually mastering the findings of this database requires substantial time and effort from the user, who might be hard-pressed to get quick results.

This is where machine learning comes in. Instead of having to read through the 2000 publications in order to attain a comprehensive understanding of the field, with the help of SyROCCo, policy scholars and practitioners can now quickly learn about global trends and potential areas of further engagement that otherwise would have taken months, if not years, to identify.

SyROCCo takes advantages of the possibilities offered by AI technology in order to increase the accessibility of the global systematic review in two ways:

• It provides an online platform to the world’s most comprehensive and searchable dataset on outcomes-based contracting resources
• It offers researchers and practitioners data analysis and exploration tools to make the most of the dataset and apply its insights to their projects

Using SyROCCo

Through a set of high-level filters, SyROCCo lets the user search through the global systematic review’s resources by using different categories:

• Policy Sector: health, education, homelessness, criminal justice, employment, child and family welfare, environment.
• Income Level (of countries mentioned in resources): high, medium, low
• Publication Year: 1990 to 2020
• Financial Mechanism: social impact bonds, targeted grants, public-private partnerships, etc.

SyROCCo quickly reads through the body of text of each paper, and uses natural language processing techniques (including topic modelling and semantic analysis) to deliver a results page tailored to our needs.

For example, by clicking on the policy sectors tab, one can choose to display only those resources related to Education and Employment. Accordingly, the search result ends up displaying 301 resources, out of the almost 2000 total publications.

The search can then be further refined by clicking on other tabs, such as financial mechanisms. Thus, if one selects to display resources related to Education and Employment, AND to show those resources that also mention Impact Bonds within these criteria, the list of relevant findings is reduced from 301 to 96.

But there is more. In the process of browsing through piles of text, SyROCCo also picks up on those resources that mention one or more countries, and maps these in an interactive way.

Going back to our original example: from the 96 resources focused on impact bond projects related to Education and Employment, the user can also click on a country in the map to further restrict the list of findings. For instance, one can click on Australia and use our country threshold feature to state that relevant resources should mention this country at least twice.

At a further level of granularity, SyROCCo can help the user by looking for keywords used in the titles of resources that meet the criteria already selected. Thus, in a list of 12 resources focused on impact bond projects related to Education and Employment in Australia, one could also input the keyword “payment” to refine the findings once again.

SyROCCo and INDIGO: Other practical uses

SyROCCo is a useful tool for anyone interested in finding the latest resources on outcomes-based contracting. Still, SyROCCo goes beyond the mere searching and sorting of information. It is also able to connect personalised search queries with the data from our INDIGO Impact Bond Dataset.

This feature adds a further layer of practicality for the policymaker, who might not only be interested in knowing more about the available scholarship on impact bonds related to Education and Employment in Australia, but also in examining details on the particular projects which have been studied in this context.

Reverting to the example used throughout this blog post, a user can see that among the 12 relevant resources in the list of findings, at least 5 impact bonds are also mentioned – and available for examination – in the INDIGO Impact Bond Dataset.

In this particular case, when clicking on the impact bond “Prevista”, SyROCCo offers us a list of the resources mentioning this impact bond and allows us to directly jump to the data available on INDIGO.

By tapping into the wealth of knowledge created by our global systematic review, SyROCCo allows users to explore the broadest range of available data on outcomes-based contracting with ease. It also allows the policymaker to quickly go from the high-level and global to the granular.

Nevertheless, SyROCCo can also be a fantastic tool for policy scholars interested in learning more about important trends in the field. From the macro to the micro, our AI-powered tool can offer researchers a range of insights:

• Key policy areas where outcomes-based financing tools have been employed
• Information about outcomes-based contracting related to specific countries
• What types of outcomes-based financing tools have been used until now
• Key organisations active in this field

These are just some possibilities, among others. SyROCCo is a work in progress. And we welcome your feedback!