An old scouting adage for the new world of SIBs: Be prepared.

Thinking ahead together

In a world living through a crisis like Covid-19, issues of preparedness and prediction of behaviours are in sharp focus.  Some would argue that participatory scenario testing helps with readiness for risks and eventualities. The mid-point account of Ways to Wellness (WtW) left me wondering how such collective approaches to scenario testing might be helpful in SIBs – especially when they rely on sophisticated design features. 

Let’s be clear, the Ways to Wellness SIB at the halfway point was - and has continued to be - successful on many fronts. Significantly, it’s been measured against a high bar – a counterfactual based on a contemporary comparison group. This is designed to demonstrate that its large-scale social prescribing model, delivered by de-risked voluntary, community and social enterprises (VCSEs), can generate tangible savings for the NHS by reducing unplanned admissions and use of out-patient and A&E services. Overall, 70% of the PbR outcomes payments were modelled to be made against this outcome. 

Yet despite its impressive credentials, the report reflects how the SIB’s relationships came under strain relatively early on. It turns out that an exacerbating factor in these tensions was an undetected error in the way that comparison group data was being computed.  This error fostered reactions that led to possibly overzealous changes, both to payments to WtW and in arrangements with service providers, who originally expected to be financially shielded by the SIB.  Which gives rise to my question:  Had all the parties to the SIB run participatory scenario testing exercises during its design, to gauge likely reactions if things didn’t go as planned, might they have handled things differently?  

In SIBs, ‘scenario testing’ is typically confined to financial modelling of outcomes – according to high, median or low performance forecasts – rather than participatory testing of assumptions, key dependencies and relational dynamics.  The CBO evaluation is increasingly finding that when these go out of kilter, reactions in the heat of the moment can be tricky.  That’s why some pre-empting might come in handy, especially where a lot is riding on things working as planned.

Ambitious design principles

The WtW SIB is a very ‘big’ deal. In a total local population of 140,000 people, it targets a hefty 10% to help mitigate the impact of their long-term health conditions.   The CCG commissioners pay against a sliding scale according to the difference in secondary care costs between the treatment and comparison group. And all the capital from its investors was at risk if savings didn’t exceed project costs.  This arrangement was seen as a sturdier value for money proposition case than comparable social prescribing services, and has been key to establishing its longer-term credentials.  

It’s not all about the savings of course. 30% of the SIB’s payments are linked to improved wellbeing and reduced isolation, using Triangle Consulting’s Long-Term Condition (LTC) Wellbeing Star. But given the CBO programme and Social Outcomes Fund cover these soft outcomes, the performance focus for the CCG, investors and WtW Board was inevitably closely trained on the harder savings measures.

When the alarm goes off, is it really a fire? And do we all head in the same direction?

So concern was triggered when data started showing variability (compared to the control group) of monthly per-head savings to the secondary care services - indicating (wrongly as it happened) a lower level of performance than expected.  Strangely, levels of client use in primary services were decreasing – so the secondary care trends didn’t have an obvious explanation.  

To add to the concern, referral data indicated that during its first 3 years, the project was engaging 32% fewer people than had been modelled.  This had already raised early warning signs that the SIB’s cash flow modelling might not work as planned. This initially translated into pressure on providers, via changes to their payment-terms, to drum up referrals.  Partly as a result, two providers dropped out, but the other two took on the gauntlet.  

Concern around the comparison-cohort data led to a request that, to compensate for low levels of savings-triggered payments, the CCGs should also make wellbeing payments (on top of those provided by CBO). This was rejected. At the same time, however, WtW convinced CBO to reflect greater-than-modelled complexity of presenting cases in its payments.

This illustrates how, when performance looks to be dipping, stakeholders tend to see the same issue through a range of different lenses, depending on their risk perspective, what the purpose of the SIB was for them, and their red lines.  The report explains how a lot was riding on making the SIB work, generating the tension over appropriate reactions in light of what appeared at the time to be underperformance. As one key stakeholder commented: 

“There was a lot of assumption that everything would go well and everyone’s interests will be aligned…when the wheels fall off, there are differences in priorities…It’s not as win-win-win as I initially thought .”  

For some commissioner representatives (including us as co-commissioners), the prospect of the service falling over while tension grew among stakeholders risked a potentially premature decline into a vicious circle.  But if we had run early exercises that envisaged situations in which the wheels could come loose, might we have urged a closer look at the data before rushing to save the SIB? 

For others, the key was to maintain a clear line of sight to whether the SIB as modelled could prove its savings-through-scale case for CCG funding, and so it was important to resist the mitigations requested of them. 

Meanwhile, for some providers, the drive to maximise engagements compromised their primary interest in delivering quality wellbeing outcomes, and was a step too far. Even the two that remained expressed doubt that the SIB arrangement had aided their performance, and being less able to influence the more ‘distant’ savings outcome that fuels most of the SIB’s cash flow, they reported feeling powerless in the face of their apparent ‘underperformance’.

Thinking ahead by reflecting back 

There have clearly been significant benefits from WtW for people with long term conditions and it remains a landmark in scouting out ways of taking social prescribing to scale.  The experience of delivering through a SIB – which depends on many factors, including data systems working as planned – hasn’t been without mixed feelings though.  The evaluation’s third visit will look closer at the detail of the comparison group error and whether the evolved incentive structure at the provider level led to improved outcomes.  It will also look at what the lessons taken from its successes and challenges mean for WtW’s future. As its stakeholders look beyond the CBO SIB, perhaps their experience can help us all consider the scenarios that it might pay to be prepared for?