Top Five Lessons Learned: Replicating Evidence-Based Interventions

| No responses | Posted by: Tricia Hackett | Theme: Research, Work with Communities, Youth & Education

In July, Realising Ambition published a report, written by the Social Research Unit at Dartington (SRU), looking at the first year of the programme. Realising Ambition is a five year £25m investment from the Big Lottery Fund in replicating evidence-based and promising approaches to help prevent children and young people (aged 8-14) from becoming involved in the criminal justice system. The report shares some early learning from the perspective of the consortium programme support team which is comprised of staff from Catch22, the Young Foundation, SRU and Substance.

Top Five Lessons: Replicating Evidence-Based Interventions

Realising Ambition aims to make a significant contribution to the sector by encouraging more widespread replication and implementation of programmes that have a range of strong evidence to demonstrate that they are improving the lives of children and young people. As the UK currently has a relatively scant evidence-base, Realising Ambition will provide a great deal of learning on a range of areas including replication, developing evidence-based practice, using diverse delivery models and how to successfully demonstrate the social impact of early intervention programmes.

We have distilled our early findings from the first year of the programme into Top Five Lessons Learned:

1. There aren’t enough evidence-based programmes in the UK

During the application phase to Realising Ambition it was explicit that only applications from well-evidenced interventions would be considered. Over 240 interventions applied yet only 4% met the Social Research Unit’s high standards of evidence. What this has meant for Realising Ambition is that only ten of 25 interventions are underpinned by the highest quality of evidence.

2. Organisations tend to overestimate the demand for their service

Early indications show that organisations might have had an unrealistic estimation of demand and has meant that there is a shortfall in meeting planned delivery numbers. Sometimes projects may struggle to engage the eligible children and families because there is insufficient or poor data about the level of need or the distribution of need in the target areas. The learning from this is the need for robust data on need and the market for the intervention.

3. It is difficult to know if promising interventions will be able to deliver impact

Whilst ten of the interventions are tightly defined and underpinned by the highest standards of evidence, the remaining interventions have a range of evidence including pre and post questionnaires, on-going quality assurance and qualitative case studies to offer preliminary evidence of impact. These methods won’t necessarily give commissioners and funders the degree of confidence that they need. In Realising Ambition we employ tools such as the SRU’s Standards of Evidence and the Young Foundation’s Organisation Health Scorecard to have a grounded understanding of both the intervention and the organisations’ readiness for successful delivery.

4. It is easy to conflate evidence-based programmes with flexible practice…and it is harder to faithfully replicate practice

In addition to having been tested and proven effective by a robust experimental evaluation, evidence-based programmes are tightly defined packages often accompanied by a structured implementation manual and set of delivery materials. Some interventions are processes which can be broken down into discrete elements and sometimes can be hard to differentiate from programmes and are common in the public service systems. Processes are more difficult to evaluate and lend themselves to more flexible practice which can be at odds with the level of fidelity that is needed for successful replication.

5. Things tend to take longer and cost more than originally planned…

New initiatives very often face big challenges getting off the ground and it seems that these challenges and underestimations are prevalent in replication programmes. For example, the time and resources required to deliver the interventions (including recruiting and training staff) have very often been underestimated across the portfolio. As replication often means working in new areas, it seems that the time and expertise required to establish new networks has at times been misjudged. For larger organisations, navigating internal structures has meant that delivery has been delayed – at times significantly.

These are some of the early learning points from the first year of Realising Ambition and we expect that many more will unfold under the course of the five year programme, so watch this space.

This blog was drawn from the Lessons of Replication and Evidence-based Interventions: One Year In report.

.

Comments

  • (will not be published)