In evaluating the impact of social services, a common inherent assumption is that interventions are delivered by one single cohesive entity. The usual reality of service delivery, however, looks more a patchwork – a mix of small, local and often voluntary organisations.
To help bridge this gap between evidence and real-world practice, CEI has partnered in a multi-site randomised controlled trial of youth mentoring, working across 17 youth organisations across the UK – with YMCA George Williams College and Bryson Purdon Social Research, and funded by the Youth Endowment Fund. With successful results in hand, the program is now moving to a full efficacy trial to measure the impact of mentoring on young people’s socio-emotional development and self-confidence.
At the heart of this novel approach is the aim to measure impact in a program that remains locally adaptable and relevant.
“We believe we have come up with a new means to evaluate social programs delivered across multiple small organisations – something that potentially has application across many social services,” says CEI Advisor Jamie Rowland.
The feasibility trial
The feasibility stage has established a shared model of mentoring practice to be used by all participating organisations. Developed in collaboration with frontline practitioners, the model outlines both core and flexible components of practice.
“The aim was not to construct the ‘most effective’ mentoring offer, but rather to bring together the most common, evidence-informed elements that can be put into practice,” Jamie explains. “That enabled us to be confident that practices are similar enough to enable assessment of comparative impact and to generate learnings.”
“This ‘common elements’ approach is the glue of the trial. It strikes a balance between being clear on core elements of mentoring delivery while encouraging high levels of flexibility in responding to young people’s needs.”
Ninety-three at-risk young people (most aged 10 to 15) were involved in the feasibility trial: 47 were randomised to the control group and waited 12 weeks before receiving mentoring support, while 46 were randomised into the intervention group and received mentoring immediately.
CEI played a leading role in supporting agencies to deliver the trial. “Through close collaboration, we ensured that young people completed relevant outcome measures, feedback surveys and qualitative interviews,” Jamie says. “Additional insights, related to implementation of the trial requirements in practice, were gained through interviews with mentors and service managers.”
Feasibility moves to efficacy
The common practice arrangements were shown to be overall feasible and acceptable to both the delivery teams and the young people involved.
“The shared practice model was delivered with good fidelity to the core elements of the program, was viewed as feasible to deliver by mentors and managers, and was acceptable to young people,” Jamie explains.
“Qualitative interviews with mentees, mentors and service managers described positive impacts for young people relating to resilience, confidence, engagement with support networks, and being guided away from anti-social behaviour and crime.”
The efficacy trial, which aims to recruit 850 young people across the 17 community organisations, commenced in February. Young people across the UK have already begun to receive mentoring support and recruitment is continuing.
“By exploring what it takes to make the multi-site approach possible, this study is key to ensuring our evaluation of youth mentoring reflects real-world practice across the UK,” says CEI Senior Advisor and project leader, Dr Sweta Gupta.
“We are continuing to build useful knowledge of how to undertake multi-site trials – a methodological approach that embeds evaluation design within existing complex, devolved social systems. This could be applied in across range of community settings.”
“Most importantly, we hope to inform a more rounded understanding of how best to support young people’s socio-emotional learning, as they continue to face ongoing marginalisation.”
CEI's Jamie Rowland, Dr Stephanie Smith, Amy Hall, Jane Lewis and Dr Sweta Gupta worked with YMCA George Williams College and Bryson Purdon Social Research to deliver the multi-site trial, funded by the Youth Endowment Fund.