Noticing the Change: What Actually Happens When You Try to Measure Outcomes

| No responses | Posted by: Mhairi Aylott | Theme: Uncategorized

Historically the need to monitor and evaluate youth programmes was driven by outputs: the things or services provided. Now organisations are being asked to articulate their outcomes for young people, and evidence the impact that they make. This change in emphasis is partly down to policy drivers around evidence-based practice, as well as the increasing difficulty for funders in making decisions about how to spend scarce resources. We know that youth sector organisations are facing great difficulties due to governmental funding cuts. Despite organisations intrinsically knowing the power and importance of their work, there is an increasing understanding that being “self-evidently good” is not good enough. The impact of youth work needs to be evidenced.

Because of this, in July 2012 The Young Foundation, as part of the Catalyst consortium, published A Framework of Outcomes for Young People. The framework makes the case for why social and emotional capabilities matter, and why funders, commissioners and investors should have more confidence in their value. This framework generated a high level of interest and engagement, and it was clear that many voluntary youth sector organisations and their partners were reflecting on their own ‘outcomes journey.’ Once we started to hear from youth sector charities, we were asked more about the practicalities of using the framework – how can it be used and where do we start? We went on to work with three organisations, each seeking to develop an outcomes focus in their work with young people. These were Brathay Trust, the British Red Cross and London Youth.

Last week we published Noticing the Change, which documents the experience of the three youth sector organisations in piloting an outcomes approach to their work. It makes recommendations for implementing this approach to all youth sector organisations and considers the wider implications for the sector as a whole. We’ve created a list of “top ten tips” for organisations who are thinking about planning, evaluation and implementing an outcomes approach. The primary aim of this work was not about pleasing funders, or giving into commissioner’s demands. It is about showing the power and importance of intrinsic outcomes and social and emotional capabilities, and ensuring that they are recognised and valued by funders and commissioners as significant.

What did we do?

As many organisations in the youth sector are under pressure from funders and commissioners to provide evidence of their impact, our research found a natural tendency by organisations to gravitate straight towards ‘measuring something.’ However, without clarity over which outcomes to measure, attempts to develop measurement approaches are not likely to be fruitful.

Our work with the three organisations focused on this, helping them to define the outcomes of their work, and thinking practically about what can be measured and how this could be done. We did this through creating a Theory of Change – a method which aims to show a charity’s path from needs to activities and from outcomes to impact. This process aims to create a causal pathway of change, which links inputs, outputs, outcomes and shows the impact of an organisation’s work. It explains the “how” of what an organisation does and shows where and how you can measure impact. This method was integral in our approach, helping them to articulate the outcomes of their work with young people and start to think about what to measure.

Key successes

Our research found that piloting an outcome focus can have many benefits for an organisation. For example, our pilots told us that having an outcomes outlook has refocused their work, helping to refine their thinking and reinforcing what they know works with young people. It has helped them articulate what they do and given them renewed enthusiasm and motivation. As one staff member commented, it “puts steam back into what we do and why we do it.” Building the evidence base also highlights what works and why. In turn this can help shape delivery, creating the best possible services for young people. Importantly, the process helped the organisations create a common language of what they do and achieve, by placing value on social and emotional capabilities.

Challenges

Implementing an outcomes focus in any organisation is not always going to be straightforward. Our pilots came up against and overcame many challenges. Some organisations told us about a resistance to evaluate by some staff -“it went against my youth work values to make explicit what I know works in my head” and a need for a change in culture. Other respondents worried that they could become too constrained by outcomes if agreed with commissioners, losing an ability to be flexible and adaptive to young peoples’ needs. All organisations talked to us about difficulties in deciding how and what to measure. Many felt concerned about the length and style of programme. Others felt nervous not to “over-claim” the impact of their work. Picking a tool was troublesome. Difficult conversations were had about whether a tried and tested tool was more applicable than a bespoke tool, whether the tool needed to be externally validated and the method of using a tool. These were all conversations and points that needed to be fleshed out, with all organisations aware of the practicalities of introducing a new outcomes and measurement approach.

Impact on the sector

Our work piloting our Outcomes Framework with the three organisations was illuminating and highlighted the potential practicalities of the approach. However, it also showed why this approach is needed and the benefits of doing so. The organisations can now better show the impact of their work – demonstrating to funders what they already knew and highlighting why youth work is important integral. Further to this, the importance of their work was reinforced, motivating staff and empowering the young people they work with. They also showed us how they were using their impact data for funding applications and how the data internally shaped their offer for young people.

The work also demonstrated genuine progress and commitment from the sector to move this agenda forward. But what we have seen is that there needs to be support for organisations within the sector that are going through this process. The more organisations work together to support an outcomes focus in the sector, the clearer the message will be to policy makers and funders about the impact of youth work for young people and for society as a whole. We look forward to being part of this journey.

 

Comments

If your comment is published, it will be displayed along with your name. We only ask for your email address to verify that you’re a person and not a robot! Your details will not be added to any list or shared with any 3rd parties.

Otherwise, the submitted information will be deleted within 28 days.

See our privacy policy for more details.

  • (will not be published)