Opinion: The best interventions often have poor results — in the short run

This article was written by Sarah Morton, Director, Matter of Focus and Honorary Fellow, The University of Edinburgh. 

Many governments, funders, programmes and projects around the world are adopting outcome approaches to express their vision for social change, and commissioning services based on high-level outcomes (for example the SDG’s). 

These outcome approaches include aspirational ideas like ending poverty, or ending violence against children. 

Research shows that these approaches can be a great way to get people together to work towards changes that are needed to improve lives and make the world a better place. They can help people stay focused in a complex and changing environment, bring partnerships and teams together around a shared vision, and help highlight what really matters to people and communities.

But working towards these high-level goals or outcomes also brings with it some challenges. 

One of the biggest of these challenges is related to how we measure progress. If the focus is solely on the data about the high level outcome, then there will be some obstacles to overcome.

It gets worse before it gets better

The key problem is that data on outcomes usually gets worse before it gets better. 

There are a couple of obvious reasons for this: For example when evaluating  a campaign to make schools safer, outcome data might include a reduction in the number of school pupil reporting experiencing violence, gathered via opinion surveys or administrative data on the number of violent incidents. Both of these measures are likely to increase when a school adopts a safer schools approach. There are a number of reasons for this: 

• School pupils and teachers get the message that the leadership are taking violence seriously so are more likely to report it.
• Awareness campaigns mean that children and staff have a better understanding of the different kinds of violence (physical, emotional, sexual, bullying etc) and so report on these different dimensions of violence.
• If new support services are implemented, people are more likely to come forward with recent and historical incidents of violence than when there was no support available.

So a good safer school or violence reduction programme should expect to see these rises in measures of violence in the short and medium term. However, this may alarm those implementing, authorising or funding a programme. 

The leadership challenge

The leadership challenge is to understand this data and work with it confidently while a programme is being implemented. 

For leaders to be able to do this they need a strong understanding of the context they work in, the relationship between activities and outcomes, and some really solid data telling them that implementation is going well.

The cornerstone for working well with this leadership challenge is an understanding of the difference between contribution and attribution. 

For complex programmes like making schools safer or reducing violence against children the relationship between programmes and their outcomes is not simple and direct. Instead there are many factors that will also influence the same outcomes:

• The attitudes and awareness of pupils, teachers and families.
• What else is going on in the community (e.g. prevalence of violence, external events, disasters).
• The policies and services of local government, NGO’s and other actors that can also offer support to the same community.

If all of these other factors are working together (e.g. violence reduction programmes elsewhere, a policy environment that is working towards violence reduction, positive changes in community cohesion, wealth, employment, domestic abuse etc.) then changes in violence levels will be achieved more quickly and more easily than if the wider context presents difficulties and challenges. 

For this reason we focus on the contribution of any programme, initiative or policy, rather than attributing simple cause and effect. Indeed contribution analysis is the foundation of our approach at Matter of Focus. 

What other data is needed?

For leaders driving change like safer schools programmes, different kinds of then other data and evidence are needed to maintain confidence that the programme is making a difference in the short term, even as high-level outcomes look worse. 

At Matter of Focus we work with programmes, projects and initiatives to understand their contribution by including the following ways of thinking about and working with complex change:

1. Understand how activities contribute to outcomes: this means having a strong theory of change for all aspects of the programme.
2. Understand who is engaging: this means knowing that the programme is reaching the right recipients and that they are engaging positively. 
3. Understand the learning that underpins behaviour change: that means knowing and getting feedback on the knowledge, skills and capacities that are important for the programme.
4. Monitoring behaviour and practice changes: collecting real-time data on early changes in behaviours and practices, however small.
5. Monitoring external factors that might affect the programme or initiative: (wider policies, services and unexpected events).

This can be done through collecting immediate feedback from people who are engaged in the programme, ensuring systematic reflections from those delivering the programme, and must include qualitative insights in order to understand the effectiveness of programme implementation. 

Working towards ending violence and creating safer schools

Over a longer time frame, if implementation goes well, changes in high level outcomes will start to emerge. 

By using a complexity-informed approach like the one described above, people involved in delivering these kind of programmes will be able to able to tell a really credible story of how they made a difference. That story will include how violence reduction has been influenced by their programmes and projects, as well as an assessment of the wider influences of the context for delivery. 

This evidence will provide valuable insights for other people seeking to implement similar programmes or initiatives in other places. More than that it will allow deep learning as programmes are implemented, meaning they are more likely to reach their outcomes and make a difference to people and communities. 

There is an imperative to work in this way if public services are going to be successful in tackling some of the big and complex challenges that we currently face. — Sarah Morton

(Photo credit: Death to the stock photo)

Source