17 August 2022
NIHR ARC West researchers have adapted the ‘Ripple Effects Mapping’ (REM) method for use in measuring impact in complex public health interventions. REM is a participatory and qualitative form of impact evaluation that aims to capture changes that are traditionally thought to be ‘hard to measure’. This makes it particularly helpful in evaluating complex public health interventions such as those designed to make people more physically active.
In a blog for BMC Medical Research Methodology, Dr James Nobles of NIHR ARC West and Dr Jennifer Hall of the Bradford Institute for Health Research, write:
“Traditional impact evaluations tend to measure change in pre-specified outcomes. Let’s say that 10 people attend a cycling training course; a traditional impact evaluation might measure anticipated outcomes, such as people’s confidence in cycling, at the beginning and end of the course.
“But what happens if a few people went on to set up a cycling club or start lobbying for safer roads where they live? These are just two examples of unanticipated impacts that could lead to further societal benefits or more sustainable effects.
“So, shouldn’t we find a way to capture wider impacts of interventions in a meaningful way? Over the last few years, we have adapted a method called Ripple Effects Mapping to do this.”
The blog and accompanying paper describe in detail how to implement an REM approach, using participatory workshops.
The research team have also created a series of bitesize training videos to describe their adapted REM methodology.
They adapted REM as part of the NIHR ARC West project to evaluate the we can move programme in Gloucestershire. We can move is a whole systems approach that brings together public, private, voluntary and community organisations to develop a shared understanding of what causes physical inactivity. This shared understanding is used to help bring about change. Given the complexity and scale of We Can Move, REM was invaluable in trying to understand what happened within the programme and what some of the associated impacts were. In total, the research team ran 19 REM workshops within the two-year evaluation period, and did so in-person and online.
They also applied it to another associated project, exploring the wider impacts of a community building initiative in Gloucester city. Community builders get to know local residents and work to encourage connection and mutual support among neighbours, as well as the development of resident-led projects and activities. REM workshops were used to help understand the impact this project had on the community.
Alan Inman-Ward, Director of Insight and Operations at Active Gloucestershire, said:
“Ripple Effects Mapping has been invaluable to Active Gloucestershire in measuring the growth and impact of we can move. Stakeholders collaborated in engaging workshops to discuss the wider impacts which had taken place, giving us new insight into the actions and impact resulting from the various projects. The act of bringing people together for evaluation purposes also enabled them to connect and inspire each other. Becoming an intervention in itself; creating further ripples.”
Will Chapman, Senior Programme Manager (Healthy Communities and Individuals), NHS Gloucestershire Integrated Care Board, said:
“Much of the work we do around tackling health inequalities and supporting people to adopt healthy lifestyles and behaviours takes place in complex systems in which direct, attributable, quantitative measurement of impact is extremely difficult. We are always looking for alternative ways of understanding if our inputs are having the desired positive effects.
“REM has given us a way of looking at this differently. It has proved to be a useful tool to represent and understand some of the direct impacts of our work and also the unforeseen, positive (or negative) consequences, or ripples. It is another useful asset in the evaluation arsenal. When combined with other forms of evaluation, REM can help us build a more holistic picture and understanding of both our impact and how we might evolve our approaches.”
Dr James Nobles, Research Fellow at NIHR ARC West, said:
“We’ve been really pleased with all the interest in this method over the last year. We think that there is so much value to it, not only as an evaluation tool, but as a tool to help drive continuous programme improvement.
“It’s relatively simple to use from a data collection perspective and can create very powerful visuals demonstrating the diversity of project activities and impacts. We’re excited to see how the method continues to evolve in the future.”