28 March 2025
A new framework for the conduct and use of natural experimental evaluations has been published to support health researchers and decision makers. The project involves Professor Frank de Vocht at the University of Bristol and NIHR ARC West.
Natural experiments are studies that take advantage of real-world events or situations that divide a population into different groups, often without the researcher’s intervention. These events or changes, like new policies, laws, or environmental shifts, allow researchers to observe and evaluate their effects on different groups.
Examples of natural experiments include the introduction of a minimum price for the sale of a unit of alcohol in Scotland, the UK-wide soft drinks industry levy, or the database errors that led to the failure of the system for testing for COVID infections in some parts of the country.
Natural experiments provides valuable opportunities for the evaluation of new and existing policies, programmes and services that may improve population health, including those where a randomised trial would be difficult to conduct.
When government policies, services or infrastructure are introduced or changed, this gives useful opportunities for evaluation of these natural experiments.
Researchers are making increasing use of these opportunities, but definitions and methods are still the subject of debate. The framework is aimed at a broad audience including health researchers, practitioners, policy and decision makers, funders and journal editors. It intends to help:
The framework updates and extends previous guidance by taking account of recent developments in approaches to evaluating complex interventions and showing how they can be applied in natural experimental studies.
The framework explains:
The framework is available in two forms:
Development of the framework was led by the MRC/CSO Social and Public Health Sciences Unit with a team of experts, an advisory group and an oversight group. The team gathered expert opinions in workshops and an online consultation with researchers from the UK, Europe, Australasia and North and South America. As well as researchers, members of research funding boards, policymakers, journal editors and representatives of national and local governments contributed views and advice.
Professor Brian Ferguson, Director, Public Health Research Programme, NIHR, said:
“Public health research requires a range of different approaches and it is good to see this latest framework for natural experiments. Studies using this methodology need to be robust and of high quality so that decision makers can have confidence in local and national policy evaluation.”
Professor Frank de Vocht, Professor of Epidemiology and Public Health at the Centre for Public Health and NIHR ARC West at the University of Bristol, and one of the authors of the new framework, said:
“The evidence from natural experiment evaluations of new policies and other interventions has been getting more and more important in recent years to inform public health and healthcare.
“As the previous guidance was over a decade old, I am really pleased we were able to develop this new framework incorporating new ideas and approaches and contemporary methodologies. It will hopefully be a valuable addition for researchers, policymakers, and others, and will ultimately result in better evidence to improve outcomes for patients and the public.”
Professor Graham Hart, Professor of Sexual Health & HIV Research at University College London, Co-Director of UCL Health of the Public, and Chair of the MRC Applied Global Health Research Board, said:
“Having chaired the Oversight Group that supported the development of the Framework I was impressed at the scientific rigour and hard work that went into getting this right. This is a critical addition to the armamentarium of evaluation methods and will strengthen public health science in the policy arena.”
Professor Nick Wareham, Chair of the MRC Population Health Sciences Group (PHSG), said:
“This new framework is very welcome. I hope it encourages the population health research community – researchers, policy-makers and practitioners – to make full use of the opportunities natural experiments provide to produce and apply the best possible evidence to support decision-making.”
Professor Peter Craig, Professor Public Health Evaluations, MRC/CSO Social and Public Health Sciences Unit and lead author of the framework, said:
“On behalf of the writing team, I’d like to thank all the people who helped us to develop the framework – our funders and advisory group, the workshop participants and everyone who responded to our consultation. I hope they’ll use the framework in their work as researchers, journal editors, research funders and public health decision-makers, and that they’ll engage with it critically so that we can continue to improve the way we all use the whole evaluation toolkit.”