The RSS training programme includes a unique course on Bayesian meta-analysis. Here our tutor, Robert Grant of BayesCamp talks us through what the course will cover and what the Bayesian approach to meta-analysis is all about.
Read more about the course and book for 11 October (London)
Meta-analysis is a statistical procedure to combine findings reported by multiple studies into one overall conclusion. It is often used in biomedical research and to develop clinical guidelines, but recently has been adopted in other policy settings too, notably in governments’ response to Covid.
In the early days of the pandemic, the sparsity of good scientific evidence and the difficulty of combining it into evidence-based policy was laid bare in the news. The extent of the challenges came as a surprise to many, but also highlighted the value of good evidence synthesis. A single study may impact on the care of many people, but a good meta-analysis can set a national or international standard that impacts on billions of people.
If we understood every nuance of every study, and every reason why they obtain different findings, then we might -- at least in theory -- make a complete statistical model for all the studies. But in reality, we must find a balance between the risk of omitting information and the risk of obtaining a misleading finding from a mish-mash of evidence.
We often hear people speak of "comparing apples and oranges"; the psychologist Hans Eysenck called this kind of erroneous combination not meta-analysis but "mega-silliness". While it is alleged that he falsified some of his own studies, his critique is still sound on this point. If we know that studies differ, and we have some clues as to why this is, then we would be neglectful to simply throw the numbers together. A more complex problem requires a more complex analysis.
Bayesian methods give us the flexibility to address these challenges. They use probability to represent uncertainty in any unknown, not just those that arise from random samples that can be repeated. They have been widely adopted in multilevel models, which take into account data that come from clusters, such as students belonging to universities or patients to hospitals.
These are not just simple random samples: if we repeated a data collection, we would get different patients but the same hospitals. The meta-analysis problem is rather like this, even though we do not usually have access to data on the individual students or patients.
Bayesian analysis gives us estimates and measures of uncertainty for the study differences as well as the overall finding, and this is important because we can use it to validate our calculations against the (often qualitative) information we have on study subjects, methods and analyses.
There are other complicating factors too. Perhaps not all the studies compare the same two interventions, which leads to the idea of "network meta-analysis". Small studies lead to inaccuracies when we rely on the usual calculations. Often, there are upper and lower limits to an outcome scale, such as a depression questionnaire, which break some standard assumptions too. Bias can also play a role, suppressing non-significant findings and over-estimating effect sizes.
These complications require a more bespoke approach, which takes meta-analysis outside the grasp of most biomedical researchers, who are not trained in advanced statistical analysis.
This is the gap that the RSS “Bayesian Meta-Analysis” course aims to bridge. It’s an introductory tour of Bayesian models for meta-analysis and the various challenges that are regularly found in the evidence base. It is aimed at researchers who know the basics of meta-analysis and the principles of Bayesian statistics (for example, from
my course using Stan software
or Richard Morey's using JAGS).
There has been little advice available until now for meta-analysts who want to get started in Bayesian methods. This course brings together a wide range of ideas from methodologists, good and bad examples of Bayesian meta-analyses, and my own experience from working on NICE guidelines and academic research.
We will use Stan and R to run flexible models of our own design (the code can easily be adapted to Python, BUGS or JAGS). Each problem we consider will be related to a real-life evidence base and grounded in considerations of the details in those studies. We will use biomedical examples, but the methods can be universally applied.
After taking the course, participants will be able to start designing and justifying Bayesian meta-analyses, running them, critiquing them and communicating them.
Photo by Jasmin Mehovic.