Summing up evidence: one answer is not always enough

doi:10.1016/S0140-6736(97)08468-7

The Lancet

Volume 351, Issue 9096, 10 January 1998, Pages 123-127

https://doi.org/10.1016/S0140-6736(97)08468-7 Get rights and content

Summary

Are meta-analyses the brave new world, or are the critics of such combined analyses right to say that the biases inherent in clinical trials make them uncombinable? Negative trials are often unreported, and hence can be missed by meta-analysts. And how much heterogeneity between trials is acceptable? A recent major criticism is that large randomised trials do not always agree with a prior meta-analysis. Neither individual trials nor meta-analyses, reporting as they do on population effects, tell how to treat the individual patient. Here we take a more rounded approach to meta-analyses, arguing that their strengths outweigh their weaknesses, although the latter must not be brushed aside.

Section snippets

Heterogeneity

The growth of meta-analysis has not been unchallenged. Critics have voiced mixed enthusiasm or objections to the principle of combining data. Some worry about combining studies with different degrees of bias: studies of different quality may reach different conclusions³ and studies with negative results may not be published.⁴ However, these biases permeate all clinical research, not just meta-analysis and, in fact, careful meta-analysis can help to identify these problems. One major controversy

Models of fixed and random effects

Most meta-analyses have asked how well a treatment works overall. This is also what trials traditionally ask. When the combined trials are a homogeneous set designed to answer the same question in the same population, a fixed-effects model, in which the estimated treatment effects vary across studies only from random error, is appropriate.¹⁰ To assess homogeneity, heterogeneity is often tested, based on the X² distribution, but this method lacks power. When heterogeneity is detected, the

Meta-regressions

Meta-regressions have been used to suggest reasons for observed heterogeneity.²⁷ As in any regression analysis, meta-regressions attempt to identify significant relations between the treatment effect (the dependent variable) and covariates of interest (the independent variables). Whereas in trials the unit of observation is the individual patient, in meta-regressions the unit of observation is the study or subgroup. From that perspective the only major difference between meta-analyses and

Conclusions

Given the problems, it is perhaps surprising that meta-analyses have agreed quite well with large trials addressing a similar “homogeneous” question.22, 24 Clinical trials and meta-analyses mostly have addressed the question of how well a treatment works overall. Both of these approaches, while useful in estimating a population effect, do not show how to treat individuals.

Patients may respond differently to treatment. To address this diversity, meta-analysis needs to evolve from deterministic

References (49)

TC Chalmers et al.
Meta-analysis stimulus for changes in clinical trials
Stat Meth Med Res
(1993)
EM Antman et al.
A comparison of results of meta-analyses of randomized controlled trials and recommendations of clinical experts: treatments for myocardial infarction
JAMA
(1992)
KF Schulz et al.
Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled clinical trials
JAMA
(1995)
K Dickersin et al.
Publication bias and clinical trials
Control Clin Trials
(1987)
KK Teo et al.
Effect of intravenous magnesium in suspected acute myocardial infarction: overview of randomised trials
BMJ
(1991)
HS Sacks et al.
Endoscopic hemostasis: an effective therapy for bleeding peptic ulcers
JAMA
(1990)
JP Ioannidis et al.
Impact of epidemic and individual heterogeneity on the population distribution of disease progression rates: an example from patient populations in trials of human immunodeficiency virus infection
Am J Epidemiol
(1996)
EJ Topol et al.
Answers to complex questions cannot be derived from “simple” trials
Br Heart J
(1992)
C Anello et al.
Exploratory or analytic meta-analysis: should we distinguish between them?
J Clin Epidemiol
(1995)
LV Hedges
Fixed effects models

CD Naylor

The case for failed meta-analyses

J Eval Clin Pract

(1996)

J Lau et al.

Quantitative synthesis in systematic reviews

Ann Intern Med

(1997)

SW Raudenbush

Random effects models

R DerSimonian et al.

Meta-analysis in clinical trials

Control Clin Trials

(1986)

TC Smith et al.

Bayesian approaches to random-effects meta-analysis: a comparative study

Stat Med

(1995)

JP Ioannidis et al.

On meta-analyses of meta-analyses

Lancet

(1996)

KL Woods et al.

Intravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2)

Lancet

(1992)

ISIS-4: A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58 050 patients with suspected acute myocardial infarction

Lancet

(1995)

effects of lisinopril and transdermal glyceryl trinitrate singly and together on 6-week mortality and ventricular function after acute myocardial infarction

Lancet

(1994)

CLASP: a randomised trial of low-dose aspirin for the prevention and treatment of pre-eclampsia among 9364 pregnant women

Lancet

(1994)

RJ Levine et al.

Trial of calcium to prevent preeclampsia

N Engl J Med

(1997)

HC Bucher et al.

Effect of calcium supplementation on pregnancy-induced hypertension and preeclampsia: a meta-analysis of randomized controlled trials

JAMA

(1996)

J Villar et al.

Predictive ability of meta-analyses of randomised controlled trials

Lancet

(1995)

J Lelorier et al.

Discrepancies between meta-analyses and subsequent large randomized, controlled trials

N Engl J Med

(1997)

Cited by (569)

Efficacy of internet-based cognitive-behavioral therapy for depression in adolescents: A systematic review and meta-analysis
2023, Internet Interventions
Internet-based cognitive behavior therapy (ICBT) may provide an accessible alternative to face-to-face treatment, but the evidence base in adolescents is limited. This systematic review and meta-analysis aims to comprehensively assess the efficacy of ICBT in addressing depression among adolescents.
Four electronic databases were searched on June 8, 2023. Randomized controlled trials (RCTs) evaluating the efficacy of ICBT for depression in adolescents were included. The quality of the studies was assessed using the risk of bias tool recommended by the Cochrane Handbook. Furthermore, the GRADE approach was employed to gauge the certainty of the obtained evidence. Meta-analysis was conducted using RevMan 5.4, and Egger's test was implemented through Stata for assessment of potential publication bias.
A total of 18 RCTs involving 1683 patients were included. In comparison to control groups like attention control, waiting list, and treatment as usual, our meta-analysis findings elucidate a significant reduction in depression scores (SMD = −0.42, 95 % CI: [−0.74, −0.11], p < .05) as well as anxiety scores (SMD = −0.34, 95 % CI: [−0.60, −0.08], p < .05) in adolescents following ICBT interventions. Furthermore, the analysis indicated no notable distinctions in patient's quality of life (QoL) scores. (SMD = 0.12, 95 % CI: [−0.10, 0.34], p > .05).
Results provide evidence of the efficacy of ICBT to reduce depressive and anxiety symptoms in adolescents. These research findings are of vital significance for the establishment of evidence-based treatment guidelines in the digital era.
PROSPERO registration: CRD42021277562.
Physical Activity Levels in People with Cancer Undergoing Chemotherapy: A Systematic Review
2023, Seminars in Oncology Nursing
Physical activity (PA) has been shown to improve chemotherapy side effects, survival rates and treatment adherence in people with cancer. This review aimed to identify whether people undergoing chemotherapy in different exercise interventions meet PA guidelines during chemotherapy.
Databases searched were CINAHL complete, PubMed, Cochrane Library, EMBASE, AMED, Joanna Briggs Institute, OVID MEDLINE, and Google Scholar. Eligibility and risk-of-bias were reviewed by two authors. PROSPERO registration: CRD42018093839.
Thirty-three studies were eligible (14 randomized controlled trials, six pre-post, and 13 cross-sectional studies), which included 2,722 people with cancer undergoing chemotherapy. Studies (n = 11/13) found PA interventions successfully increased or maintained PA participation levels, whereas PA declined without intervention. This review has identified that due to limited evidence it is not possible to determine the most appropriate intervention to improve PA for people undergoing chemotherapy. Despite the well-documented benefits of PA, most people undergoing chemotherapy do not achieve the PA guidelines. Clinicians should be aware that during treatment, participants may not meet PA guidelines even if they participate in an intervention. However, interventions may prevent PA levels from declining. Further research is required to determine the most effective approaches to increase PA levels while undergoing chemotherapy.
Nurses are well placed to have conversations with patients undergoing chemotherapy regarding PA levels. PA should be considered as part of a treatment regimen. Tailored advice must be provided aiming to improve health outcomes. Referral to an exercise professional is best practice to improve the uptake of PA.
Using the bayesmeta R package for Bayesian random-effects meta-regression
2023, Computer Methods and Programs in Biomedicine
Background: Random-effects meta-analysis within a hierarchical normal modeling framework is commonly implemented in a wide range of evidence synthesis applications. More general problems may even be tackled when considering meta-regression approaches that in addition allow for the inclusion of study-level covariables.
Methods: We describe the Bayesian meta-regression implementation provided in the bayesmeta R package including the choice of priors, and we illustrate its practical use.
Results: A wide range of example applications are given, such as binary and continuous covariables, subgroup analysis, indirect comparisons, and model selection. Example R code is provided.
Conclusions: The bayesmeta package provides a flexible implementation. Due to the avoidance of MCMC methods, computations are fast and reproducible, facilitating quick sensitivity checks or large-scale simulation studies.
Modernizing evidence synthesis for evidence-based medicine
2023, Clinical Decision Support and beyond: Progress and Opportunities in Knowledge-Enhanced Health and Healthcare
Due to the exponential expansion of the biomedical literature, systematic reviews and meta-analyses are crucial to evidence-based medicine. To answer a specific research question, a systematic review identifies, appraises, and synthesizes all practically-identifiable studies that fit pre-specified eligibility criteria. Meta-analysis refers to a set of statistical techniques for combining quantitative results from the studies included in a systematic review. This chapter provides a high-level introduction to the premise, uses, and steps of systematic reviews and meta-analyses. We also discuss how systematic reviews and meta-analyses can be accessed and the current and potential roles that they play in the evidence ecosystem. Throughout, we discuss the potential for medical informatics to improve the process of systematic reviews and meta-analysis.
Why one answer is not always enough: Reply to “CBASP may not be superior to other treatments for chronic depression”
2022, Journal of Affective Disorders
Methodological quality and risk of bias of meta-analyses of pharmacy services: A systematic review
2022, Research in Social and Administrative Pharmacy
Citation Excerpt :
All clinical decisions should be based on good-quality systematic reviews that provide a synthesis of the current best evidence, regardless of the positive or negative conclusions produced.13 However, previous studies showed that suboptimal and conflicted systematic reviews and meta-analyses have been published in the healthcare area.12,14–17 Ioannidis et al. revealed that good and truly informative studies represent only 3% of all meta-analyses published, whilst the remaining are redundant, unnecessary, unpublished, not useful or have serious methodological flaws, in part due to their poor conduct and report.15
A suboptimal meta-analysis with misleading conclusions, frequently published in the healthcare journals, can compromise decision making in clinical practice.
To evaluate the reporting quality, methodological quality, and risk of bias of meta-analyses of pharmacy services.
Systematic searches to identify all the meta-analyses reporting the effect of pharmacy services were performed in PubMed, Scopus, and Web of Science. The reporting quality, the methodological quality, and the risk of bias of the included meta-analyses were evaluated using PRISMA checklist, R-AMSTAR, and ROBIS, respectively.
A total of 109 meta-analyses were eligible for the study. The heterogeneity, the quality of evidence, and the quality analyses were poorly reported on authors’ conclusions (14.3%, 14.7%, and 17.4%, respectively). The median scores of PRISMA and R-AMSTAR tolls were 24 (IQR 21.75–25), and 30 (IQR 27–32.5), respectively. Additionally, most of the studies were considered as high risk of bias (n = 83, 76.1%). No association between the date of publication and guideline compliance exists. PRISMA score was higher in studies published in high impact factor journals (rho = 0.313; p = 0.002), in articles that reported the quality of evidence obtained (p = 0.018), and in those that stated the need for future studies in their conclusions (p = 0.011). R-AMSTAR score was higher in studies published in high impact factor journals (rho = 0.338; p = 0.001), in those which reported the quality of evidence (p = 0.002), and in articles that described the quality analyses in their conclusions (p = 0.046). An association between the risk of bias and the recognition of the need for further studies in their conclusions (p = 0.041) was also found.
The rapid increase of the meta-analyses of pharmacy services was not associated with higher quality. Mechanistic meta-analyses with poor conclusions are commonly published. Quality of the analyses, strength of evidence, heterogeneity, and absence of confrontation with current guidelines are rarely considered when synthetizing evidence and making recommendations.

View all citing articles on Scopus

View full text

SeriesSumming up evidence: one answer is not always enough

Summary

Section snippets

Heterogeneity

Models of fixed and random effects

Meta-regressions

Conclusions

Meta-analysis stimulus for changes in clinical trials

Stat Meth Med Res

A comparison of results of meta-analyses of randomized controlled trials and recommendations of clinical experts: treatments for myocardial infarction

JAMA

Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled clinical trials

JAMA

Publication bias and clinical trials

Control Clin Trials

Effect of intravenous magnesium in suspected acute myocardial infarction: overview of randomised trials

BMJ

Endoscopic hemostasis: an effective therapy for bleeding peptic ulcers

JAMA

Impact of epidemic and individual heterogeneity on the population distribution of disease progression rates: an example from patient populations in trials of human immunodeficiency virus infection

Am J Epidemiol

Answers to complex questions cannot be derived from “simple” trials

Br Heart J

Exploratory or analytic meta-analysis: should we distinguish between them?

J Clin Epidemiol

Fixed effects models

The case for failed meta-analyses

J Eval Clin Pract

Quantitative synthesis in systematic reviews

Ann Intern Med

Random effects models

Meta-analysis in clinical trials

Control Clin Trials

Bayesian approaches to random-effects meta-analysis: a comparative study

Stat Med

On meta-analyses of meta-analyses

Lancet

Intravenous magnesium sulphate in suspected acute myocardial infarction: results of the second Leicester Intravenous Magnesium Intervention Trial (LIMIT-2)

Lancet

ISIS-4: A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58 050 patients with suspected acute myocardial infarction

Lancet

effects of lisinopril and transdermal glyceryl trinitrate singly and together on 6-week mortality and ventricular function after acute myocardial infarction

Lancet

CLASP: a randomised trial of low-dose aspirin for the prevention and treatment of pre-eclampsia among 9364 pregnant women

Lancet

Trial of calcium to prevent preeclampsia

N Engl J Med

Effect of calcium supplementation on pregnancy-induced hypertension and preeclampsia: a meta-analysis of randomized controlled trials

JAMA

Predictive ability of meta-analyses of randomised controlled trials

Lancet

Discrepancies between meta-analyses and subsequent large randomized, controlled trials

N Engl J Med

Series
Summing up evidence: one answer is not always enough