Review ArticleLanguage of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary
Introduction
Systematic reviewers have little control over random errors but can exert some influence over systematic errors (bias). Including only a portion of all available evidence in a systematic review may introduce bias into the review process and threaten its validity. The most comprehensive search strategies would include all relevant literature, regardless of language of publication, but identifying, obtaining, and translating non–English language reports can significantly increase the time, cost, and effort required for investigators working in English. Grégoire et al. [1] reported that 78% of identified systematic reviews had language of publication restrictions. The majority (93%) of these restrictions were at the expense of excluding reports of randomized controlled trials (RCTs) published in languages other than English (LOE).
The question of whether language restrictions are a sensible policy for systematic reviewers has been explored in two methodological directions. In earlier work from our group, Moher et al. [2] set out to address whether the quality of reporting in LOE differs in some meaningful way from English-language (EL) reports. Their findings provide little ground for the language-restriction policy, because there were no differences between LOE and EL reports in quality of reporting with respect to randomization, double-blinding, dropouts and withdrawals, and allocation concealment.
In another methodological direction, several authors have examined the impact of excluding reports in LOE on the meta-analytical results of systematic reviews. Case studies on the issue result in varying conclusions. Excluding LOE reports did not change estimates of the intervention effectiveness in two systematic reviews, one examining the efficacy of beta-blockers and the other examining the efficacy of intravenous streptokinase for acute myocardial infarction [3], [4]. The addition of LOE reports in four language-restricted systematic reviews changed the treatment effect estimate in one review [1].
Members of our group have approached this issue in a systematic way. We earlier [5] identified 18 systematic reviews that explicitly stated using no language restrictions in their search to identify RCTs and included reports in LOE in their data synthesis (i.e., language-inclusive systematic reviews). Results of each systematic review were examined in which LOE reports were either included or excluded from the analysis. That earlier review provided no evidence that language-restricted systematic reviews lead to different estimates of intervention effectiveness, compared with language-inclusive ones. Most (68%) of the systematic reviews we examined in that study, however, included only a single LOE report. We then suggested additional research to replicate our study using different sampling frames, clinical areas and interventions [5].
We further report here on the extent to which including LOE reports influences the results of systematic reviews using a broad dataset of language-inclusive systematic reviews, including both conventional medicine (CM) and complementary and alternative medicine (CAM) interventions. Using this dataset, we have examined whether language restrictions affect the estimates of an intervention's effectiveness, whether any such impact is similar for CM and CAM interventions, and whether the results are influenced by other issues in the systematic review process, including publication bias and statistical heterogeneity.
Section snippets
Systematic review eligibility criteria
A systematic review was included if it was published in English, if the primary data sources were reports of RCTs, and if the methodology section of the report explicitly stated whether only English reports were eligible or whether trials reported in other languages were considered. In addition, the language-inclusive systematic reviews had to include at least one LOE report on the meta-analytic outcomes of interest to us.
Search strategy
The search strategy aimed to identify systematic reviews of RCTs
Results
Descriptive details regarding the systematic reviews included in these analyses are presented in a companion article [19]. Briefly, the 42 systematic reviews (34 CM and 8 CAM) from 41 publications were published in a variety of paper-based peer-reviewed journals and the Cochrane Database of Systematic Reviews between 1984 and 1999 (see Appendix). These systematic reviews included 662 RCTs (120,545 participants), of which 133 were LOE trials (17,810 participants). The trials investigated a broad
Discussion
Our results suggest that limiting the language of publication of trial reports to English in systematic reviews of CM interventions does not change the estimates of an intervention's effectiveness. This is specific to systematic reviews of CM interventions and is consistent with those reported previously [5]. In that work, Moher et al. identified 18 systematic reviews that included 178 English-language trial reports and 33 publications in seven other languages. Excluding these LOE reports
Acknowledgments
The present study was funded by the National Health Service (NHS) Research & Development Programme (Health Technology Assessment, Grant no. 96/52/99). The authors thank Manchun Fang for help in generating Fig. 1, Fig. 2.
References (32)
- et al.
Selecting the language of the publications included in a meta-analysis: is there a Tower of Babel bias?
J Clin Epidemiol
(1995) - et al.
Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews
Lancet
(1996) - et al.
What contributions do languages other than English make on the results of meta-analyses?
J Clin Epidemiol
(2000) - et al.
Should meta-analysts search Embase in addition to Medline?
J Clin Epidemiol
(2003) - et al.
Meta-analysis in clinical trials
Control Clin Trials
(1986) - et al.
Do certain countries produce only positive results? A systematic review of controlled trials
Control Clin Trials
(1998) - et al.
Language bias in randomised controlled trials published in English and German
Lancet
(1997) - et al.
Location bias in controlled clinical trials of complementary/alternative therapies
J Clin Epidemiol
(2000) - et al.
Meta-analysis of clinical trials as a scientific discipline. I: Control of bias and comparison with large co-operative trials
Stat Med
(1987) - et al.
Meta-analysis of clinical trials as a scientific discipline. II: Replicate variability and comparison of studies that agree and disagree
Stat Med
(1987)
Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses
Health Technol Assess
The inclusion of reports of randomised trials published in languages other than English in systematic reviews
Health Technol Assess
Holistic pediatrics: a research agenda
Pediatrics
Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research
Stat Med
Combining estimates of the odds ratio: the state of the art
Stat Methods Med Res
A general estimator for the variance of the Mantel-Haenszel odds ratio
Am J Epidemiol
Cited by (164)
PARENTAL EDUCATION, CARIES EXPERIENCE, PLAQUE ACCUMULATION, AND Mutans Streptococci COUNT MAY PREDICT THE INCREMENT OF EARLY CHILDHOOD CARIES IN PRESCHOOL CHILDREN OVER 24 MONTHS
2024, Journal of Evidence-Based Dental PracticeLanguage abilities in children and adolescents with DLD and ADHD: A scoping review
2023, Journal of Communication DisordersRestricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review
2021, Journal of Clinical EpidemiologyCitation Excerpt :No changes in the overall review conclusions were reported due to language restrictions. [22] Changes in statistical heterogeneity due to language restrictions were assessed in one study, published in two articles. [4,10] The authors noted a reduced statistical heterogeneity, from an I2 greater than 0.30 to less or equal to 0.3 (no heterogeneity) in 3/42 reviews (one related to conventional and two to complementary and alternative topics) after the exclusion of non-English publications.
Language restrictions in systematic reviews should not be imposed in the search strategy but in the eligibility criteria if necessary
2021, Journal of Clinical EpidemiologyChoose your shortcuts wisely: COVID-19 rapid reviews of traditional, complementary and integrative medicine
2020, Integrative Medicine Research