Review Article
Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary

https://doi.org/10.1016/j.jclinepi.2004.08.021Get rights and content

Abstract

Objective

To assess whether language of publication restrictions impact the estimates of an intervention's effectiveness, whether such impact is similar for conventional medicine and complementary medicine interventions, and whether the results are influenced by publication bias and statistical heterogeneity.

Study Design and Setting

We set out to examine the extent to which including reports of randomized controlled trials (RCTs) in languages other than English (LOE) influences the results of systematic reviews, using a broad dataset of 42 language-inclusive systematic reviews, involving 662 RCTs, including both conventional medicine (CM) and complementary and alternative medicine (CAM) interventions.

Results

For CM interventions, language-restricted systematic reviews, compared with language-inclusive ones, did not introduce biased results, in terms of estimates of intervention effectiveness (random effects ratio of odds ratios ROR = 1.02; 95% CI = 0.83–1.26). For CAM interventions, however, language-restricted systematic reviews resulted in a 63% smaller protective effect estimate than language-inclusive reviews (random effects ROR = 1.63; 95% CI = 1.03–2.60).

Conclusion

Language restrictions do not change the results of CM systematic reviews but do substantially alter the results of CAM systematic reviews. These findings are robust even after sensitivity analyses, and do not appear to be influenced by statistical heterogeneity and publication bias.

Introduction

Systematic reviewers have little control over random errors but can exert some influence over systematic errors (bias). Including only a portion of all available evidence in a systematic review may introduce bias into the review process and threaten its validity. The most comprehensive search strategies would include all relevant literature, regardless of language of publication, but identifying, obtaining, and translating non–English language reports can significantly increase the time, cost, and effort required for investigators working in English. Grégoire et al. [1] reported that 78% of identified systematic reviews had language of publication restrictions. The majority (93%) of these restrictions were at the expense of excluding reports of randomized controlled trials (RCTs) published in languages other than English (LOE).

The question of whether language restrictions are a sensible policy for systematic reviewers has been explored in two methodological directions. In earlier work from our group, Moher et al. [2] set out to address whether the quality of reporting in LOE differs in some meaningful way from English-language (EL) reports. Their findings provide little ground for the language-restriction policy, because there were no differences between LOE and EL reports in quality of reporting with respect to randomization, double-blinding, dropouts and withdrawals, and allocation concealment.

In another methodological direction, several authors have examined the impact of excluding reports in LOE on the meta-analytical results of systematic reviews. Case studies on the issue result in varying conclusions. Excluding LOE reports did not change estimates of the intervention effectiveness in two systematic reviews, one examining the efficacy of beta-blockers and the other examining the efficacy of intravenous streptokinase for acute myocardial infarction [3], [4]. The addition of LOE reports in four language-restricted systematic reviews changed the treatment effect estimate in one review [1].

Members of our group have approached this issue in a systematic way. We earlier [5] identified 18 systematic reviews that explicitly stated using no language restrictions in their search to identify RCTs and included reports in LOE in their data synthesis (i.e., language-inclusive systematic reviews). Results of each systematic review were examined in which LOE reports were either included or excluded from the analysis. That earlier review provided no evidence that language-restricted systematic reviews lead to different estimates of intervention effectiveness, compared with language-inclusive ones. Most (68%) of the systematic reviews we examined in that study, however, included only a single LOE report. We then suggested additional research to replicate our study using different sampling frames, clinical areas and interventions [5].

We further report here on the extent to which including LOE reports influences the results of systematic reviews using a broad dataset of language-inclusive systematic reviews, including both conventional medicine (CM) and complementary and alternative medicine (CAM) interventions. Using this dataset, we have examined whether language restrictions affect the estimates of an intervention's effectiveness, whether any such impact is similar for CM and CAM interventions, and whether the results are influenced by other issues in the systematic review process, including publication bias and statistical heterogeneity.

Section snippets

Systematic review eligibility criteria

A systematic review was included if it was published in English, if the primary data sources were reports of RCTs, and if the methodology section of the report explicitly stated whether only English reports were eligible or whether trials reported in other languages were considered. In addition, the language-inclusive systematic reviews had to include at least one LOE report on the meta-analytic outcomes of interest to us.

Search strategy

The search strategy aimed to identify systematic reviews of RCTs

Results

Descriptive details regarding the systematic reviews included in these analyses are presented in a companion article [19]. Briefly, the 42 systematic reviews (34 CM and 8 CAM) from 41 publications were published in a variety of paper-based peer-reviewed journals and the Cochrane Database of Systematic Reviews between 1984 and 1999 (see Appendix). These systematic reviews included 662 RCTs (120,545 participants), of which 133 were LOE trials (17,810 participants). The trials investigated a broad

Discussion

Our results suggest that limiting the language of publication of trial reports to English in systematic reviews of CM interventions does not change the estimates of an intervention's effectiveness. This is specific to systematic reviews of CM interventions and is consistent with those reported previously [5]. In that work, Moher et al. identified 18 systematic reviews that included 178 English-language trial reports and 33 publications in seven other languages. Excluding these LOE reports

Acknowledgments

The present study was funded by the National Health Service (NHS) Research & Development Programme (Health Technology Assessment, Grant no. 96/52/99). The authors thank Manchun Fang for help in generating Fig. 1, Fig. 2.

References (32)

  • D. Moher et al.

    Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses

    Health Technol Assess

    (1999)
  • D. Moher et al.

    The inclusion of reports of randomised trials published in languages other than English in systematic reviews

    Health Technol Assess

    (2003)
  • K.J. Kemper et al.

    Holistic pediatrics: a research agenda

    Pediatrics

    (1999)
  • J.A. Sterne et al.

    Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research

    Stat Med

    (2002)
  • J.D. Emerson

    Combining estimates of the odds ratio: the state of the art

    Stat Methods Med Res

    (1994)
  • J. Robins et al.

    A general estimator for the variance of the Mantel-Haenszel odds ratio

    Am J Epidemiol

    (1986)
  • Cited by (164)

    • Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review

      2021, Journal of Clinical Epidemiology
      Citation Excerpt :

      No changes in the overall review conclusions were reported due to language restrictions. [22] Changes in statistical heterogeneity due to language restrictions were assessed in one study, published in two articles. [4,10] The authors noted a reduced statistical heterogeneity, from an I2 greater than 0.30 to less or equal to 0.3 (no heterogeneity) in 3/42 reviews (one related to conventional and two to complementary and alternative topics) after the exclusion of non-English publications.

    View all citing articles on Scopus
    View full text