Original Article
The reporting quality of meta-analyses improves: a random sampling study

https://doi.org/10.1016/j.jclinepi.2007.10.008Get rights and content

Abstract

Objectives

To determine the overall quality of reporting of meta-analyses (QUOROMs) of randomized controlled trials (RCTs) based on the QUOROM statement, to compare the reporting quality of paper-based articles and Cochrane reviews, and to determine whether compliance with the statement improves over time.

Study Design and Setting

A random sample of systematic reviews or meta-analyses of RCTs was selected from Medline (2000–2005).

Results

A total of 161 articles were included. The mean QUOROM score was 12.3 (95% confidence interval [CI]: 12.0, 12.6), which rose from 10.5 (95% CI: 8.8, 12.1) in 2000 to 13.0 (95% CI: 12.2, 13.8) in 2005. The mean QUOROM scores of Cochrane reviews and paper-based articles were 14.2 (95% CI: 13.9, 14.5) and 11.7 (95% CI: 11.3, 12.1), respectively. Compared with the paper-based articles, the Cochrane reviews had better reporting quality in the abstract section, while the quality of their trial flows was poor. The fulfillment of most QUOROM items improved with time. A linear relation of the QUOROM score with time was revealed.

Conclusions

The reporting quality of meta-analyses improves with time. The reporting quality of Cochrane reviews is better compared with paper-based articles. Room still exists for improvements in the reporting quality of both Cochrane and paper-based articles.

Introduction

The number of meta-analyses of randomized controlled trials (RCTs) has increased drastically in recent years. Meta-analysis, as a quantitative approach to combine results from similar RCTs, has earned a crucial position in providing useful information for evidence-based medicine (EBM) and health care decision making [1], [2], and the results it generates are regarded as the highest level in EBM studies [3].

Like any research enterprise, a meta-analysis can be flawed [4]. The authors of a meta-analysis report not only on the methods they used to review the articles, but also on the research methods in the articles. With this in mind, there has been an increased focus on the quality of reporting of meta-analyses (QUOROMs). A 1987 survey has discussed and explored several aspects of a meta-analysis of randomized trials including the influence of study design, control of bias, statistical analysis, and applicability of the results [5]. In 1996, the QUOROMs conference took place, where individuals with various backgrounds came together to develop guidelines for the reporting of meta-analyses. Their enormous efforts led to the publication of the QUOROM statement in 1999, which consists of a checklist and a flow diagram [6]. The checklist includes 18 items placed in six categories. It is meant to help authors improve their reporting quality and, to some extent, help with the evaluation of quality of meta-analyses.

Since its release, many journal editors and reviewers, including those involved in the Cochrane Collaboration, have pursued compliance with the QUOROM statement to provide sound, clear, and reproducible results [4], [7], [8], [9]. Several articles have evaluated the compliance with the statement [2], [10], [11], [12], [13], but the fields they focused on were limited and very small samples were given, generating results that were not always consistent with each other. In the articles, further research efforts dedicated to the influences of the QUOROM statement were called for. The overall impact of the statement on the design, conduct, and reporting of meta-analyses is yet to be unveiled.

With the exception of a few studies completed by Shea et al. [14] and Jadad et al. [15], little is known about the quality of Cochrane reviews and paper-based articles. To our knowledge, the comparison of reporting quality between Cochrane reviews and paper-based articles using the QUOROM checklist has not been evaluated nor studied.

The main purposes of this study are to determine the overall QUOROMs of RCTs based on the QUOROM statement, and to determine whether the reporting quality of meta-analyses improves with time. We also intend to compare the reporting quality of the electronic Cochrane reviews with that of the paper-based articles.

Section snippets

Search strategy and sample size

A literature search was conducted using Medline from January 2000 to December 2005 to identify meta-analyses of RCTs. The following strategy was used for search: (randomized controlled trial$ .mp. [mp = title, abstract, name of substance word, subject heading word]) AND ((systematic review OR meta analys$) .mp. [mp = title, original title, abstract, name of substance word, subject heading word]). Only articles published in English were accepted. We planned to select a random sample, a minimum

Results

Figure 1 is the flowchart for the selection of eligible articles. A total of 5,218 articles were retrieved, of which 368 were excluded for duplicates. Of the remaining 4,850 articles, 5% were selected as the study sample (n = 243) by random sampling. Of the 243 sampled articles, 82 were excluded, 48 provided no quantitative synthesis, 16 combined data from both RCTs and non-RCTs, 9 were articles of methodological research, 5 were Cochrane reviews updated in 2006, 3 were unclear whether or not the

Discussion

We often tell clinicians, health care policy makers, students, and consumers that systematic reviews represent “the best available evidence.” Unfortunately, there is relatively little empirical data on the QUOROMs or systematic reviews of RCTs. The primary purpose of this study was to determine the overall QUOROMs of RCTs. Although the results demonstrated a basically acceptable reporting quality (mean QUOROM score = 12.3), certain aspects in reporting still need improving.

In terms of complying

References (17)

There are more references available in the full text version of this article.

Cited by (62)

  • Quality of meta-analyses in major leading orthopedics journals: A systematic review

    2017, Orthopaedics and Traumatology: Surgery and Research
    Citation Excerpt :

    It was estimated that at least 2500 new MAs (including systematic reviews, SRs) were indexed in Medline yearly [3]. However, lots of MAs were methodologically poorly reported and could introduce bias and impair the reliability of conclusions although rigorous methodology is a feature of MAs [4,5]. To prevent such incidents, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA) was developed in 2009 (Supplemental data 1) [1].

View all citing articles on Scopus
View full text