Mass colorectal cancer screening: Methodological quality of practice guidelines is not related to their content validity

https://doi.org/10.1016/j.clinbiochem.2007.12.020Get rights and content

Abstract

Objectives

It is not clear if good methodological quality in developing practice guidelines (GLs) necessarily leads to valid recommendations that, when implemented, are more likely to improve the balance between benefits and harms/costs. We assessed whether or not there is a link between methodological quality and recommendation validity in GLs for the use of fecal occult blood test (FOBT) as a screening test for colorectal cancer (CRC) in the average-risk population.

Methods

We systematically searched for such practice GLs published in English or in French within the last 7 years. Our inclusion criteria limited the initial 36 GLs to 12. Scores for methodological quality based on the AGREE criteria were attributed to each of these 12 GLs. Likewise, we searched for meta-analysis and other systematic reviews (SRs) addressing the issue, and we selected for inclusion 8 SRs that met basic quality criteria according to the Critical Appraisal Skills Programme (CASP) of the National Health Service of the United Kingdom (NHS). We used the results and conclusions of these 8 SRs to establish the validity of recommendations made in the 12 included GLs.

Results

Regarding methodological quality, the GLs were labeled either “strongly recommend” (n = 3), “recommend with provisos or alterations" (n = 3), “would not recommend" (n = 2), or “unsure” (n = 4). The nine GLs recommending for, as well as the three GLs recommending against, mass-screening are equally valid, because the former base their recommendation on the fact that this can decrease CRC-mortality, whereas the latter base their recommendation on the facts that: (1) this procedure would be too expensive and/or not adapted to their local organization of care, and (2) to a lesser extent, the balance between benefits and harms is not crystal-clear from an individual patient perspective.

Conclusion

The fact that the 12 GLs fell short of basic quality criteria confirms many previous observations in various areas of medicine. Because the 12 GLs were found to be equally valid regarding their FOBT-related recommendations, no relation could be found between their methodological quality and their content validity.

Introduction

The practice of evidence-based medicine (EBM) means integrating the best research evidence with clinical expertise and patient values [1]. It is clearly established that the current methodological quality of clinical practice guidelines (GLs), including evidence-based GLs, needs to improve considerably [2], [3]. As a consequence, a lot of energy is currently being spent to try to improve the situation [4]. However, it is not really established whether or not better methodological quality leads to more valid recommendations, i.e. those that are supported with consistent research evidence or, when evidence is conflicting or lacking, with sufficient consensus among the GL development team, and that, when implemented, are more likely to improve the balance between benefits and harms/costs.

In previous work [5], we showed that validity, and methodological quality of recommendations in GLs were not related to each other in 11 GLs providing advice for the use of laboratory tests in non-small cell lung cancer patients. However, this work had two main limitations: (1) some of the systematic reviews (SRs) that we used to establish the validity of the laboratory-related recommendations were our own SRs; (2) the evidence in the field we investigated was not of high quality; in particular, no randomized controlled trials (RCTs) had been published. We therefore wished to undertake similar work in an area where the evidence is of higher quality (i.e. RCTs are available), and where more SRs have been published, by workers other than ourselves.

In colorectal cancer (CRC) patients, cure or long-term survival is much more likely to be achieved when the disease is diagnosed at an early stage. SRs of RCTs have shown that fecal occult blood test (FOBT) is a screening test that can decrease CRC-mortality at a population level. Many governmental or professional organizations worldwide advocate FOBT mass-screening for CRC.

Section snippets

Practice guidelines

We systematically searched for GLs providing advice for the use of FOBT as a screening test in colorectal cancer, using the strategy described previously which includes manual searches [6]. Key search terms were “colorectal cancer” and “practice guidelines”. We excluded GLs published before 2000 for the same reasons as those which led us to exclude SRs published before 2000 (see next section). We also excluded GLs whose recommendations were unclear or not relevant to our work, e.g. GLs

Practice guidelines

Our literature search retrieved 36 GLs, and our selection criteria enabled us to include 12 of these [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23]. The FOBT-related recommendations made in these 12 GLs are summarized in Table 1. All GLs agree, formally or implicitly, that FOBT can only be used as a screening test for colorectal cancer in asymptomatic, average-risk patients, thus excluding patients with a familial or personal history of colorectal cancer, as well as

Discussion

Many of the 12 GLs fell short of basic quality criteria, a result that confirms previous observations in various areas of medicine [3]. Among the six domains comprising the AGREE instrument which are generally accepted to cover the key elements of the GL development process [4], the 1st domain, i.e. scope and purpose, is the one which is the most satisfactorily addressed, with scores above 80% in 11 of 12 GLs. Likewise, the domain which is the least satisfactorily addressed is the 6th domain,

Conclusion

It is generally believed that evidence-based GLs can provide health-care professionals with valid recommendations, i.e., that are supported with consistent research evidence or sufficient consensus among the GL development team when evidence is conflicting or lacking. These, when implemented, should be more likely to improve the balance between benefits and harms, than the opposite [32]. Our preliminary work [5] as well as the present study would not really confirm this belief, at least in the

Acknowledgments

The authors acknowledge the helpful comments from Dr Catherine Dubé, Gastroenterologist, Calgary, Alberta, Canada. This work was presented during a symposium entitled “evidence in action” (S.Sandberg, W.Oosterhuis, Chairmen) at EUROMEDLAB on the 3 rd of June 2007 (RAI Congress Centre, Amsterdam, The Netherlands).

References (37)

  • A.R. Horvath et al.

    Quality of guidelines for the laboratory management of diabetes mellitus

    Scand J. Clin. Lab. Invest Suppl.

    (2005)
  • V.M. Montori et al.

    For the Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey

    BMJ

    (2005)
  • P. Hewitson et al.

    Screening for colorectal cancer using the faecal occult blood test

    Hemoccult. Cochrane Database Syst Rev.

    (Jan 2007)
  • A.D. Oxman et al.

    Users' guides to the medical literature. VI. How to use an overview. Evidence-based medicine working group

    JAMA

    (1994)
  • D.L. Sackett et al.

    Evidence-based medicine: what it is and what it isn't

    BMJ

    (1996)
  • R.A. Smith et al.

    American Cancer Society guidelines for the early detection of cancer

    CA Cancer J. Clin.

    (2006)
  • J.M. Rhodes

    Colorectal cancer screening in the UK: joint position statement by the British Society of Gastroenterology, The Royal College of Physicians, and The Association of Coloproctology of Great Britain and Ireland

    Gut

    (2000)
  • D. Leddin et al.

    Canadian Association of Gastroenterology and the Canadian Digestive Health Foundation: guidelines on colon cancer screening

    Can. J. Gastroenterol.

    (2004)
  • Cited by (16)

    • Critical care guidelines on pain, agitation and delirium management: Which one to use? A systematic literature search and quality appraisal with AGREE II

      2020, Journal of Critical Care
      Citation Excerpt :

      AGREE II evaluates the methodology of guideline development, technical aspects and the dissemination, but not the quality of a guideline's content or validity of its statements [24]. This is an important distinction, as high-quality developmental process does not necessarily translate into high-quality content [40]. The number of recommendations varies significantly, which suggests a difference in either depth and/or covered scope of the CPGs.

    • Systematic review of the methodological quality of clinical guideline development for the management of chronic disease in Europe

      2012, Health Policy
      Citation Excerpt :

      The reasons for study exclusion during the full text screening are reported in Table 2. Nine studies reported in 10 papers were included, analysing a total of 28 European guidelines from eight European countries and two pan-European organisations [33–42] (Table 3). The clinical remit and scope of the guidelines varied, with four of the nine papers focusing on type 2 diabetes (management of diabetes in pregnancy [37], laboratory management of type 2 diabetes [36], clinical management of diabetes type 2, e.g. management of blood glucose, lipids and renal disease [34,40,41]); two studies focused on cancer (colorectal cancer screening [33] and breast cancer [42]), one on cardiovascular disease [38], one on depression [35] and one on the management of knee osteoarthritis [39].

    • Quality evidence important for quality guidelines

      2010, CMAJ. Canadian Medical Association Journal
    View all citing articles on Scopus
    View full text