Research gaps identified during systematic reviews of clinical trials: glass-ionomer cements

Abstract

Background

To report the results of an audit concerning research gaps in clinical trials that were accepted for appraisal in authored and published systematic reviews regarding the application of glass-ionomer cements (GIC) in dental practice

Methods

Information concerning research gaps in trial precision was extracted, following a framework that included classification of the research gap reasons: ‘imprecision of information (results)’, ‘biased information’, ‘inconsistency or unknown consistency’ and ‘not the right information’, as well as research gap characterization using PICOS elements: population (P), intervention (I), comparison (C), outcomes (O) and setting (S). Internal trial validity assessment was based on the understanding that successful control for systematic error cannot be assured on the basis of inclusion of adequate methods alone, but also requires empirical evidence about whether such attempt was successful.

Results

A comprehensive and interconnected coverage of GIC-related clinical topics was established. The most common reasons found for gaps in trial precision were lack of sufficient trials and lack of sufficient large sample size. Only a few research gaps were ascribed to ‘Lack of information’ caused by focus on mainly surrogate trial outcomes. According to the chosen assessment criteria, a lack of adequate randomisation, allocation concealment and blinding/masking in trials covering all reviewed GIC topics was noted (selection- and detection/performance bias risk). Trial results appear to be less affected by loss-to-follow-up (attrition bias risk).

Conclusion

This audit represents an adjunct of the systematic review articles it has covered. Its results do not change the systematic review’s conclusions but highlight existing research gaps concerning the precision and internal validity of reviewed trials in detail. These gaps should be addressed in future GIC-related clinical research.

Introduction

Systematic reviews are defined, according to the Cochrane collaboration, as scientific literature reviews aimed at answering clearly formulated questions through use of systematic and explicit methods for identifying, selecting and critically appraising relevant research, and for collecting and analysing data from the literature included in the review [1]. Through this process, systematic reviews provide a unique opportunity for also identifying gaps in the currently available research on a particular topic and providing recommendations for addressing these. In order for such objective to become part and parcel of the regular systematic review process, it has been suggested that the identification of research gaps should follow a systematic approach that is based on widely accepted elements and provides transparent and reproducible results [2].

Robinson et al. (2011) identified the lack of a systematic process for the development of ‘future research’ sections in systematic reviews and developed a framework for determining research gaps [2]. The framework included a classification of research gap reasons and research gap characterization, using the PICOS elements: population (P), intervention (I), comparison (C), outcomes (O) and setting (S). Reasons for research gaps are classified as ‘imprecision of information (results)’, ‘biased information’, ‘inconsistency or unknown consistency’ and ‘not the right information’. In addition, a worksheet with instructions for facilitating the use of such framework during or after systematic reviews was developed [2].

In order to systematically identify potential research gaps in clinical trials on the topic of glass-ionomer cements (GIC) in dentistry, an audit of all authored and previously published GIC-related systematic reviews [3–10] was conducted on the basis of recommendations by Robinson et al. (2011) [2]. All systematic reviews were conducted and published between March 2010 and March 2011 on clinical topics covering the application of conventional (C-GIC) and resin-modified GIC (RM-GIC). Although each contained recommendations for future research in their discussion section, these were not derived on the basis of a systematic framework.

Therefore, the purpose of this article is to provide an addition to authored and previously published GIC-related systematic reviews by reporting on the results of an audit concerning identified research gaps in clinical trials covering the application of glass-ionomer cements in dental practice.

Materials and method

All authored systematic reviews published on the topic of glass-ionomer cement (GIC) [3–10] were selected and reviewed in relation to aspects of the precision and internal validity of the accepted clinical trials.

Information concerning trial precision was extracted, following recommendations by Robinson et al. regarding a framework for identifying research gaps during systematic reviews [2]. In the context of this audit, the following classification was adopted for investigating research gaps in the precision of clinical trials:

• (i) Imprecision of results: e.g. confidence intervals (95% CI) wide enough to suggest both superiority and inferiority at the same time; no, or too few trials found; too small sample size of clinical trials accepted in systematic reviews [2, 11, 12].

• (ii) Inconsistency/unknown consistency of results: e.g. conflicting directions of effect sizes; magnitude of differences in effect sizes and significance of such differences; non-overlapping confidence intervals (95% CI); unexplained clinical and/or statistical heterogeneity; lack of sufficient trials (at least two trials are needed in order to assess consistency) [2, 11].

• (iii)  Lack of right information: e.g. only surrogate outcomes investigated; follow-up period too short [2].

PICOS characteristics of research gaps included information (if available) on (P): age, gender, ethnicity and clinical patient characteristics; (I) and (C): specific name(s) of treatment(s); (O): the measured/indented clinical outcomes (S): type of clinical setting. All extracted information was entered into a modified abstraction worksheet. Modifications of the worksheet included:

• (i) Exclusion of ‘Bias information’ (assessment for systematic error/bias risk was undertaken separately in more depth than per the original recommendations);

• (ii) Inclusion of all gap reasons, instead of only the most important one (thus addressing one of the identified weaknesses of the framework) [2];

• (iii)  Sub-grouping of clinical topics per systematic review (where appropriate).

The internal validity of clinical trials was assessed through three of the systematic reviews [3, 4, 7], following recommendations by Berger [13]. Such assessment of internal trial validity is based on the understanding that successful control for systematic error cannot be assured on the basis of inclusion of adequate methods alone; i.e. adequate generation of random sequence, allocation concealment, blinding/masking, as part of the trial methodology. Such inclusion, and even the adequate reporting of such inclusion by following the CONSORT statement [14], merely indicates an attempt to control adequately against systematic error. In addition, empirical evidence is required as part of the trial report, from which it may be ascertained that such an attempt was indeed successful. Without such evidence, internal validity of clinical trials cannot be assured [13].

This form of assessment was not yet adopted in five systematic reviews [5, 6, 8–10]. Thus all accepted clinical trials of these were re-assessed, using the more recent criteria (Table 1). All results of the internal validity assessment were collated with each clinical topic per systematic review.

Table 1 Quality assessment criteria of trials

Results

Research gaps in the precision of clinical trial results

Extracted information indicative of existing research gaps concerning the precision of clinical trial results related to GIC are shown in Table 2. The most common reasons for gaps are found in the ‘Imprecision of results – A’, followed by ‘Inconsistency of results – B’. These are due to a general lack of clinical trials; insufficient sample size in available trials and wide 95% confidence intervals, as well as to large differences in effect sizes, unexplained existing he-terogeneity of data and the difficulty in judging consistency from single trials, respectively (Table 3). Only few research gaps have been ascribed to ‘Lack of information – C’; i.e. assessment of mainly surrogate, instead of clinical out-comes.

Table 2 Research gaps abstraction worksheet related to trial precision

Table 3 Details of identified research gaps in clinical topics related to trial precision

The result of the PICOS assessment (Table 2) shows that in total 19 different clinical topics were appraised through the eight systematic reviews [3–10]. Of these, nine topics related to interventions (I) using RM-GIC and eight to C-GIC (of which five were related to ART implementation). Two topics comprised clinical comparisons (I and C) of both GIC classes with each other, concerning the permanent and primary dentition (P) with regard to caries in restoration margin (O). The clinical work of most reviewed trials was performed in standard dental clinical settings (S).

The comparison (C) for seven of the eight C-GIC topics included amalgam restorations and one resin-based pit and fissure sealant as controls. The subject population (P) of the C-GIC topics were patients with permanent (5 topics) and primary (3 topics) dentition. The outcomes (O) were related to caries on pits and fissures, caries on restoration margins and restoration success rates for one, two and five topics, respectively.

RM-GIC was compared (C) against composite resin (with and without fluoride), resin-based fissure sealant material and calcium hydroxide cement in seven, one and one topics, respectively. The outcomes (O) were related to caries on pits and fissures (1 topic), caries on restoration margins (4 topics), caries around orthodontic brackets (1 topic), caries on adjacent teeth (2 topics) and pulp response (1 topic). The studied population (P) comprised patients with permanent (6 topics) and primary (3 topics) dentition.

Table 3 provides further information as to the specific clinical topics affected by the identified research gaps. Gaps were identified in regard to clinical comparisons of C-GIC and RM-GIC versus amalgam, composite resin (with or without fluoride) and calcium hydroxide cement to clinical outcomes, related to caries on restoration margins as well pits and fissures, restoration success rates, orthodontic bracket bonding failures and pulp response in deep cavities.

Research gaps in the internal validity of clinical trial results

Table 4 presents the results of the assessment of systematic error/bias risk affecting the internal validity of clinical trial results. This table shows the number of datasets judged according to the chosen assessment criteria (Table 1). Only criteria “A” indicate adequacy. According to the chosen assessment criteria, a complete lack of adequate randomisation, allocation concealment and blinding/masking in all the reviewed GIC topics can be noted. Trials related to the topics of GIC-based fissure sealants and GIC (including ART) comparison to amalgam restorations included limited evidence for equivocal baseline characteristics between intervention groups. Trial results for most GIC topics appear to be less affected by loss-to follow up (attrition bias).

Table 4 Assessment of systematic error/bias risk of clinical trials per topic

Discussion

This is the first attempt to adopt a framework for determining research gaps in systematic reviews [2] on a specific subject in dentistry. This attempt is limited to authored systematic review articles concerning GIC only, and may thus not reflect the totality of potentially existing research gaps regarding GIC. However, one of the challenges reported by the team that developed the framework was that they were not involved with the conduct of the evidence review and the writing of its results [2]. In seeking to avoid any negative impact of such challenge, it was decided that only authored systematic reviews relevant to GIC should be selected and other systematic reviews based on a systematic literature search on the GIC topic would not be included. For that reason, the present audit should not be considered in a stand-alone capacity, but as an adjunct and partial update of the systematic review articles [3–10] it has covered. As the scope of this audit did not affect reported effect sizes and overall values of established internal validity, its results do not change the systematic reviews’ conclusions. Instead, the audit results give empirical support to the general recommendations, already made in the original systematic review reports [3–10], for further research needs in terms of precision and validity.

Identified research gaps in the precision of clinical trial results

The PICOS framework in Table 2 shows a comprehensive and interconnected coverage of GIC-related clinical topics that were appraised through the eight systematic reviews [3–10]: Topics focused on the two relevant clinical classes of glass-ionomer cement (GIC): conventional GIC and resin-modified GIC (RM-GIC). Both material classes are related to each other but also differ in aspects of clinical indication and biocompatibility. Hence, systematic review evidence was appraised for both in terms of anticariogenic effects regarding preventive, i.e. pit and fissure protection [3, 4] and restorative indications [4, 6, 7]. In addition, evidence for RM-GIC was appraised for its orthodontic indication [7] and claims of its potential pulp toxicity were reviewed [10]. Further evidence regarding comparison of the potential anticariogenic effects of GIC and RM-GIC [9] was assessed. With the development of the atraumatic restorative treatment approach (ART), on a C-GIC basis, current clinical evidence concerning the overall success rate of ART fillings in comparison to amalgam as prevailing gold standard was included [5].

The results presented in the modified abstraction worksheet suggest a general lack of trials on many GIC-related topics as the main reason for research gaps (Table 2). The same topics suffer from weak statistical power due to small sample sizes of available trials and related to wide confidence intervals that render their results inconclusive (Table 3).

A particular need for more trials of suitable sample size has been identified for clinical topics concerning results that were identified as statistically significant: the longevity (restoration success rate) of Class II and V ART restorations in permanent teeth, in comparison to that of amalgam fillings placed with the use of traditional high-speed drills. Systematic review results concerning Class II and V restorations identified a statistically significant improvement of longevity favouring ART by 28% over amalgam after 6.3 years, and by 61% after 2.3 years, respectively [5]. However, according to the findings in this audit (Table 2 and 3), the precision of these results are to be regarded as low and thus future studies are needed in order confirm or reject the initial results.

In a meta-analysis comparing C-GIC and amalgam restorations, amalgam restorations were found to have a 65% higher chance than C-GIC restorations of having caries on restoration margins after 6 years [4]. Although this difference in effect size was reported to be statistically significant (p = 0.001), its precision is limited, owing to the small number of trials (n = 2) included.

Identified research gaps in the internal validity of clinical trial results

According to the chosen assessment criteria (Table 1), none of the clinical trials accepted for further appraisal in the eight systematic reviews [3–10] included adequate randomisation/allocation concealment or blinding/masking, thus suggesting a high risk of selection and detection/performance bias (Table 4). Some of the trials from three systematic reviews [3–5] reported on the lack of statistically significant differences between selected covariates of the test- and control groups at baseline. Such lack of significant difference means a lack of evidence that randomisation was ineffective. However, this may not provide enough evidence that randomisation was effective either, as other non-measured and/or unknown covariates may exist that differ significantly between groups as a result of potential lack of effective randomisation. Hence, reasons for doubt remain regarding this point and risk of selection bias may still be high.

Bias or systematic error constitutes any factor in the knowledge acquisition process that systematically diverts its outcomes away from true values [15]. Systematic error, therefore, limits the internal validity of acquired knowledge and thus renders all clinical results included in this audit - even those for which no research gaps in terms of precision were identified (Table 2 and 3) - in need for confirmation by higher quality future trials.

Meta-epidemiological studies have attempted to investigate and quantify the bias risk resulting from lack of adequate randomisation/allocation concealment or blinding/masking. The empirical evidence of these studies was appraised by a Cochrane systematic review in regard to randomisation and allocation concealment [16]. Its conclusions affirmed that the effect of not using randomisation could be as large as, or larger than, the expected effects of intervention, and therefore recommend the use of random allocation and adequate allocation concealment. However, the identified evidence for this conclusion was identified as weak [16]: larger effect estimates were found in non-randomised trials than in randomised studies of the same intervention and measured outcome. This is in agreement with the suggestion that effect-overestimation due to bias is likely to be most common [17]. However, the results originated from only one single meta-epidemiological study [18] and were based on analysis of vote counting, with no quantitative estimates of effect [16]. Other studies that investigated randomised versus non-randomised trials of different interventions and/or measured outcomes provided overall only inconsistent and conflicting evidence [16]. More studies that consider the influence of heterogeneity due to differences in clinical intervention and outcome measures are needed in order to provide better evidence concerning the impact of potential bias risk on clinical trial results.

In the absence of strong empirical evidence of bias, e.g. due to lack of adequate randomisation, arguments for correcting potential bias impact still rely largely on logical considerations. Berger [13] has added to the debate by presenting the logical consideration that successful control for bias cannot be assured on a basis of inclusion of adequate methods, (adequate generation of random sequence, allocation concealment, blinding/masking) alone. Such inclusion would only indicate an attempt to control bias adequately but additionally requires evidence, as part of the trial report, from which it may be ascertained that such an attempt was indeed successful. Without such evidence, so it is argued, high internal validity of clinical trials cannot be assured [13]. Even though the empirical basis for this argument remains weak, the initial evidence [16] appears to support its adoption into the methodology of randomised controlled trials (RCT). Such adoption would also have the benefit of generating a volume of specific trials that would be available for future comparison in meta-epidemiological studies: by comparing the outcome of RCTs, which have adopted the above considerations with those that have not. An empirical investigation still provides the best method for conclusively answering questions concerning adequate bias control. Against this background, the following recommendations may be made for future GIC-related RCTs:

• (i) Inclusion into the trial report of a detailed description of the randomisation process, methods and restriction, as well as detailed description of the allocation concealment process;

• (ii) Inclusion into the trial methodology of any possible form of quantitative test that may identify the presence of selection bias; e.g. test for association of the trial end point with the probability P{E} that a certain patient receives a certain treatment (test- or control intervention), as well as inclusion of such test results into the trial report;

• (iii)  Inclusion into the trial report of a detailed description of any form of blinding/masking attempt;

• (iv)  Inclusion into the trial methodology of any possible form of quantitative test that may identify the presence of detection/performance bias; e.g. appraisal of the opinions of patients and independent evaluators as to which type of treatment a patient may have received, followed by statistical comparison of such data with the actual allocated treatment sequence, as well as inclusion of the test results into the trial report.

Conclusions

This audit identified research gaps in the precision and internal validity of clinical trials on topics regarding the application of glass-ionomer cements. Gaps concerning precision are mainly manifest in the insufficient trial number and insufficient sample size. Gaps concerning internal validity are related to uncertainties of potential bias risk due to inadequate randomisation, concealment of the random allocation, as well as blinding/masking. It is recommended that these research gaps may be addressed by focussing in future randomised controlled trials on the identified reasons.

Author’s contribution

SM developed the concept and outline and wrote this paper. The author read and approved the final manuscript.