Can there be a more patient-centred approach to determining clinically important effect sizes for randomized treatment trials?

doi:10.1016/0895-4356(94)90176-7

Journal of Clinical Epidemiology

Volume 47, Issue 7, July 1994, Pages 787-795

https://doi.org/10.1016/0895-4356(94)90176-7 Get rights and content

Abstract

Sample sizes for treatment trials with categorical outcomes are conventionally derived by balancing three elements: a difference between alternative treatments in the event rates for the outcomes of interest (commonly termed the clinically important difference), the alpha error tolerance (false positive risk) and the beta error tolerance (false negative risk). Clinically important differences used to plan trials are chosen in part based on earlier experience with similar interventions (i.e. biological or clinical plausibility). Methodological conventions and clinicians' perceptions will also affect choices. Lastly, practical concerns about the feasibility of accruing large numbers of subjects may drive trialists to specify bigger differences as clinically important, with a view to containing sample size requirements. We suggest that patients or other members of the public be given an active role in determining the magnitude of the clinically important treatment effect for trial planning. Probability trade-offs could be constructed to enable patients and/or healthy volunteers to indicate the degree of benefit they would want from a “new” treatment, given the potential side-effects of the same treatment. This method has the advantage of respecting patient autonomy and principles of informed consent. It provides an additional consideration when plausible effect sizes and error tolerances on hypothesis tests are balanced against feasibility of accruing various sample sizes. Its primary disadvantage is inconvenience, as it adds another step to trial design. On the other hand, if patient-based clinically important differences are generated for a variety of disease states and types of treatments, specific trade-off exercises may be needed only for unusual trials. Another disadvantage is that patients' perspectives may differ markedly from clinicians' or from society at large, leading to conflicts of perspective that could prove difficult to resolve. However, we believe that raising and addressing such conflicts of perspective is itself a useful part of the evaluative process.

References (33)

B. Burnand et al.
Indexes and boundaries for “quantitative significance” in statistical decisions
J Clin Epidemiol
(1990)
A.S. Detsky
Using economic analysis to determine the resource consequences of choices made in planning clinical trials
J Chron Dis
(1985)
H.A. Llewellyn-Thomas et al.
Patients' willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation
Soc Sci Med
(1991)
D.L. Sackett et al.
The utility of different health states as perceived by the general public
J Chron Dis
(1978)
A.M.C. O'Connor et al.
Eliciting preferences for alternative drug therapies in oncology: Influence of treatment outcome descrip tion, elicitation technique, and treatment experience on preference
J Chron Dis
(1987)
A.M. O'Connor
Effects of framing and level of probability on patients' preferences for cancer chemotherapy
J Clin Epidemiol
(1989)
L. Forrow et al.
Absolutely relative: how research results are summarized can affect treatment decisions
Am J Med
(1992)
C.L. Meinert
Clinical Trials. Design, Conduct, Analysis
(1986)
J. Freiman et al.
The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial
N Engl J Med
(1978)
A.S. Detsky et al.
When was a “negative” clinical trial big enough? How many patients you needed depends on what you found
Arch Intern Med
(1985)

D.L. Sackett et al.

Clinical Epidemiology: a Basic Science for Clinical Medicine

S. Yusuf et al.

Why do we need some large, simple randomized trials?

Stat Med

(1984)

A.S. Detsky et al.

Establishing therapeutic equivalency. What is a clinically significant difference?

Arch Intern Med

(1986)

H.A. Llewellyn-Thomas et al.

Patients versus surrogates: whose opinion counts on ethics review panels?

Clin Res

(1989)

A. Tversky et al.

The framing of decisions and the psychology of choice

Science

(1981)

GISSI-2: a factorial randomised trial of alteplase versus strepokinase and heparin versus no heparin among 12,490 patients with acute myocardial infarction

Lancet

(1990)

Cited by (65)

A critical review of methods used to determine the smallest worthwhile effect of interventions for low back pain
2012, Journal of Clinical Epidemiology
Citation Excerpt :
Alternatively, the trial could be designed to detect effects that are at least as large as the median, or the 20th percentile, or the 80th percentile of the smallest worthwhile effect. Naylor and Llewellyn-Thomas [62] have suggested that it would be prudent to use the effect considered to be worthwhile by just 25% of participants. Similar considerations apply when data on the smallest worthwhile effect are used to make decisions about the clinical importance of an effect reported in a clinical trial.
To critically and systematically review methods used to estimate the smallest worthwhile effect of interventions for nonspecific low back pain.
A computerized search was conducted of MEDLINE, CINAHL, LILACS, and EMBASE up to May 2011. Studies were included if they were primary reports intended to measure the smallest worthwhile effect of a health intervention (although they did not need to use this terminology) for nonspecific low back pain.
The search located 31 studies, which provided a total of 129 estimates of the smallest worthwhile effect. The estimates were given a variety of names, including the Minimum Clinically Important Difference, Minimum Important Difference, Minimum Worthwhile Reductions, and Minimum Important Change. Most estimates were obtained using anchor- or distribution-based methods. These methods are not (or not directly) based on patients’ perceptions, are not intervention-specific, and are not formulated in terms of differences in outcomes with and without intervention.
The methods used to estimate the smallest worthwhile effect of interventions for low back pain have important limitations. We recommend that the benefit–harm trade-off method be used to estimate the smallest worthwhile effects of intervention because it overcomes these limitations.
Minimal change is sensitive, less specific to recovery: A diagnostic testing approach to interpretability
2011, Journal of Clinical Epidemiology
Citation Excerpt :
Each will be described briefly. The most common approach to interpreting serial measures of health is the discernment of the minimal clinically important change/difference (MCID); approaches that estimate the much sought after threshold for a minimal amount of change; in a patient-reported outcome score, which indicates that a patient is better (or worse) [6–14]. The McMaster group led the early work in this area balancing this small but important change against the absence of excessive costs and risks [10,11] .
The pursuit of interpretability of longitudinal measures of patient outcome has led to several methods for defining minimal amounts of change or final states that are important. Little is known about the best method. The purpose of this study was to directly compare methods using diagnostic utility to evaluate their usefulness.
Secondary analysis of longitudinal cohort data of persons attending physiotherapy for shoulder pain. Disability of the arm, shoulder, and hand outcome fielded at baseline and 3 months. Published methods were used to define positive response in scores: minimal change, final state, and combined change and final state. Proportions described as improved were compared (Kappa) and diagnostic testing techniques used to evaluate the strengths of each.
Only moderate agreement was found between methods (Kappa = 0.47). Minimal clinically important differences were most sensitive but not specific. Final states were less sensitive, more specific, and most accurate. Combinations were slightly less specific.
A new approach allowed us to evaluate the relative merits and risks of different approaches to interpreting longitudinal patient outcomes. Our study points to a combination of change greater than error and/or a final score within general population norms as being the most clinically sensible with strong diagnostic accuracy.
Patient-reported outcome instrument selection: Designing a measurement strategy
2007, Value in Health
To discuss issues in the design of a measurement strategy related to the use of patient-reported outcomes (PROs) in support of a labelling claim.
In association with the release by the US Food and Drug Administration of its draft guidance on the use of PROs to support labeling claims, the Mayo/FDA Patient-Reported Outcomes Consensus Writing Group was formed. This paper, part of a series of manuscripts produced by the Writing Group, focuses on designing a PRO measurement strategy.
Developing a PRO measurement strategy begins with a clear statement about the proposed label claim that will derive from the PRO data. Investigators should identify the relevant domains to measure, develop a conceptual framework, identify alternative approaches for measuring the domains, and synthesize the information to design the measurement strategy.
Often, there is not an already existing single instrument that has been developed and validated for the purposes of a given study. In such cases, investigators may consider supplementing an already existing questionnaire with additional scales or questions, modifying already existing instruments for a new application or patient population, or developing a new instrument altogether. The level of revalidation required for modifications and adaptations depends on the extent of the changes made. Revalidation requirements may range from cognitive testing/debriefing to confirm that subjects respond to the new instrument as expected to full-scale reliability and validity evaluations.
A position of “reasonable pragmatism” is recommended such that the best available measurement strategy be considered as evidence for labeling.
Estimating sample size in critical care clinical trials
2005, Journal of Critical Care
Estimating the required sample size for a study is necessary during the design phase to ensure that it will have maximal efficiency to answer the primary question of interest. Clinicians require a basic understanding of the principles underlying sample size calculation to interpret and apply research findings. This article reviews the critical components of sample size calculation, including the selection of a primary outcome, specification of the acceptable types I and II error rates, identification of the minimal clinically important difference, and estimation of the error associated with measuring the primary outcome. The relationship among confidence intervals, precision, and study power is also discussed.
Preference assessment of recruitment into a randomized trial for adolescent idiopathic scoliosis
2008, Journal of Bone and Joint Surgery
Citation Excerpt :
One could argue that the preferences and participation decisions from the surrogate group should not be used to predict the responses of patients approached during actual trial recruitment. Many previous simulations of recruitment for a randomized controlled trial have used surrogates in the place of patients with the actual condition under study40–43,45,72. There are two main reasons for the use of surrogates in place of actual patients.
Randomized controlled trials are powerful tools to evaluate the outcomes of clinical treatments. However, these trials tend to be expensive and time-consuming, and their conclusions can be threatened by several limitations. This study estimated the strength of three common limitations (underenrollment, selective enrollment, and nonadherence to protocol) in a proposed study of adolescent idiopathic scoliosis.
Patients with scoliosis and their parents were asked to complete a web-based survey about their preferences concerning a hypothetical randomized trial. Adolescents without scoliosis and their parents also participated. Surveys included questions about treatment preference, likelihood of participation, required risk reduction, and propensity to drop out or choose a different treatment while enrolled in the study.
Ninety adolescents and eighty-three parents participated. Observation was preferred to bracing by the majority of subjects. Overall, 33% of the parents and adolescents would both agree to participate in the hypothetical trial. Of the subjects who would not agree to participate, the majority would rather share the decision-making responsibility with the physician than have the treatment chosen in a random fashion. Many of the subjects would consider changing treatments during the course of the trial if they were not satisfied with the outcomes; the majority of parents who preferred bracing would consider crossing over to the bracing arm if their children were randomized to observation.
Recruitment into a randomized trial of bracing compared with observation for the treatment of adolescent idiopathic scoliosis may well be problematic, considering the relatively small percentage of families who said they would consider randomization. Additionally, the threat of nonadherence to protocol may be strong and must be addressed in the protocol of the trial. Most families wanted to make the treatment decision with the physician in lieu of randomization; therefore, the role of the physician in patient recruitment and retention should not be underestimated.
Are colorectal surgery patients willing to accept an increased risk of surgical site infection to avoid mechanical bowel preparation? Implications for future trial design
2022, Colorectal Disease

View all citing articles on Scopus

View full text

Original articleCan there be a more patient-centred approach to determining clinically important effect sizes for randomized treatment trials?

Abstract

J Clin Epidemiol

J Chron Dis

Soc Sci Med

J Chron Dis

J Chron Dis

J Clin Epidemiol

Am J Med

Clinical Trials. Design, Conduct, Analysis

The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial

N Engl J Med

When was a “negative” clinical trial big enough? How many patients you needed depends on what you found

Arch Intern Med

Clinical Epidemiology: a Basic Science for Clinical Medicine

Why do we need some large, simple randomized trials?

Stat Med

Establishing therapeutic equivalency. What is a clinically significant difference?

Arch Intern Med

Patients versus surrogates: whose opinion counts on ethics review panels?

Clin Res

The framing of decisions and the psychology of choice

Science

GISSI-2: a factorial randomised trial of alteplase versus strepokinase and heparin versus no heparin among 12,490 patients with acute myocardial infarction

Lancet

Original article
Can there be a more patient-centred approach to determining clinically important effect sizes for randomized treatment trials?