Original ArticleA most stubborn bias: no adjustment method fully resolves confounding by indication in observational studies
Introduction
Confounding by indication remains an often intractable threat to validity in observational studies [1]. Although confounding is best controlled by a randomized design, randomization is not always feasible. For example, patients cannot be randomized to receive placebo when an efficacious therapy is available [2]. Furthermore, trials often exclude patients with preexisting conditions [3], particularly older adults [4]. Nonrandomized designs must evaluate the effectiveness of therapies whose efficacy has been established in select groups by clinical trials, but not in broader populations that might react differently to the therapy. For these and other reasons [3], nonrandomized studies of therapy effectiveness will remain important [1]. In addition, generalizing results from clinical trials with select patient populations may actually cause harm in the heterogeneous populations treated in clinical practice [5].
As an example, clinical trials of adjuvant chemotherapy in women aged 40–59 years with early-stage breast cancer demonstrate its efficacy with reductions in 5-year mortality between 20% and 40% [6], but it is uncertain whether these benefits extend to older women, who bear the majority burden of breast cancer occurrence [7]. Nonrandomized studies of older women with early-stage breast cancer suffered from differences in prognosis between women who received adjuvant chemotherapy and women who did not receive adjuvant chemotherapy [8], [9], and thus are potentially biased by confounding by indication.
When the validity of a study is threatened by confounding by indication, it is not straightforward to determine which method of adjustment, if any, is most effective in obtaining a valid and precise estimate of effect. Conventional methods to adjust for confounding, such as restriction and multivariable regression, leave residual confounding because of unmeasured factors. Thus, propensity score (PS) adjustment and the instrumental variable (IV) approach have become increasingly popular [10], [11], [12], [13], with the intent to address this residual confounding by simulating a randomized environment. PS adjustment theoretically increases comparability between the comparison groups by creating pseudorandomization of measured confounders [14]. The goal of the IV approach is to reduce confounding by indication through the use of a variable that is associated with the exposure, unrelated to the confounders, and has no direct association with the outcome other than through the exposure [15]. However, several investigators have cautioned that these alternative methods are not universal solutions to the problem of confounding by indication [10], [11], [13], [16], [17], [18].
Three observational studies used the SEER-Medicare [19] linked data set and found that adjuvant chemotherapy decreased the rate of breast cancer-specific mortality [20] and all-cause mortality [20], [21], [22] in older women, with the greatest benefit seen in women with node-positive, estrogen receptor negative tumors [20], [21]. Based on these results and those of clinical trials among middle-aged women [6], we expect adjuvant chemotherapy to be protective against breast cancer recurrence in older women. With this prior information in mind, we compared methods used to reduce confounding. We implemented restriction, multivariable regression, PS adjustment, and an IV-like method to estimate incidence rates of breast cancer recurrence in women who received adjuvant chemotherapy compared with women who did not, in the Breast Cancer Treatment Effectiveness in Older Women (BOW) cohort [8], [9], [23].
Section snippets
Study population
The BOW cohort study was conducted at six integrated health care systems that are part of the 14-system consortium of the Cancer Research Network (CRN) [24]. The overall goal of the CRN is to increase the effectiveness of preventive, curative, and supportive interventions for major cancers through a program of collaborative research, and to determine the effectiveness of cancer control interventions that span the natural history of major cancers among diverse populations and health systems. The
Results
Frequencies for demographic and tumor characteristics for the unrestricted cohort who received primary therapy (n = 1798), the cohort restricted to women at high risk for recurrence (n = 946), the PS analytic sample (n = 723), and the IV analytic sample (n = 539) are presented in Table 1 by receipt of chemotherapy. For women classified as at high risk, 20% experienced a breast cancer recurrence. In the unrestricted, restricted, PS, and IV samples, a higher proportion of women who received adjuvant
Discussion
The association between receipt of adjuvant chemotherapy and recurrence risk in older women with breast cancer provides a useful example of the manner in which confounding by indication can complicate nonrandomized studies of treatments in general populations. When considering treatment recommendations to reduce breast cancer recurrence, oncologists treating geriatric patients take into account tumor prognostic factors and additional factors, such as life expectancy, physical function, and
Acknowledgment
Source of financial support: Supported by Public Health Service Grant R01 CA093772 (Breast Cancer Treatment Effectiveness in Older Women, Rebecca A. Silliman, PI) from the National Cancer Institute, National Institutes of Health, Department of Health and Human Services.
The authors thank Dr. Alan Brookhart for his thoughtful review and guidance regarding the instrumental variable analysis. We also thank Dr. Terry Field, site-principal investigator at Fallon Clinic/Meyers Primary Care Institute,
References (39)
- et al.
Mortality impact of less-than-standard therapy in older breast cancer patients
J Am Coll Surg
(2008) - et al.
A new method of classifying prognostic comorbidity in longitudinal studies: development and validation
J Chronic Dis
(1987) - et al.
Limits of observational data in determining outcomes from cancer therapy
Cancer
(2008) - et al.
The continuing unethical use of placebo controls
N Engl J Med
(1994) - et al.
Beyond randomized controlled trials: A critical comparison of trials with nonrandomized studies
Hepatology
(2006) In defense of pharmacoepidemiology - embracing the yin and yang of drug research
N Engl J Med
(2007)- et al.
Relation between prepublication release of clinical trial results and the practice of carotid endarterectomy
JAMA
(2000) Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials
Lancet
(2005)- et al.
Re: prevalence of cancer
J Natl Cancer Inst
(1998) - et al.
Recurrences and second primary breast cancers in older women with initial early-stage disease
Cancer
(2007)
A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods
J Clin Epidemiol
Indications for propensity scores and review of their use in pharmacoepidemiology
Basic Clin Pharmacol Toxicol
Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders
Am J Epidemiol
Preference-based instrumental variable methods for the estimation of treatment effects: assessing validity and interpreting results
Int J Biostat
The central role of the propensity score in observational studies for causal effects
Biometrika
An introduction to instrumental variables for epidemiologists
Int J Epidemiol
The use of linear instrumental variables methods in health services research and health economics: a cautionary note
Health Serv Res
Instruments for causal inference
Epidemiology
Instrumental variables: application and limitations
Epidemiology
Cited by (297)
Postmastectomy radiation therapy in breast cancer patients with micrometastatic disease in sentinel node dissection: A cohort study and meta-analysis
2024, Clinical and Translational Radiation OncologyThe importance of timing in postcardiotomy venoarterial extracorporeal membrane oxygenation: A descriptive multicenter observational study
2023, Journal of Thoracic and Cardiovascular SurgeryThe use of causal language and inferences in observational shoulder arthroplasty database studies
2023, Journal of Shoulder and Elbow SurgeryAmoxicillin therapeutic drug monitoring for endocarditis: A comparative study (EI-STAB)
2023, International Journal of Antimicrobial AgentsBronchiolitis therapies and misadventures: Plus ça change, plus c'est la même chose
2023, Paediatric Respiratory ReviewsCitation Excerpt :However, five of the six included studies were subjective in terms of their reported assessment of the overall improvement in severity of illness among participants who received ribavirin [55]. For the pediatric ICU, several case series [56–58] have reported on comparative effectiveness of ribavirin and have failed to detect an advantage with its use [59]. Overall, while its effectiveness has been incompletely investigated, ribavirin has gone out of favour since the publication of the 2006 AAP clinical practice guideline [60] and current guidelines recommend against its use (except in life-threatening RSV infections in immunocompromised patients) due to concerns regarding patient toxicity and the safety of healthcare professionals, an inconvenient route of administration, and failure to show a meaningful impact upon clinically relevant outcomes [61].