Original ArticleTime-dependent bias was common in survival analyses published in leading clinical journals
Introduction
Bias is the systematic deviation of study results or inferences from the truth. Because bias can lead to erroneous conclusions, its minimization is a critical goal of all good research [1]. Bias can arise during all study phases, including its design, conduct, or analysis [2]. Several studies have found extensive amounts of inappropriate statistical methodology in published papers [3], [4], [5]. Analytical bias is important because it often can be avoided with proper statistical methodology.
In a time-to-event or survival analysis [6], problems can occur when variables in the model change value after the start of patient observation. Such variables are called “time-dependent,” because their value can change over time. There are two general categories of time-dependent variables. “Baseline measurable” time-dependent variables, like systolic blood pressure and body mass index, can change over time but are measurable at baseline. These “baseline measurable” time-dependent variables are frequently analyzed as fixed, or unchanging, variables in survival analyses. In contrast, “baseline immeasurable” time-dependent variables cannot be measured at baseline and indicate what happened to patients during observation.
Biased estimates can occur if “baseline immeasurable” time-dependent variables are analyzed as fixed variables. Consider a hypothetic study determining prognosticators for patients who have a perforation of the sigmoid and undergo emergency hemicolectomy with colostomy. Patients who die in the first several months after the operation will never undergo closure of their colostomy. If this “baseline immeasurable” time-dependent factor (“Was colostomy closed?”) is analyzed in a survival analysis as a fixed variable, one would associate no colostomy closure with a worse survival. This association is erroneous, because death results in the colostomy not getting closed, rather than vice versa.
For a real example, consider a study in which we examined the effect of discharge summary dissemination upon readmission to hospital [7]. Our “baseline-immeasurable” time-dependent variable was “Did the patient have a follow-up visit with a physician who had received the discharge summary.” When it was analyzed as a fixed variable, we found a large reduction in readmission to hospital when patients saw physicians with the summary (adjusted hazard ratio [HR] 0.35, 95% CI 0.24–0.52). This is a biased association, because patients who are readmitted to the hospital early after discharge do not have a chance to see such physicians and are placed in the “no-summary” group. This makes outcomes for the “no-summary” group particularly poor, thereby making follow-up with physicians who had the summary look beneficial by comparison. When a time-dependent analysis is used, we found a much smaller effect of follow-up with a physician who had received the summary (adjusted HR 0.74, 95% CI 0.50–1.11).
In these two examples, biased conclusions occur because patient categorization with respect to the “baseline-immeasurable” time-dependent variable was partially due to their outcome. Glesby and Hoover termed this “survivor treatment selection bias” [8]. Because this bias is not limited to treatment variables or analyses in which survival is the outcome, we will use the more generic term “time-dependent bias” in this review. Time-dependent bias can be avoided with the proper use of a time-dependent covariate analysis [8], [9], [10], [11].
There are several reasons why time-dependent bias could be common in the medical literature. The use of time-to-event analyses has increased in the medical literature over the last 20 years [12]. In our experience, time-dependent analysis of survival data is infrequently taught during research training and is often given a cursory review in some epidemiologic textbooks. In many journals, articles infrequently undergo formal statistical review prior to publication [13].
We critically reviewed the survival analyses published in prominent general medical journals to determine whether time-dependent bias due to improper analytical methodology is common. We also wanted to determine if its prevalence has recently changed, and how often study conclusions would qualitatively change if the analysis had properly accounted for the time-dependent nature of the data.
Section snippets
Study selection and inclusion criteria
We reviewed studies containing survival analyses that were published in nine medical journals including the American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal (BMJ), Circulation, Chest, Journal of the American Medical Association (JAMA), Lancet, and New England Journal of Medicine. These journals were selected because they are general medical, respiratory, and cardiovascular journals with a broad readership, large circulation, and a
Results
Our search strategy resulted in 1,184 papers (see Appendix A). Four hundred seventy-seven papers did not have a survival analysis (Fig. 1). Of the 707 survival analyses, 25 were excluded because they did not determine the association of an outcome with any patient factor.
This left 682 survival analyses that determined the association of at least one variable with an outcome (Table 1). The number of survival analyses published per year increased during the study period. The Kaplan-Meier method,
Discussion
We found that time-dependent bias due to improper analytic methodology is surprisingly common in these medical journals. When present, time-dependent bias frequently affected the strength of a purported association that was highlighted in the abstract. The prevalence of time-dependent bias appears to be decreasing. In over half of the papers susceptible to time-dependent bias, its correction could have changed the studies' conclusion.
Time-dependent bias is important for several reasons. It can
Acknowledgements
Dr. van Walraven is an Ontario Ministry of Health Career Scientist. We greatly appreciate comments from Drs. Don Redelmeier, Dean Fergusson, Paul Hebert, and David Sackett regarding previous versions of this study.
References (97)
- et al.
Fundamentals of epidemiologic research
Bias in analytic research
J Chronic Dis
(1979)The use of statistics in the British Journal of Psychiatry
Br J Psychiatry
(1995)- et al.
Review of statistics usage in the American Journal of Obstetrics and Gynecology
Am J Obstet Gynecol
(1996) - et al.
Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain
J Clin Epidemiol
(2001) - et al.
Time to event (survival) data
BMJ
(1998) - et al.
The effect of discharge summary availability during post-discharge outpatient visits on readmission to hospital
J Gen Intern Med
(2002) - et al.
Survivor treatment selection bias in observational studies: examples from the AIDS literature
Ann Intern Med
(1996) Regression models and life tables
J R Stat Soc
(1972)Estimating Cox-regression models with PROC PHREG. Survival analysis using the SAS system
(2000)