Original Article
Time-dependent bias was common in survival analyses published in leading clinical journals

https://doi.org/10.1016/j.jclinepi.2003.12.008Get rights and content

Abstract

Objective

In survival analysis, “baseline immeasurable” time-dependent factors cannot be recorded at baseline, and change value after patient observation starts. Time-dependent bias can occur if such variables are not analyzed appropriately. This study sought to determine the prevalence of such time-dependent bias in highly-cited medical journals.

Study design and setting

We searched Medline databases to identify all observational studies that used a survival analysis in American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal, Chest, Circulation, Journal of the American Medical Association, Lancet, and New England Journal of Medicine between 1998 and 2002. Studies with “baseline immeasurable” time-dependent factors were susceptible to time-dependent bias if a time-dependent covariate analysis was not used.

Results

Of 682 eligible studies, 127 (18.6%, 95% CI 15.8–21.8%) contained a “baseline immeasurable” time-dependent factor and 52 (7.6% [5.8–9.9%] of all survival analyses/40.9% [32.3–50.0%] of studies with a time-dependent factor) were susceptible to time-dependent bias. In 35 studies (5.1% [3.7–7.1%]/27.6% [20.5–35.9%]), the bias affected a variable highlighted in the study abstract and correction of the bias could have qualitatively changed the study's conclusion in over half of studies.

Conclusion

In medical journals, time-dependent bias is concerningly common and frequently affects key factors and the study's conclusion.

Introduction

Bias is the systematic deviation of study results or inferences from the truth. Because bias can lead to erroneous conclusions, its minimization is a critical goal of all good research [1]. Bias can arise during all study phases, including its design, conduct, or analysis [2]. Several studies have found extensive amounts of inappropriate statistical methodology in published papers [3], [4], [5]. Analytical bias is important because it often can be avoided with proper statistical methodology.

In a time-to-event or survival analysis [6], problems can occur when variables in the model change value after the start of patient observation. Such variables are called “time-dependent,” because their value can change over time. There are two general categories of time-dependent variables. “Baseline measurable” time-dependent variables, like systolic blood pressure and body mass index, can change over time but are measurable at baseline. These “baseline measurable” time-dependent variables are frequently analyzed as fixed, or unchanging, variables in survival analyses. In contrast, “baseline immeasurable” time-dependent variables cannot be measured at baseline and indicate what happened to patients during observation.

Biased estimates can occur if “baseline immeasurable” time-dependent variables are analyzed as fixed variables. Consider a hypothetic study determining prognosticators for patients who have a perforation of the sigmoid and undergo emergency hemicolectomy with colostomy. Patients who die in the first several months after the operation will never undergo closure of their colostomy. If this “baseline immeasurable” time-dependent factor (“Was colostomy closed?”) is analyzed in a survival analysis as a fixed variable, one would associate no colostomy closure with a worse survival. This association is erroneous, because death results in the colostomy not getting closed, rather than vice versa.

For a real example, consider a study in which we examined the effect of discharge summary dissemination upon readmission to hospital [7]. Our “baseline-immeasurable” time-dependent variable was “Did the patient have a follow-up visit with a physician who had received the discharge summary.” When it was analyzed as a fixed variable, we found a large reduction in readmission to hospital when patients saw physicians with the summary (adjusted hazard ratio [HR] 0.35, 95% CI 0.24–0.52). This is a biased association, because patients who are readmitted to the hospital early after discharge do not have a chance to see such physicians and are placed in the “no-summary” group. This makes outcomes for the “no-summary” group particularly poor, thereby making follow-up with physicians who had the summary look beneficial by comparison. When a time-dependent analysis is used, we found a much smaller effect of follow-up with a physician who had received the summary (adjusted HR 0.74, 95% CI 0.50–1.11).

In these two examples, biased conclusions occur because patient categorization with respect to the “baseline-immeasurable” time-dependent variable was partially due to their outcome. Glesby and Hoover termed this “survivor treatment selection bias” [8]. Because this bias is not limited to treatment variables or analyses in which survival is the outcome, we will use the more generic term “time-dependent bias” in this review. Time-dependent bias can be avoided with the proper use of a time-dependent covariate analysis [8], [9], [10], [11].

There are several reasons why time-dependent bias could be common in the medical literature. The use of time-to-event analyses has increased in the medical literature over the last 20 years [12]. In our experience, time-dependent analysis of survival data is infrequently taught during research training and is often given a cursory review in some epidemiologic textbooks. In many journals, articles infrequently undergo formal statistical review prior to publication [13].

We critically reviewed the survival analyses published in prominent general medical journals to determine whether time-dependent bias due to improper analytical methodology is common. We also wanted to determine if its prevalence has recently changed, and how often study conclusions would qualitatively change if the analysis had properly accounted for the time-dependent nature of the data.

Section snippets

Study selection and inclusion criteria

We reviewed studies containing survival analyses that were published in nine medical journals including the American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal (BMJ), Circulation, Chest, Journal of the American Medical Association (JAMA), Lancet, and New England Journal of Medicine. These journals were selected because they are general medical, respiratory, and cardiovascular journals with a broad readership, large circulation, and a

Results

Our search strategy resulted in 1,184 papers (see Appendix A). Four hundred seventy-seven papers did not have a survival analysis (Fig. 1). Of the 707 survival analyses, 25 were excluded because they did not determine the association of an outcome with any patient factor.

This left 682 survival analyses that determined the association of at least one variable with an outcome (Table 1). The number of survival analyses published per year increased during the study period. The Kaplan-Meier method,

Discussion

We found that time-dependent bias due to improper analytic methodology is surprisingly common in these medical journals. When present, time-dependent bias frequently affected the strength of a purported association that was highlighted in the abstract. The prevalence of time-dependent bias appears to be decreasing. In over half of the papers susceptible to time-dependent bias, its correction could have changed the studies' conclusion.

Time-dependent bias is important for several reasons. It can

Acknowledgements

Dr. van Walraven is an Ontario Ministry of Health Career Scientist. We greatly appreciate comments from Drs. Don Redelmeier, Dean Fergusson, Paul Hebert, and David Sackett regarding previous versions of this study.

References (97)

  • D.G. Kleinbaum et al.

    Fundamentals of epidemiologic research

  • D.L. Sackett

    Bias in analytic research

    J Chronic Dis

    (1979)
  • S.M. McGuigan

    The use of statistics in the British Journal of Psychiatry

    Br J Psychiatry

    (1995)
  • G.E. Welch et al.

    Review of statistics usage in the American Journal of Obstetrics and Gynecology

    Am J Obstet Gynecol

    (1996)
  • S.C. Bagley et al.

    Logistic regression in the medical literature: standards for use and reporting, with particular attention to one medical domain

    J Clin Epidemiol

    (2001)
  • D.G. Altman et al.

    Time to event (survival) data

    BMJ

    (1998)
  • C. van Walraven et al.

    The effect of discharge summary availability during post-discharge outpatient visits on readmission to hospital

    J Gen Intern Med

    (2002)
  • M.J. Glesby et al.

    Survivor treatment selection bias in observational studies: examples from the AIDS literature

    Ann Intern Med

    (1996)
  • D.R. Cox

    Regression models and life tables

    J R Stat Soc

    (1972)
  • P.D. Allison

    Estimating Cox-regression models with PROC PHREG. Survival analysis using the SAS system

    (2000)
  • L.D. Fisher et al.

    Time-dependent covariates in the Cox proportional-hazards regression model

    Annu Rev Public Health

    (1999)
  • T.R. Fleming et al.

    Survival analysis in clinical trials: past developments and future directions

    Biometrics

    (2000)
  • S.N. Goodman et al.

    Statistical reviewing policies of medical journals: caveat lector?

    J Gen Intern Med

    (1998)
  • D.V. Cicchetti et al.

    High agreement but low kappa:II. Resolving the paradoxes

    J Clin Epidemiol

    (1990)
  • K.R. Flaherty et al.

    Steroids in idiopathic pulmonary fibrosis: a prospective assessment of adverse reactions, response to therapy, and survival

    Am J Med

    (2001)
  • J.D. Christie et al.

    Primary graft failure following lung transplantation

    Chest

    (1998)
  • H. Yoshida et al.

    Interferon therapy reduces the risk for hepatocellular carcinoma: national surveillance program of cirrhotic and noncirrhotic patients with chronic hepatitis C in Japan. IHIT Study Group. Inhibition of Hepatocarcinogenesis by Interferon Therapy

    Ann Intern Med

    (1999)
  • A.P. Guerin et al.

    Impact of aortic stiffness attenuation on survival of patients in end-stage renal failure

    Circulation

    (2001)
  • I. Yoshino et al.

    Unfavorable prognosis of patients with stage II non-small cell lung cancer associated with macroscopic nodal metastases

    Chest

    (1999)
  • A. Kawai et al.

    SYT-SSX gene fusion as a determinant of morphology and prognosis in synovial sarcoma

    N Engl J Med

    (1998)
  • H. Elsaleh et al.

    Association of tumour site and sex with survival benefit from adjuvant chemotherapy in colorectal cancer

    Lancet

    (2000)
  • P.C. Wang et al.

    Prognostic role of pericardial fluid cytology in cardiac tamponade associated with non-small cell lung cancer

    Chest

    (2000)
  • P. Ang et al.

    Primary intrathoracic malignant effusion: a descriptive study

    Chest

    (2001)
  • T. Taylor et al.

    Outcome after treatment of high-risk papillary and non-Hurthle-cell follicular thyroid carcinoma

    Ann Intern Med

    (1998)
  • W. Zhou et al.

    Counting alleles to predict recurrence of early-stage colorectal cancers

    Lancet

    (2002)
  • A.G. Jensen et al.

    Risk factors for hospital-acquired Staphylococcus aureus bacteremia

    Arch Intern Med

    (1999)
  • A. Coutsoudis et al.

    Influence of infant-feeding patterns on early mother-to-child transmission of HIV-1 in Durban, South Africa: a prospective cohort study. South African Vitamin A Study Group

    Lancet

    (1999)
  • J.B. Cologne et al.

    Longevity of atomic-bomb survivors

    Lancet

    (2000)
  • L.M. Brumble et al.

    Prevention of cytomegalovirus infection and disease after lung transplantation : results using a unique regimen employing delayed ganciclovir

    Chest

    (2002)
  • R. Childs et al.

    Regression of metastatic renal–cell carcinoma after nonmyeloablative allogeneic peripheral-blood stem-cell transplantation

    N Engl J Med

    (2000)
  • C.R. Pound et al.

    Natural history of progression after PSA elevation following radical prostatectomy

    JAMA

    (1999)
  • D.A. Redelmeier et al.

    Survival in Academy Award-winning actors and actresses

    Ann Intern Med

    (2001)
  • H.R. Valantine

    Metabolic abnormalities characteristic of dysmetabolic syndrome predict the development of transplant coronary artery disease: a prospective study

    Circulation

    (2001)
  • H.M. Connolly et al.

    Severe aortic stenosis with low transvalvular gradient and severe left ventricular dysfunction:result of aortic valve replacement in 52 patients

    Circulation

    (2000)
  • P.A. Rowlings et al.

    Factors correlated with progression-free survival after high-dose chemotherapy and hematopoietic stem cell transplantation for metastatic breast cancer

    JAMA

    (1999)
  • V. Miller et al.

    Relations among CD4 lymphocyte count nadir, antiretroviral therapy, and HIV-1 disease progression: results from the EuroSIDA study

    Ann Intern Med

    (1999)
  • B. Ledergerber et al.

    AIDS-related opportunistic illnesses occurring after initiation of potent antiretroviral therapy: the Swiss HIV Cohort Study

    JAMA

    (1999)
  • K. Nademanee et al.

    Treating electrical storm: sympathetic blockade versus advanced cardiac life support-guided therapy

    Circulation

    (2000)
  • V. Rocha et al.

    Graft-versus-host disease in children who have received a cord-blood or bone marrow transplant from an HLA-identical sibling. Eurocord and International Bone Marrow Transplant Registry Working Committee on Alternative Donor and Stem Cell Sources

    N Engl J Med

    (2000)
  • I. Kristensen et al.

    Routine vaccinations and child survival: follow up study in Guinea-Bissaum West Africa

    BMJ

    (2000)
  • S. Hariharan et al.

    Improved graft survival after renal transplantation in the United States, 1988 to 1996

    N Engl J Med

    (2000)
  • M. Brenner et al.

    Survival following bilateral staple lung volume reduction surgery for emphysema

    Chest

    (1999)
  • T. Rokkas et al.

    The association between CagA status and the development of esophagitis after the eradication of Helicobacter pylori

    Am J Med

    (2001)
  • I. Olivotto et al.

    Impact of atrial fibrillation on the clinical course of hypertrophic cardiomyopathy

    Circulation

    (2001)
  • R.W. Morse et al.

    Rest-redistribution 201-Tl single-photon emission CT imaging for determination of myocardial viability: relationship among viability, mode of therapy, and long-term prognosis

    Chest

    (1999)
  • D.M. Forrest et al.

    Determinants of short- and long-term outcome in patients with respiratory failure caused by AIDS-related Pneumocystis carinii pneumonia

    Arch Intern Med

    (1999)
  • A. Pasquet et al.

    Prognostic value of myocardial ischemia and viability in patients with chronic left ventricular ischemic dysfunction

    Circulation

    (1999)
  • A.N. Cheema et al.

    Nonsustained ventricular tachycardia in the setting of acute myocardial infarction: tachycardia characteristics and their prognostic implications

    Circulation

    (1998)
  • Cited by (0)

    View full text