Abstract
OBJECTIVE: This study explores the alignment between physicians’ confidence in their diagnoses and the “correctness” of these diagnoses, as a function of clinical experience, and whether subjects were prone to over-or underconfidence.
DESIGN: Prospective, counterbalanced experimental design.
SETTING: Laboratory study conducted under controlled conditions at three academic medical centers.
PARTICIPANTS: Seventy-two senior medical students, 72 senior medical residents, and 72 faculty internists.
INTERVENTION: We created highly detailed, 2-to 4-page synopses of 36 diagnostically challenging medical cases, each with a definitive correct diagnosis. Subjects generated a differential diagnosis for each of 9 assigned cases, and indicated their level of confidence in each diagnosis.
MEASUREMENTS AND MAIN RESULTS: A differential was considered “correct” if the clinically true diagnosis was listed in that subject’s hypothesis list. To assess confidence, subjects rated the likelihood that they would, at the time they generated the differential, seek assistance in reaching a diagnosis. Subjects’ confidence and correctness were “mildly” aligned (k=.314 for all subjects, .285 for faculty, .227 for residents, and .349 for students). Residents were overconfident in 41% of cases where their confidence and correctness were not aligned, whereas faculty were overconfident in 36% of such cases and students in 25%.
CONCLUSIONS: Even experienced clinicians may be unaware of the correctness of their diagnoses at the time they make them. Medical decision support systems, and other interventions designed to reduce medical errors, cannot rely exclusively on clinicians’ perceptions of their needs for such support.
Similar content being viewed by others
References
Hersh WR. “A world of knowledge at your fingertips”: the promise, reality, and future directions of online information retrieval. Acad Med. 1999;74:240–3.
Kohn LT, Corrigan JM, Donaldson MS, eds. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000.
Bates DW, Gawande AA. Error in medicine: what have we learned? Ann Intern Med. 2000;132:763–7.
Leape LL, Bates DW, Cullen DJ, et al. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA. 1995;274:35–43.
Wyatt JC. Clinical data systems, part 3: development and evaluation. Lancet. 1994;344:1682–7.
Norman DA. Melding mind and machine. Technol Rev. 1997;100:29–31.
Chueh H, Barnett GO. “Just in time” clinical information. Acad Med. 1997;72:512–7.
Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. JAMA. 1998;280:1339–46.
Miller RA. Medical diagnostic decision support systems—past, present, and future. J Am Med Inform Assoc. 1994;1:8–27.
Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Eng J Med. 1998;338:232–8.
McDonald CJ, Overhage JM, Tierney WM, et al. The Regenstrief Medical Record System: a quarter century experience. Int J Med Inform. 1999;54:225–53.
Wagner MM, Pankaskie M, Hogan W, et al. Clinical event monitoring at the University of Pittsburgh. Proc AMIA Annu Fall Symp. 1997;188–92.
Cimino JJ, Elhanan G, Zeng Q. Supporting infobuttons with terminological knowledge. Proc AMIA Annu Fall Symp. 1997;528–32.
Miller PL. Building an expert critiquing system: ESSENTIAL-ATTENDING. Methods Inf Med. 1986;25:71–8.
Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science. 1974;185:1124–31.
Lichtenstein S, Fischhoff B. Do those who know more also know more about how much they know? Organ Behav Hum Perform. 1977;20:159–83.
Christensen-Szalanski JJ, Bushyhead JB. Physicians’ use of probabilistic information in a real clinical setting. J Exp Psychol. 1981;7:928–35.
Tierney WM, Fitzgerald J, McHenry R, et al. Physicians’ estimates of the probability of myocardial infarction in emergency room patients with chest pain. Med Decis Making. 1986;6:12–7.
Friedman CP, Elstein AS, Wolf FM, et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA. 1999;282:1851–6.
Mann D. The relationship between diagnostic accuracy and confidence in medical students. Presented at the annual meeting of the American Educational Research Association, Atlanta, 1993.
Friedman C, Gatti G, Elstein A, Franz T, Murphy G, Wolf F. Are Clinicians Correct When They Believe They Are Correct? Implications for Medical Decision Support. Proceedings of the Tenth World Congress on Medical Informatics. London; 2000. Medinfo. 2001;10(PP. 1):454–8.
Swets JA, Pickett RM. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. New York, NY: Academic Press; 1982.
McCullagh P, Nelder JA. Generalized Linear Models. 2nd ed. New York, NY: Chapman and Hall; 1991.
Liang K, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73:13–22.
Neter J, Kutner MH, Nachstsheim CJ, Wasserman W. Applied Linear Regression Models. Chicago, IL: Irwin; 1996.
SAS Institute Inc. SAS/STAT User’s Guide, Version 8. Cary, NC: SAS Institute Inc.; 1999.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.
Author information
Authors and Affiliations
Corresponding author
Additional information
The authors have no conflicts of interest to report.
This work was supported by grant R01-LM-05630 from the National Library of Medicine.
Rights and permissions
About this article
Cite this article
Friedman, C.P., Gatti, G.G., Franz, T.M. et al. Do physicians know when their diagnoses are correct?. J GEN INTERN MED 20, 334–339 (2005). https://doi.org/10.1111/j.1525-1497.2005.30145.x
Accepted:
Issue Date:
DOI: https://doi.org/10.1111/j.1525-1497.2005.30145.x