Abstract
Conflict between clinical importance and statistical significance is an important problem in medical research. Although clinical importance is best described by asking for the effect size or how much, statistical significance can only suggest whether there is any difference. One way to combine statistical significance and effect sizes is to report confidence intervals. We therefore assessed the reporting of confidence intervals in the orthopaedic literature and factors influencing this frequency. In parallel, we tested the predictive value of statistical significance for effect size. In a random sample of predetermined size, we found one in five orthopaedic articles reported confidence intervals. Participation of an individual trained in research methods increased the odds of doing so fivefold. The use of confidence intervals was independent of impact factor, year of publication, and significance of outcomes. The probability of statistically significant results to predict at least a 10% between-group difference was only 69% (95% confidence interval, 55%–83%), suggesting that a high proportion of statistically significant results do not reflect large treatment effects. Confidence intervals could help avoid such erroneous interpretation by showing the effect size explicitly.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akobeng AK. Confidence intervals and p-values in clinical decision making. Acta Paediatr. 2008;97:1004–1007.
Altman D, Bland JM. Confidence intervals illuminate absence of evidence. BMJ. 2004;328:1016–1017.
Altman DG. Why we need confidence intervals. World J Surg. 2005;29:554–556.
Altman DG, Bland JM. Diagnostic tests 2: predictive values. BMJ. 1994;309:102.
Altman DG, Machin D, Bryant TN, Gardner S. Statistics with Confidence. 2nd Ed. London, England: BMJ Books; 2000.
Bailey CS, Fisher CG, Dvorak MF. Type II error in the spine surgical literature. Spine. 2004;29:1146–1149.
Bhandari M, Montori VM, Schemitsch EH. The undue influence of significant p-values on the perceived importance of study results. Acta Orthop. 2005;76:291–295.
Bhardwaj SS, Camacho F, Derrow A, Fleischer AB Jr, Feldman SR. Statistical significance and clinical relevance: the importance of power in clinical trials in dermatology. Arch Dermatol. 2004;140:1520–1523.
Blume J, Peipert JF. What your statistician never told you about p-values. J Am Assoc Gynecol Laparosc. 2003;10:439–444.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd Ed. Hillsdale, NJ: Lawrence Earlbaum Associates; 1988.
Egol KA, Paksima N, Puopolo S, Klugman J, Hiebert R, Koval KJ. Treatment of external fixation pins about the wrist: a prospective, randomized trial. J Bone Joint Surg Am. 2006;88:349–354.
Emerson J, Burdick E, Hoaglin D, Mosteller F, Chalmers T. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Control Clin Trial. 1990;11:339–352.
Freedman KB, Back S, Bernstein J. Sample size and statistical power of randomised, controlled trials in orthopaedics. J Bone Joint Surg Br. 2001;83:397–402.
Garfield E. How can impact factors be improved? BMJ. 1996;313:411–413.
Garfield E. Impact factors, and why they won’t go away. Nature. 2001;411:522.
Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90–93.
Kocher M, Zurakowski D. Clinical Epidemiology and biostatistics: a primer for orthopaedic surgeons. J Bone Joint Surg Am. 2004;86:607–620.
Langman MJ. Towards estimation and confidence intervals. BMJ. 1986;292:716.
Lochner HV, Bhandari M, Tornetta P 3rd. Type-II error rates (beta errors) of randomized trials in orthopaedic trauma. J Bone Joint Surg Am. 2001;83:1650–1655.
Nakagawa S, Cuthill IC. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev Camb Philos Soc. 2007;82:591–605.
Rothman KJ. Writing for epidemiology. Epidemiology. 1998;9:333–337.
Sexton SA, Ferguson N, Pearce C, Ricketts DM. The misuse of ‘no significant difference’ in British orthopaedic literature. Ann R Coll Surg Engl. 2008;90:58–61.
Sierevelt IN, van Oldenrijk J, Poolman RW. Is statistical significance clinically important? A guide to judge the clinical relevance of study findings. J Long Term Eff Med Implants. 2007;17:173–179.
Thompson WD. On the comparison of effects. Am J Public Health. 1987;77:491–492.
Thompson WD. Statistical criteria in the interpretation of epidemiologic data. Am J Public Health. 1987;77:191–194.
Vavken P, Culen G, Dorotka R. Management of confounding in controlled orthopaedic trials: a cross-sectional study. Clin Orthop Relat Res. 2008;466:985–989.
Vavken P, Kotz R, Dorotka R. Minimally invasive hip replacement: a meta-analysis. Z Orthop Unfall. 2007;145:152–156.
Author information
Authors and Affiliations
Corresponding author
Additional information
Each author certifies that he or she has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
This work was performed at Children’s Hospital Boston and Medical University of Vienna.
About this article
Cite this article
Vavken, P., Heinrich, K.M., Koppelhuber, C. et al. The Use of Confidence Intervals in Reporting Orthopaedic Research Findings. Clin Orthop Relat Res 467, 3334–3339 (2009). https://doi.org/10.1007/s11999-009-0817-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11999-009-0817-7