Abstract
The ECFMG® Clinical Skills Assessment (CSA®)was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN rating scan be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.
Similar content being viewed by others
References
American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing. Washington, DC: Author.
Australian Council for Educational Research (1998). ACER ConQuest: Generalized Item Response Modelling Software. Melbourne, Australia: Author.
Ayers, W.R. & Boulet, J.R. (2001). Establishing the validity of test score inferences: Performance of 4th-year U.S. medical students on the ECFMG Clinical Skills Assessment. Teaching and Learning in Medicine 13(4): 214-220.
Boulet, J.R., Ben David, M.F., Ziv, A., Burdick,W.P., Curtis,M. & Peitzman, S. et al. (1998a). Using standardized patients to assess the interpersonal skills of physicians. Academic Medicine 73(10 Suppl.): 94-96.
Boulet, J.R., Friedman Ben-David, M., Hambleton, R.K., Burdick, W.P., Ziv, A. & Gary, N.E. (1998b). An investigation of the sources of measurement error in the post-encounter written scores from standardized patient examinations. Advances in Health Sciences Education: Theory and Practice 3: 89-100.
Boulet, J.R., Friedman Ben-David, M., Ziv, A., Burdick, W.P. & Gary, N.E. (2000). The use of holistic scoring for post-encounter written exercises. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment (pp. 254-260). Philadelphia, PA: National Board of Medical Examiners.
Boulet, J.R., McKinley, D.W., Norcini, J.J. & Whelan, G.P. (2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice 7: 85-97.
Boulet, J.R., van Zanten, M., McKinley, D.W. & Gary, N.E. (2001). Evaluating the spoken English proficiency of graduates of foreign medical schools. Medical Education 35(8): 767-773.
Brennan, R.L. (2001). Generalizability Theory. New York: Springer-Verlag.
Cradock, J., Young, A.S. & Sullivan, G. (2001). The accuracy of medical record documentation in schizophrenia. Journal of Behavioral Health Services & Research 28(4): 456-465.
Crossley, G.M., Howe, A., Newble, D., Jolly, B. & Davies, H.A. (2001). Sheffield Assessment Instrument for Letters (SAIL): Performance assessment using outpatient letters. Medical Education 35(12): 1115-1124.
Friedman Ben-David, M.F., Boulet, J.R., Burdick, W.P., Ziv, A., Hambleton, R.K. & Gary, N.E. (1997). Issues of validity and reliability concerning who scores the post-encounter patientprogress note. Academic Medicine 72(10 Suppl. 1): 79-81.
Friedman Ben-David, M.F., Klass, D.J., Boulet, J., De Champlain, A., King, A.M. & Pohl, H.S. et al. (1999). The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: A collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG). Medical Education 33(6): 439-446.
Grace-Farfaglia, P. & Rosow, P. (1995). Automating clinical dietetics documentation Journal of the American Dietetic Association 95(6): 687-690.
Hallock, J.A. ECFMG and the Challenges Facing International Medical Graduates (2002). Reporter 11(8): 2-3. Washington, DC: Association of American Medical Colleges.
Harden, R.M. & Gleeson, F.A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education 13(1): 41-54.
Howell, J., Chisholm, C., Clark, A. & Spillane, L. (2000). Emergency medicine resident documentation: Results of the 1999 American Board of Emergency Medicine in-training examination survey. Academic Emergency Medicine 7(10): 1135-1138.
Larimore, W.L. & Jordan, E.V. (1995). SOAP to SNOCAMP: improving the medical record format. Journal of Family Practice 41(4): 393-398.
Medical Council of Canada (2002). Qualifying Examination Part II, Information Pamphlet. Ottawa, Canada: Author.
Prentice, D.A. & Miller, D.T. (1998). When small effects are impressive. In A.E. Kazdin (ed.), Methodological Issues and Strategies in Clinical Research (pp. 163-173). Washington, DC: American Psychological Association.
Regehr, G., MacRae, H., Reznick, R.K. & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine 73(9): 993-997.
Rollnick, S., Kinnersley, P. & Butler, C. (2002). Context-bound communication skills training: Development of a new method. Medical Education 36(4): 377-383.
SAS Institute, Inc. (1989). SAS/STAT User's Guide: Version 6 (4th ed.), Cary, NC: Author.
Shapiro, J. (1999). Correlates of family-oriented physician communications. Family Practice 16(3): 294-300.
Slater, S.C. & Boulet, J.R. (2001). Predicting holistic ratings of written performance assessments from analytic scoring. Advances in Health Sciences Education: Theory and Practice 6(2): 103-119.
Sleszynski, S.L., Glonek, T. & Kuchera, W.A. (1999). Standardized medical record: A new outpatient osteopathic SOAP note form: Validation of a standardized office form against physician's progress notes. Journal of the American Osteopathic Association 99(10): 516-529.
Swanson, D.B. & Norcini, J.J. (1989). Factors influencing reproducibility of tests using standardized patients. Teaching and Learning in Medicine 1(3): 158-166.
van Dalen, J., Kerkhofs, E., Verwijnen, G.M., Knippenberg-Van Den Berg, B.W., van Den Hout, H.A., Scherpbier, A.J. et al. (2002). Predicting communication skills with a paper-and-pencil test. Medical Education 36(2): 148-153.
van der Vleuten, C.P., Norman, G.R. & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: Issues of reliability. Medical Education 25(2): 110-118.
Weber, D.O. (2002). Charting a course toward legible medical records: Perfect paperwork can mean financial savings, better patient care. Physician Executive 28(1): 8-13.
Williams, S., Weinman, J. & Dale, J. (1998). Doctor-patient communication and patient satisfaction: A review. Family Practice 15(5): 480-492.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Boulet, J.R., Rebbecchi, T.A., Denton, E.C. et al. Assessing the Written Communication Skills of Medical School Graduates. Adv Health Sci Educ Theory Pract 9, 47–60 (2004). https://doi.org/10.1023/B:AHSE.0000012216.39378.15
Issue Date:
DOI: https://doi.org/10.1023/B:AHSE.0000012216.39378.15