Assessing Clinical Reasoning in Pediatric Emergency Medicine: Validity Evidence for a Script Concordance Test

doi:10.1016/j.annemergmed.2008.07.024

Annals of Emergency Medicine

Volume 53, Issue 5, May 2009, Pages 647-652

https://doi.org/10.1016/j.annemergmed.2008.07.024 Get rights and content

Study objective

Clinical reasoning is a crucial skill for all residents to acquire during their training. During most patient encounters in pediatric emergency medicine, physicians and trainees are challenged by diagnostic, investigative, and treatment uncertainties. The Script Concordance Test may provide a means to assess reasoning skills in the context of uncertainty in the practice of pediatric emergency medicine. We gathered validity evidence for the use of a pediatric emergency medicine Script Concordance Test to evaluate residents' reasoning skills.

Methods

A 1-hour test containing 60 questions nested in 38 cases was administered to 53 residents at the end of their pediatric emergency medicine rotation at 1 academic institution. Twelve experienced pediatricians were part of a reference panel to establish the basis for the scoring process.

Results

An optimized version of the test, based on positive item discrimination data, contained 30 cases and 50 questions. Scores ranged from 48% to 82%, with a mean score of 69.9 (SD=11.5). The reliability of the optimized test (Cronbach's α) was 0.77. Performance on the test increased as the level of experience of the residents increased. The residents considered the Script Concordance Test true to real-life clinical problems and had enough time to complete the test.

Conclusion

This pediatric emergency medicine Script Concordance Test was reliable and useful to assess the progression of clinical reasoning during residency training.

Introduction

Clinical reasoning is a crucial skill for all future physicians to acquire during their training. During most patient encounters in pediatric emergency medicine, physicians and trainees are challenged by diagnostic, investigative, and treatment uncertainties. The clinical supervision of medical students and residents often takes place without direct observation of the trainee's history-taking and physical examination skills. After a brief period of reflection and charting, the trainee reports the findings to an attending physician. From these brief reporting encounters, the attending physician judges the clinical competence of each trainee and reports the judgments on clinical ratings forms. These forms often represent the sole means used to assess clinical reasoning.¹ Although clinical ratings are easy to administer, unobtrusive, and low cost, this assessment system results in many “above-average” ratings, often based on subjective impressions, that do not discriminate well among trainees.²

In an effort to improve the validity of the assessment of trainees, both the Accreditation Council for Graduate Medical Education in the United States and the Royal College of Physicians and Surgeons of Canada have asked residency programs to better assess and certify residents' key competencies in becoming qualified physicians, including clinical reasoning. One of the recommendations of the Accreditation Council for Graduate Medical Education advocates the use of more than 1 assessment tool to better evaluate different aspects of competence.

An assessment tool recently developed to assess clinical reasoning, the Script Concordance Test,³ was implemented in a pediatric emergency medicine program to assess residents at the end of their rotation. The purpose of the present study was to gather validity evidence for the use of Script Concordance Tests in pediatric emergency medicine.

Section snippets

Study Design

An instrument validation study was conducted. All residents in a pediatric emergency medicine clinical rotation during a 7-month period were asked to complete a Script Concordance Test. The test was administered in a supervised setting during the last week of each 4-week pediatric emergency medicine rotation. The Script Concordance Test was completed individually during a maximum 60-minute period after standardized instructions given by a research assistant acting as supervisor. Residents were

Results

During the 7-month study period, 53 of 55 eligible residents consented to participate (96%): 38 women (72%) and 15 men (28%). Thirty-four residents were from family medicine (64%), 10 from pediatrics (19%), 3 from emergency medicine (6%), and 6 from other programs (11%; 5 from radiology and 1 from dermatology). Twenty-one residents (40%) were in their first postgraduate year of training, 21 were PGY-2s (40%), and the 11 others (20%) were in their senior years.

The optimization process of

Limitations

This study used a nonrandom group of residents, a convenience sample.¹³ Although this method is less ideal than a stratified random sample, the study setting did not allow for random sampling. However, the group of residents in this study can be considered representative of residents in academic pediatric emergency medicine services in Canada and the United States. For example, the proportion of emergency medicine residents in this study (16%) is similar to that of residents in a study reported

Discussion

The results from this study contribute positively to a growing body of literature on the Script Concordance Test approach. Previous studies addressed issues of validity, reliability, feasibility, and applicability in different clinical disciplines and contexts.3, 5, 8, 15 The development of a Script Concordance Test for pediatric emergency medicine and the positive validity evidence gathered in the present study provide additional support for the use of the Script Concordance Test.

For any

References (19)

F. Caire et al.
Auto-évaluation des internes en neurochirurgie par tests de concordance de script (TCS): processus d'élaboration des tests
Neurochirurgie
(2004)
G.W. Bandiera et al.
Predictive validity of the Global Assessment Form used in a final-year undergraduate rotation in emergency medicine
Acad Emerg Med
(2002)
J.D. Gray
Global rating scales in residency education
Acad Med
(1996)
B. Charlin
Standardized Assessment of Ill-defined Clinical Problems: the Script Concordance Test [PhD thesis]
(2002)
B. Charlin et al.
Scripts and clinical reasoning
Med Educ
(2007)
B. Charlin et al.
Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach
Eval Health Prof
(2004)
G. Page et al.
Developing key-feature problems and examinations to assess clinical decision-making skills
Acad Med
(1995)
B. Charlin et al.
The Script Concordance Test: a tool to assess the reflective clinician
Teach Learn Med
(2000)
R. Gagnon et al.
Assessment in context of uncertainty: how many members are needed on the panel of reference of a script concordance test?
Med Educ
(2005)

There are more references available in the full text version of this article.

Cited by (55)

A Video-Based Assessment Tool to Measure Intraoperative Laparoscopic Suturing Using a Modified Script Concordance Methodology
2023, Journal of Surgical Education
Laparoscopic suturing (LS) is a challenging laparoscopic skill to teach. Its complexity and nuances are not modeled or measured in current simulation and assessment platforms.The script concordance test (SCT) is used to assess clinical reasoning.The purpose of this study is to provide evidence for validity of this novel SCT based online assessment for LS skills.
We designed a video-based online SCT for LS using a cognitive task analysis and expert panelists.The CTA yielded 4 LS domains: needle handling (NH), tissue handling (TH), knot tying techniques (KT) and operative ergonomics (OE). Five-point scales with anchoring descriptors from -2 to +2 were used. Scoring was based on a modified SCT methodology.
The test was administrated to 37 subjects (18 experts and 19 novices). There was no time limit given. A different expert group from the minimal invasive surgery (MIS) panelist were recruited. Experts were defined as surgeons and fellows with LS experience of >25 cases annually. Validity was assessed by comparing SCT scores of experienced and inexperienced surgeons. Cronbach's alpha was used to assess the internal consistency of the test.
The survey started off with 47 questions in each of the following domains: 13 NH, 4 TH, 20 KT and 10 OE. Thirty-seven surgeons (18 experts and 19 inexperienced surgeons). Questions that demonstrated a large discrepancy among experts and panelists with a weighted score difference more than 40 were discarded (n = 20). One question was discarded because it received a 100% score from all participants. This yielded 26 remaining questions in the following domains: 8 NH, 2 TH, 11 KT and 5 OE. The test reliability level (Cronbach a) was 0.80. The mean score was 72 ± 9% and 63 ± 15% (p = 0.02) for experts and inexperienced surgeons, respectively. The mean time to complete the test was 21 minutes.
This study provides validity evidence for a novel intraoperative LS assessment. The variability of responses between experts and panelists suggests that SCT may capture the clinical differences/surgeon preferences in performing LS intraoperatively.
Clinical reasoning evaluation using script concordance test in primary care residents
2022, Anales de Pediatria
Actualmente desconocemos el razonamiento clínico objetivo de los residentes durante su formación, lo que impide orientar su reciclaje formativo a solventar los déficits detectados. El script concordance test (SCT) evalúa el razonamiento clínico planteando situaciones clínicas reales, pero aún no ha sido utilizado en España con contenidos propios de Atención Primaria. Por ello consideramos relevante diseñar un SCT de Atención Primaria que cumpla los criterios de validez, fiabilidad y aceptabilidad descritos en la bibliografía.
Elaboración, corrección y validación de un SCT para evaluar el razonamiento clínico en Atención Primaria de los residentes y que incluya variables sociolaborales para estudiar su posible relación con la puntuación obtenida.
Nuestro cuestionario fue aprobado por un comité de expertos, alcanzó una fiabilidad y accesibilidad adecuadas, y distinguió a los expertos de los residentes. No se observaron diferencias estadísticamente significativas en función de la edad, género, tipo y duración de la formación recibida en Atención Primaria, y la realización de un curso sobre dicha formación.
Se construyó un SCT que obtuvo la aprobación por parte de un comité de expertos, cumplió con los criterios de fiabilidad y accesibilidad, y permitió objetivar diferencias significativas en el razonamiento clínico de los expertos y residentes. Excepto en el 2.^o año de residencia, no se observaron diferencias estadísticamente significativas respecto al año formativo dentro de la residencia, la edad, el género, la realización y duración de la rotación en Atención Primaria y la realización de un curso sobre dicha formación.
Actual unawareness about paediatric resident's residency program factual clinical reasoning precludes professional retraining directed to solve deficiencies. Script concordance test (SCT) evaluates clinical reasoning due to its orientation to usual clinical practice but surprisingly it has not been used in Spain for Paediatric Primary Care clinical reasoning evaluation so far. Due to this we consider to be of relevance to design a paediatric primary care SCT which meets validity, reliability and accessibility criteria described in bibliography.
Development, validation and application of an SCT questionnaire for clinical reasoning analysis in paediatric primary care applied on a population of paediatric residents and which includes demographic and employment data in order to study possible relationship between them and achieved scores.
Our SCT was approved by an experts committee. It met reliability and accessibility criteria and it allowed distinguishing experts from paediatric internal residents. No statistically significant differences were found concerning age, gender, type and duration of the training received in Primary Care, and the completion of a course on that training.
We developed an SCT that was approved by a Paediatric Experts Committee, it met reliability and accessibility criteria and it allowed distinguishing clinical reasoning from experts and paediatric internal residents. Except second year residency program, we did not objectified relevant differences in residency program year, age, gender, duration and realization of Paediatric Primary Care rotation, and training course realisation.
Usefulness of SCT in detecting clinical reasoning deficits among pediatric professionals
2021, Progress in Pediatric Cardiology
Citation Excerpt :
So far, the SCT has been applied in many specialties of adult medicine, including Cardiology [16–18], in order to assess the condition of clinical reasoning in those professionals. However, there are still not many studies within the pediatric field [13,19–23] and there is none including contents on Pediatric Cardiology (PC), in spite of the fact that this pediatric discipline includes potentially severe pathologies and represents, according to different Spanish studies (there are no international studies which take this variable into account), about 1% of the total amount of reasons for consultation [1,24]. For this reason, we consider it is relevant to create an SCT for PC complying with the criteria of validity, reliability, and acceptability described in the bibliography, and whose specific objectives are:
Nowadays, in some Spanish autonomous communities, between 30% to 50% of the Primary CarePediatric positions are carried out by professionals without specific training in the specialty.
In addition, when Spanish general practitioners with pediatric practice complete their Resident Medical Intern program are not evaluated on their Pediatric Cardiology clinical reasoning. Actual unawareness about this knowledge precludes professional retraining directed to solve deficiencies in this collective.
Script Concordance Test is a new clinical reasoning assessment tool which places examinees in real clinical situations in which they have to interpret data and assess hypothesis to make decisions. This tool allows getting consistent data through variables analysis but it still hasn't been used in Spain for health professional's evaluation so far.
Our goal is the development, validation, and application of a Script Concordance Test for clinical reasoning analysis in Pediatric Cardiology to ensure if practitioners attending children have enough Pediatric Cardiology knowledge for safe practice and to learn if qualification, Pediatric Cardiology formation, years of pediatric practice, and training course completion influence the score obtained.
Our method describes a Script Concordance Test which was approved by a Pediatric Cardiology experts committee, met reliability criteria, allowed distinguishing evaluated subject different formative grade (188 pediatricians and 38 general practitioners), and was accessible for investigators. A statistical study including demographic and employment variables and achieved scores was performed to study a possible relationship between them.
4 pediatricians (2.1%) and 1 GP (2.6%) didn't pass the questionnaire. Pediatric Cardiology qualification and training programs influenced more in Pediatric Cardiology clinical reasoning than the accumulation of years of pediatric practice and the frequency of this practice. We observed that the older the professional was, the lowest the score in the questionnaire.
Pediatric Hospitalists’ Performance and Perceptions of Script Concordance Testing for Self-Assessment
2021, Academic Pediatrics
Citation Excerpt :
Many previous studies of SCT focus on the test's validity and reliability among different specialties3–7 and across the spectrum of learners from medical students20 to postgraduate trainees18 and physicians in practice.9 While studies have concluded that the SCT format is both reliable and valid,4–8 recent studies have questioned the test's psychometric properties.32,33 Lubarsky34 and others have raised concerns that test takers can “game” the test by avoiding answers at either extreme of the Likert scale.
The cognitive expertise of Pediatric Hospitalists (PH) lies not in standard knowledge but in making decisions under conditions of uncertainty. To maintain expertise, PH should engage in deliberate practice via self-assessments that promote higher-level cognitive processes necessary to address problems with missing or ambiguous information. Higher levels of cognition are purported with Script Concordance Test (SCT) questions compared to Multiple Choice Questions (MCQ). To determine if PH use higher levels of cognition when answering SCT versus MCQ questions and to analyze participants’ perceptions of the utility of using SCT self-assessment for deliberate practice in addressing clinical problems encountered in daily practice.
This is a mixed methods study comparing the cognitive level expressed according to Bloom’s Taxonomy by PH answering MCQ versus SCT questions using a “think aloud” (TA) exercise, followed by qualitative analysis of interviews conducted afterward.
A significantly greater percentage of comments were coded as higher cognitive processes (apply, analyze, evaluate, and create) for SCT versus MCQ (74% vs 19%) compared with lower order (remember, understand); chi-square P < .00001. Analysis of interviews revealed 6 themes.
SCT questions elicited higher level cognition essential to clinical reasoning compared to MCQ questions. PH-indicated MCQ questions measure standard knowledge, while SCT questions better measure decision-making under conditions of uncertainty. PH-perceived SCT could be useful for deliberate practice in Pediatric Hospital Medicine decision-making if they could compare their rationale in answering questions with that of experts.
The Script Concordance Test: A multimodal teaching tool
2018, Revue de Medecine Interne
Durant ces dernières années, un intérêt majeur a été porté à l’évaluation des apprenants lors de leur éducation médicale. Le noyau de la compétence médicale est le raisonnement clinique qui devrait être acquis par tous les acteurs en médecine. Son évaluation comprend la capacité d’intégrer et d’appliquer différents types de connaissances, de peser les preuves de manière critique et de réfléchir au processus utilisé pour arriver à un diagnostic. Parmi les différentes méthodes d’évaluation, le test de concordance de script, spécifiquement construit pour un usage éducationnel médical, évalue le raisonnement clinique en contexte d’incertitude. Les facultés de médecine françaises devront l’inclure dans la formation et l’évaluation des étudiants courant 2017. Il devrait donc faire partie des nouvelles modalités docimologiques des Épreuves classantes nationales informatisées à partir de 2020. Dans cet article, nous nous proposons de faire une revue spécifique de la littérature concernant les bases théoriques, la méthodologie et l’utilisation des TCS dans le champ de la santé. Pour se faire, nous avons utilisé les bases de données PubMed, EMBASE et PsycINFO et avons sélectionné les articles en langues anglaise et française en utilisant seul ou en combinaison les mots clés suivants : script, concordance, test de concordance de script, études de médecine, validité, fidélité, propriétés psychométriques. Nous avons identifié 62 articles qui répondaient à notre recherche. Construit sur une méthodologie bien ciblée, le TCS est un outil pédagogique reproductible, fidèle et valide. Il permet de discriminer les niveaux de pratique entre externes, internes et seniors en médecine, d’évaluer la progression et les compétences du raisonnement clinique. Il peut être administré en ligne, en multicentrique, à un niveau national ou international. Il existe cependant des limites que les enseignants doivent prendre en compte.
In recent years, there has been a major interest in assessing learners during their medical education. The core of medical competence is the clinical reasoning that should be acquired by all medical actors. Its assessment includes the ability to integrate and apply different types of knowledge, weigh critical evidence and think about the process used to have a diagnosis. French faculties of medicine will have to include it in the training and evaluation of students during 2017. It should therefore be part of the new docimological modalities of the computerized “Épreuves Classantes Nationales” from 2020 onwards. In this article, we did a specific review of the literature concerning the theoretical foundations, methodology and use of Script Concordance Test (SCT) in the field of health. To do so, we used the PubMed, EMBASE and PsycINFO databases and selected articles in English and French language using the following keywords alone or in combination: script, concordance, script concordance test, medical studies, validity, fidelity, psychometric properties. We have found 62 articles that matched our search. Built on a well-targeted methodology, the SCT is a reliable and valid pedagogical tool. It can discriminate the levels of practice between medical students, residents and medical doctors, evaluate the progression and skills of clinical reasoning. It can be administered online, in multicentric centers, at a national or international level. There are, however, limitations that teachers must take into account.
Novel Transfer of Care Sign-out Assessment Tool in a Pediatric Emergency Department
2018, Academic Pediatrics
Citation Excerpt :
Senior resident was defined as PGY2 or higher. PGY status is frequently used as a proxy for maturity in educational studies evaluating progression of skill and knowledge.24–26 Finally, group comparisons took into consideration the variable of continuity of care.
Transfer of care sign-outs (TOCS) for admissions from a pediatric emergency department have unique challenges. Standardized and reliable assessment tools for TOCS remain elusive. We describe the development, reliability, and validity of a TOCS assessment tool.
Video recordings of resident TOCS were assessed to capture 4 domains: completeness, synopsis, foresight, and professionalism. In phase 1, 56 TOCS were used to modify the tool and improve reliability. In phase 2, 91 TOCS were used to examine validity. Analyses included Cronbach's alpha for internal structure, intraclass correlation and Cohen's kappa for interrater reliability, Pearson's correlation for relationships between variables, and 95% confidence interval of the mean for resident group comparisons.
Cronbach's alpha was 0.52 for internal structure of the tool's subjective rating scale. Intraclass correlation for the subjective rating scale items ranged from 0.70 to 0.80. Cohen's kappa for most objective checklist items ranged from 0.43 to 1. Content completeness was significantly correlated with synopsis, foresight, and professionalism (Pearson's r ranged from 0.36 to 0.62, P values were <0.001). House staff senior residents scored higher (on average) than interns and rotating senior residents in synopsis and foresight. Also, house staff interns scored higher (on average) than rotating senior residents in professionalism. House staff senior residents scored higher (on average) than rotating senior residents in content completeness.
We provide validity evidence to support using scores from the TOCS tool to assess higher-level transfer of care comprehension and communication by pediatric emergency department residents and to test interventions to improve TOCS.

View all citing articles on Scopus

: Provide feedback on this article at the journal's Web site, www.annemergmed.com.

: Supervising editor: Peter C. Wyer, MD

: Author contributions: B Carrière, RG, B Charlin, and GB designed the study. B Carrière and B Charlin supervised the data collection during the study period. RG and SD provided statistical advice, and RG analyzed the data. B Carrière wrote the article, with major contributions from B Charlin, SD, and GB for revision and content in the discussion. B Carrière takes responsibility for the paper as a whole.

: Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article that might create any potential conflict of interest. The authors have stated that no such relationships exist. See the Manuscript Submission Agreement in this issue for examples of specific conflicts covered by this statement.

: Publication date: Available online August 22, 2008.

: Reprints not available from the authors.

View full text

Article preview

Annals of Emergency Medicine

Study objective

Methods

Results

Conclusion

Introduction

Section snippets

Study Design

Results

Limitations

Discussion

References (19)

Auto-évaluation des internes en neurochirurgie par tests de concordance de script (TCS): processus d'élaboration des tests

Neurochirurgie

Predictive validity of the Global Assessment Form used in a final-year undergraduate rotation in emergency medicine

Acad Emerg Med

Global rating scales in residency education

Acad Med

Standardized Assessment of Ill-defined Clinical Problems: the Script Concordance Test [PhD thesis]

Scripts and clinical reasoning

Med Educ

Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach

Eval Health Prof

Developing key-feature problems and examinations to assess clinical decision-making skills

Acad Med

The Script Concordance Test: a tool to assess the reflective clinician

Teach Learn Med

Assessment in context of uncertainty: how many members are needed on the panel of reference of a script concordance test?

Med Educ

Cited by (55)

A Video-Based Assessment Tool to Measure Intraoperative Laparoscopic Suturing Using a Modified Script Concordance Methodology

Clinical reasoning evaluation using script concordance test in primary care residents

Usefulness of SCT in detecting clinical reasoning deficits among pediatric professionals

Pediatric Hospitalists’ Performance and Perceptions of Script Concordance Testing for Self-Assessment

The Script Concordance Test: A multimodal teaching tool

Novel Transfer of Care Sign-out Assessment Tool in a Pediatric Emergency Department

Education/original researchAssessing Clinical Reasoning in Pediatric Emergency Medicine: Validity Evidence for a Script Concordance Test

Study objective

Methods

Results

Conclusion

Introduction

Section snippets

Study Design

Results

Limitations

Discussion

Neurochirurgie

Predictive validity of the Global Assessment Form used in a final-year undergraduate rotation in emergency medicine

Acad Emerg Med

Global rating scales in residency education

Acad Med

Standardized Assessment of Ill-defined Clinical Problems: the Script Concordance Test [PhD thesis]

Scripts and clinical reasoning

Med Educ

Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach

Eval Health Prof

Developing key-feature problems and examinations to assess clinical decision-making skills

Acad Med

The Script Concordance Test: a tool to assess the reflective clinician

Teach Learn Med

Assessment in context of uncertainty: how many members are needed on the panel of reference of a script concordance test?

Med Educ

Education/original research
Assessing Clinical Reasoning in Pediatric Emergency Medicine: Validity Evidence for a Script Concordance Test