ReviewCurrent Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application
Section snippets
Validity, constructs, and meaningful interpretation of instrument scores
Validity refers to “the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of tests.”19 In other words, validity describes how well one can legitimately trust the results of a test as interpreted for a specific purpose.
Many instruments measure a physical quantity such as height, blood pressure, or serum sodium. Interpreting the meaning of such results is straightforward.20 In contrast, results from assessments of patient symptoms,
Reliability: Necessary, but not Sufficient, for valid inferences
Reliability refers to the reproducibility or consistency of scores from one assessment to another.19 Reliability is a necessary, but not sufficient, component of validity.21, 29 An instrument that does not yield reliable scores does not permit valid interpretations. Imagine obtaining blood pressure readings of 185/100 mm Hg, 80/40 mm Hg, and 140/70 mm Hg in 3 consecutive measurements over a 3-minute period in an otherwise stable patient. How would we interpret these results? Given the wide
Practical application of validity concepts in selecting an instrument
Consumers of previously developed psychometric instruments in clinical practice, research, or education need to carefully weigh the evidence supporting the validity of the interpretations they are trying to make. Scores from a popular instrument may not have evidence to justify their use. Many authors cite evidence from only one or two sources, such as reliability or correlation with another instrument’s scores, to support the validity of interpretations. Such instruments should be used with
Practical application of validity concepts in developing an instrument
When developing psychometric instruments, careful attention should again be given to each category of validity evidence in turn. To illustrate the application of these principles, we will discuss how evidence could be planned, collected, and documented when developing an assessment of clinical performance for internal medicine residents.
The first step in developing any instrument is to identify the construct and corresponding content. In our example we could look at residency program objectives
Conclusion
A clear understanding of validity and reliability in psychometric assessment is essential for practitioners in diverse medical settings. As Foster and Cone note, “Science rests on the adequacy of its measurement. Poor measures provide a weak foundation for research and clinical endeavors.”18 Validity concerns the degree to which scores reflect the intended underlying construct, and refers to the interpretation of results rather than the instrument itself. It is best viewed as a carefully
Acknowledgments
We thank Steven M. Downing, PhD (University of Illinois at Chicago, Department of Medical Education), for his insights and constructive critique.
References (65)
- et al.
The American Urological Association symptom index for benign prostatic hyperplasia
J Urol
(1992) - et al.
Development and evaluation of the Kansas City Cardiomyopathy Questionnairea new health status measure for heart failure
J Am Coll Cardiol
(2000) - et al.
Quality of life assessment in individuals with lung cancertesting the Lung Cancer Symptom Scale (LCSS)
Eur J Cancer
(1993) - et al.
The American Urological Association symptom indexdoes mode of administration affect its psychometric properties?
J Urol
(1995) - et al.
Distinction between symptoms of voiding and filling in benign prostatic hyperplasiafindings from the Health Professionals Follow-up Study
Urology
(1998) - et al.
Benign prostatic hyperplasia specific health status measures in clinical researchhow much change in the American Urological Association symptom index and the benign prostatic hyperplasia impact index is perceptible to patients?
J Urol
(1995) - et al.
Filling and voiding symptoms in the American Urological Association symptom indexthe value of their distinction in a Veterans Affairs randomized trial of medical therapy in men with a clinical diagnosis of benign prostatic hyperplasia
J Urol
(2000) - et al.
Correlation of the American Urological Association symptom index with self-administered versions of the Madsen-Iversen, Boyarsky and Maine Medical Assessment Program symptom indexes
J Urol
(1992) - et al.
Radical retropubic prostatectomy reduces symptom scores and improves quality of life in men with moderate and severe lower urinary tract symptoms
J Urol
(1999) - et al.
Relationship of symptoms of prostatism to commonly used physiological and anatomical measures of the severity of benign prostatic hyperplasia
J Urol
(1993)
The American Urological Association symptom score in the evaluation of men with lower urinary tract symptomsat 2 years of followup, does it work?
J Urol
The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults
Chest
Detecting alcoholismthe CAGE questionnaire
JAMA
Utility of a new procedure for diagnosing mental disorders in primary care. The PRIME-MD 1000 study
JAMA
Changes in youth cigarette use and intentions following implementation of a tobacco control programfindings from the Florida Youth Tobacco Survey, 1998-2000
JAMA
The mini-CEXa method for assessing clinical skills
Ann Intern Med
Factorial validation of a widely disseminated educational framework for evaluating clinical teachers
Acad Med
Evidence-Based MedicineHow to Practice and Teach EBM
Interpretation of Diagnostic Tests
How reliable are assessments of clinical teaching? A review of the published instruments
J Gen Intern Med
Burnout and self-reported patient care in an internal medicine residency program
Ann Intern Med
Patient-physician communication about out-of-pocket costs
JAMA
Hand hygiene among physiciansperformance, beliefs, and perceptions
Ann Intern Med
Validity
Validity issues in clinical assessment
Psychol Assess
Standards for Educational and Psychological Testing
Statistics notesvalidating scales and indexes
BMJ
Validityon the meaningful interpretation of assessment data
Med Educ
An argument-based approach to validity
Psychol Bull
Cited by (1055)
Work Ability Index: Psychometric Testing in Aeronautical Industry Workers
2024, Safety and Health at WorkThe reliability and validity of the Timed Up and Go test in patients ongoing or following lumbar spine surgery: a systematic review and meta-analysis
2024, Egyptian Journal of Neurology, Psychiatry and NeurosurgeryInternal consistency and dimensionality assessment of the Cognitive Social Capital Scale among Colombian adults
2024, Medicina Clinica y Social