Abstract
The identification and reduction of measurement errors is a major challenge in psychological testing. Most investigators rely solely on classical test theory for assessing reliability, whereas most experts have long recommended using generalizability theory instead. One reason for the common neglect of generalizability theory is the absence of analytic facilities for this purpose in popular statistical software packages. This article provides a brief introduction to generalizability theory, describes easy to use SPSS, SAS, and MATLAB programs for conducting the recommended analyses, and provides an illustrative example, using data (N= 329) for the Rosenberg Self-Esteem Scale. Program output includes variance components, relative and absolute errors and generalizability coefficients, coefficients for D studies, and graphs of D study results.
Article PDF
Similar content being viewed by others
References
Blascovich, J., &Tomaka, J. (1991). Measures of self-esteem. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.),Measures of personality and social psychological attitudes (pp. 115–160). New York: Academic Press.
Brennan, R. L. (2000a). (Mis)conceptions about generalizability theory.Educational Measurement: Issues & Practice,19, 5–10.
Brennan, R. L. (2000b). Performance assessments from the perspective of generalizability theory.Applied Psychological Measurement,24, 339–353.
Brennan, R. L. (2001).Generalizability theory. New York: Springer.
Crick, J. E., &Brennan, R. L. (1983).Manual for GENOVA: A generalized analysis of variance system (American College Testing Technical Bulletin 43). Iowa City: ACT, Inc.
Cronbach, L. J., Gleser, G. C., Nanda, H., &Rajaratnam, N. (1972).The dependability of behavioral measurements: Theory of generalizability of scores and profiles. New York: Wiley.
Eason, S. (1991). Why generalizability theory yields better results than classical test theory: A primer with concrete examples. In B. Thompson (Ed.),Advances in educational research: Substantive findings, methodological Developments (Vol. 1, pp. 83–98). Greenwich, CT: JAI.
Hoyt, W. T., &Melby, J. N. (1999). Dependability of measurement in counseling psychology: An introduction to generalizability theory.Counseling Psychologist,27, 325–352.
Kieffer, K. M. (1999). Why generalizability theory is essential and classical test theory is often inadequate. In B. Thompson (Ed.),Advances in social science methodology (Vol. 5, pp. 149–170). Stamford, CT: JAI.
Marcoulides, G. A. (1999). Generalizability theory: Picking up where the Rasch IRT model leaves off? In S. E. Embretson & S. L. Hershberger (Eds.),The new rules of measurement: What every psychologist and educator should know (pp. 129–152). Mahwah, NJ: Erlbaum.
Marcoulides, G. A. (2000). Generalizability theory. In H. E. A. Tinsley & S. D. Brown (Eds.),Handbook of applied multivariate statistics and mathematical modeling (pp. 527–551). San Diego: Academic Press.
Rosenberg, M. (1979).Conceiving the self. New York: Basic Books.
Searle, S. R., Casella, G., &McCulloch, C. E. (1992).Variance Components. New York: Wiley.
Shavelson, R. J., &Webb, N. M. (1991).MMSS generalizability theory: A primer. Newbury Park, CA: Sage.
Shavelson, R. J., Webb, N. M., &Rowley, G. L. (1989). Generalizability theory.American Psychologist,44, 922–932.
Shrout, P. E., &Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability.Psychological Bulletin,86, 420–428.
Strube, M. J. (2000). Reliability and generalizability theory. In L. G. Grimm & P. R. Yarnold (Eds.),Reading and understanding more multivariate statistics (pp. 23–66). Washington, DC: American Psychological Association.
Webb, N. M., Rowley, G. L., &Shavelson, R. J. (1988). Using generalizability theory in counseling and development.Measurement & Evaluation in Counseling & Development,21, 81–90.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by a grant from the Social Sciences and Humanities Research Council of Canada.
Rights and permissions
About this article
Cite this article
Mushquash, C., O’Connor, B.P. SPSS and SAS programs for generalizability theory analyses. Behavior Research Methods 38, 542–547 (2006). https://doi.org/10.3758/BF03192810
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03192810