Inpatient diagnostic assessments: 1. Accuracy of structured vs. unstructured interviews

Abstract

This study compared structured vs. unstructured interviews for making psychiatric diagnoses. Three clinicians independently diagnosed 56 inpatient-subjects, each using a different method: (1) the unstructured Traditional Diagnostic Assessment (TDA), the standard method of clinical practice; (2) the Structured Clinical Interview for DSM-Clinical Version (SCID-CV), a widely used structured method; and (3) the Computer Assisted Diagnostic Interview (CADI), a structured computer-based method. Once finished, the three clinicians developed a Consensus diagnosis, using Spitzer's LEAD Standard (L=Longitudinal evaluation of symptomatology, E=Evaluation by expert consensus, AD=All Data from multiple sources). Diagnoses were assigned to one of 10 groups (cognitive impairment, general medical condition-induced, alcohol-induced, drug-induced, mania, depression, schizophrenia, schizoaffective, psychosis NOS, and anxiety). Diagnostic accuracy for each method, measured against Consensus, was as follows: TDA — agreement=53.8%, kappa=0.4325 (‘fair’); SCID-CV — agreement=85.7%, kappa=0.8189 (‘excellent’); CADI — agreement=85.7%, kappa=0.8147 (‘excellent’). All three methods reached acceptable levels of diagnostic accuracy. Structured methods (SCID-CV, CADI) were significantly better than the unstructured TDA.

Introduction

DSM-I (1952) and DSM-II (1968) defined syndromes more as psychological reactions than as biopsychosocial conditions and solicited clinicians’ subjective evaluations (‘clinical judgment’) as much as their objective assessments. Consequently, many clinicians focused more on patient's ‘problems’ arising from their psychodynamic status than on mental disorders generated by biopsychosocial conditions; treatment of choice was usually psychotherapy (Whitehorn, 1944, Sullivan, 1970). Prior to 1980, ‘the classification system and specialty of psychiatry [were] often held as less ‘medical’ or scientific than other branches of medicine. Extensive efforts to correct this perception resulted in a paradigmatic shift from hermeneutic [interpretive…theory based] to empirically based approaches, and the development of a nosology intended to increase diagnostic reliability and facilitate research efforts. These changes are embodied in…DSM-III [1980] and DSM-IV [1994]’ (Bogenschutz and Nurnberg, 2000, p. 824).

The Traditional Diagnostic Assessment (TDA), the unstructured interview that is the standard of practice for that task throughout psychiatry (Othmer and Othmer, 1994, Hales et al., 1995, Kaplan and Sadock, 1998), has evolved alongside DSM. To increase diagnostic validity and reliability, clinicians since 1980 have expanded the Mental Status Examination (MSE) and the history of present and past illnesses, and are using more structured formats: ‘Most clinicians are now imbued not only with the content of the DSM-IV criteria sets but also with a different method of interviewing patients and eliciting psychopathology. Compared with pre-DSM-III days, clinical evaluations are now much more likely to be more semi-structured and less open-ended. Clinicians are much more likely to ask specific questions to elicit the $\text{symptoms and course information}$ [i.e. MSE and history of illness; emphasis added] necessary to make a DSM diagnosis...no evaluation is complete unless the diagnostic questions are asked’ (Frances et al., 1995, p. 66).

Accurate diagnosis is pragmatically as well as theoretically important, because ‘treatment plans are often based on …diagnostic type. The advent of disease-specific treatment protocols has heightened the necessity for accurate diagnostic procedures’ (Basco et al., 2000).

Textbooks seem not to express concerns or list references regarding the validity and reliability of the TDA (Othmer and Othmer, 1994, Hales et al., 1995, Kaplan and Sadock, 1998). Core Readings in Psychiatry (Sacks et al., 1995), the APA bibliography, has 11 references about diagnostic validity, but none about TDA.

Researchers have reported the diagnostic variance of the TDA (Williams et al., 1992, McGorry et al., 1995, Mojtabai and Nicholson, 1995, Hill et al., 1996, van Praag, 1997), but these findings have not led to significant changes in clinical practice.

Although clinicians use the TDA with unquestioning faith, researchers mostly avoid using it as an exclusive diagnostic method in clinical trials. Researchers also tend to avoid using TDAs exclusively in their work to validate syndromes for DSM-IV (Widiger et al., 1994, Widiger et al., 1996).

Following DSM-III's introduction in 1980, researchers began to evaluate how clinicians used it. Lipton and Simon (1985, p. 370) found that ‘documentation of DSM-III criteria for assigned chart diagnoses was not present in 80% of the 131 charts reviewed’. Skodol et al. (1984) found that 75% of incorrect diagnoses made with DSM-III resulted from incorrect application of criteria. Robinson et al. (1985) found that a university-affiliated faculty misidentified 13–48% of DSM-III criteria for major depression. Greist (1998) examined how clinicians followed diagnostic rules and found error rates between 10 and 37%. Garb (1998) reviewed research on clinical judgment and found that clinicians often failed to adhere to diagnostic criteria. Such reports justified the development of methods to improve diagnostic accuracy of the TDA.

In 1980, the corresponding author undertook to ‘computerize’ DSM. This took its present form in 1994 with the advent of CADI, which directs the clinician to evaluate all relevant criteria in DSM-IV algorithms, by sequentially displaying questions and answers on the computer screen. It is a structured interview. Clinicians enter their assessments into the computer, and the program matches them with DSM-IV algorithms to make diagnoses. CADI has been in beta testing since 1996. CADI has been previously reported at national meetings (Miller, 1996, Miller, 1998, Miller, 1999, Miller, 2000a, Miller, 2000b).

The MSE underlies DSM diagnostics, so the corresponding author restructured it to use Key Criteria as MSE items. Key Criteria are those criteria (listed first in DSM algorithms) that must be evaluated before the linked diagnosis can be ruled in or ruled out. Table 1 shows 13 diagnostic groups and their linked 25 Key Criteria. If any Key Criterion is positive, the linked diagnosis must be evaluated completely. If all Key Criteria for a linked diagnosis are negative, that diagnosis can be ruled out. Thus, CADI operates in the same way as ‘Decision Trees for Differential Diagnosis’ (DSM-IV, Appendix A).

The CADI and the paper-and-pencil SCID both assess DSM algorithms. The CADI differs in that it has an MSE (based on Key Criteria), while SCID does not. The CADI makes diagnoses programmatically, while the paper-and-pencil SCID requires the clinician to choose the diagnoses.

Lieff (1987), Taintor et al. (1997), and Blacker (2000) have described different computerized diagnostic interviews, a topic outside the compass of this article.

The purposes of the study were twofold: (1) to compare structured vs. unstructured methods for making psychiatric diagnosis; (2) to assess the use of computer assistance for structured diagnosis and the validity and reliability of the CADI.

We used the paper-and-pencil format of the SCID, because it is the format most widely used, and because it enabled the comparison of three distinct methods — unstructured paper-and-pencil (TDA), structured paper-and-pencil (SCID-CV), and structured computer-assisted (CADI).