Abstract
In schizophrenia, treatments that improve outcomes have not been reliably disseminated. A major barrier to improving care has been a lack of routinely collected outcomes data that identify patients who are failing to improve or not receiving effective treatments. To support high quality care, the VA Mental Health QUERI used literature review, expert interviews, and a national panel process to increase consensus regarding outcomes monitoring instruments and strategies that support quality improvement. There was very good consensus in the domains of psychotic symptoms, side-effects, drugs and alcohol, depression, caregivers, vocational functioning, and community tenure. There are validated instruments and assessment strategies that are feasible for quality improvement in routine practice.
Introduction
Psychosocial and medication treatments have consistently and repeatedly been found to improve outcomes in schizophrenia (American Psychiatric Association 2004; Kreyenbuhl et al. 2010). However, key evidence-based treatments have not been widely disseminated (Institute of Medicine 2006; Lehman 1999; President’s New Freedom Commission on Mental Health 2003). Increasingly, policy makers and healthcare organizations are interested in improving the quality of care. One leading approach to improving care starts by identifying patients who would benefit from a specific evidence-based treatment, and then ensuring that they have access to that treatment (Rubenstein et al. 2000). While this approach has improved outcomes in chronic disorders such as diabetes (Asch et al. 2004; Kupersmith et al. 2007), it has proven very difficult to implement in mental illness, and especially schizophrenia (Young et al. 2004). A major challenge has been a lack of valid, routinely collected outcomes data to identify patients who are failing to improve clinically and functionally, or who are not receiving appropriate care. In part, the lack of such data reflects uncertainty regarding which outcome domains to monitor and how to feasibly collect routine outcomes data in large populations of patients. Quality improvement efforts would benefit from better consensus regarding methods for routine monitoring of outcomes of schizophrenia.
Most successful efforts to improve healthcare quality have been conducted outside of mental health, and few have occurred in schizophrenia. In diabetes, for example, medical records and electronic data routinely include information needed to evaluate and improve care. These include data on demographics, hemoglobin A1c (HgA1c, a measure of diabetes severity), prescriptions, and procedures received. Since HgA1c and prescription data are routinely collected and recorded electronically, policy makers have been able to identify patients who have poorly controlled diabetes and apply national quality improvement efforts to improve diabetes care. In schizophrenia, there are also measures of illness severity and need for treatment. At present, however, these outcomes are not reliably documented in medical records (Young et al. 2004). As a result, policy makers have had difficulty determining the extent to which quality improvement and additional treatment resources would have value in schizophrenia.
There are multiple barriers to gathering outcome data in schizophrenia. Outcome assessment methods used in research protocols have proved too costly and time-intensive to widely adopt in routine practice. Clinicians can be trained to achieve high levels of reliability in standardized instruments such as the Brief Psychiatric Rating Scale (Ventura et al. 1993). However, substantial staff time is required for administration and for training required to achieve and maintain adequate reliability. Most provider organizations cannot afford to dedicate staff to this assessment role or to intensively train staff to reliably perform assessments.
There are alternative strategies for outcomes monitoring. These include monitoring based on billing or administrative data, assessment by clinicians without intensive training in standardized instruments, or patient self-assessment. Administrative and billing data are relatively easy to obtain for large patient populations, and can be used to track basic demographics and sometimes medication prescriptions. However, these data can be inaccurate or incomplete, and lack detailed clinical content. Alternatively, it is possible to ask clinicians to routinely complete rating instruments as part of treatment visits, without intensive training and reliability checks. Some organizations have, for example, mandated regular assessments using the Global Assessment of Functioning (GAF) Scale. This has met with limited success. Reliability with untrained raters is very poor (Niv et al. 2007b) and many clinicians resist incorporating standardized rating and documentation into routine practice. In contrast, monitoring based on patient self-report and/or self-assessment is relatively easy to implement in clinical settings (Brown et al. 2005; Chinman et al. 2007). While validity varies, it can be quite good with certain domains and instruments (Eisen et al. 2004). In addition, there is inherent value in patient-centered assessment (Niv et al. 2007a).
This manuscript describes a process within the U.S. Department of Veterans Affairs (VA) to (1) identify outcome indicators that can be used for routine monitoring in support of projects to improve the quality of care for schizophrenia, and (2) identify instruments and methods for monitoring these outcome indicators. The VA provides care for more than 100,000 people with schizophrenia and is a national leader in healthcare quality and the use of clinical information systems to improve care (Jha et al. 2003; Kupersmith et al. 2007). In 1998, the VA established a national Quality Enhancement Research Initiative (QUERI) to inform quality improvement and identify effective strategies for implementing research evidence into routine clinical practice across nine disorders (Francis and Perlin 2006; McQueen et al. 2004). The Mental Health QUERI focuses on improving care delivered to individuals with depression and schizophrenia, and is a partnership between leading national researchers, policy makers, and treatment experts. Since prior efforts to develop outcome measurement in schizophrenia (Cuffel et al. 1997; Veterans Health Administration 2003) did not meet the QUERI’s goals, a Schizophrenia Outcomes Workgroup was convened in August 2004. This manuscript presents the methods used and products developed by this Workgroup.
Methods
The first task of the Workgroup was to identify outcome domains that can support implementation of high-priority, evidence-based treatments. Based on a review of national treatment guidelines and the new emphasis on recovery, the Workgroup chose to focus on the following outcomes: psychotic symptoms, medication side-effects, drug and alcohol use, depression, family and caregivers, vocational functioning, community tenure and housing. The domains of negative and cognitive symptoms were acknowledged to also be important, but were not included because national guidelines do not contain evidence-based practices to improve these outcomes. Comorbid medical conditions were also acknowledged as critically important, since patients with schizophrenia are at high risk for a variety of medical problems, including diabetes mellitus, cardiovascular disease, hypertension, chronic obstructive pulmonary disease, and infectious diseases (American Psychiatric Association 2004). However, development of consensus in this area would require review of a very large body of guidelines from outside of specialty mental health, and was beyond the scope of this Workgroup.
For each domain, the Workgroup sought to identify indicators of: (a) need for treatment, and (b) change in clinical status associated with treatment. Potential assessment strategies, including patient self-assessment, rater or clinician-based assessment, and use of administrative or billing data, were reviewed. It was agreed that feasibility of routine assessment and documentation was critical, and therefore this manuscript focuses on self-report and administrative data approaches. The Workgroup recognized that to improve care, it is also necessary to know whether patients who need treatment have received it. This can be assessed using data on treatment utilization and treatment fidelity. While this data is quite important, discussion of alternatives for assessing treatment utilization and fidelity indicators was beyond the scope of this Workgroup.
A modified Delphi process was used to increase consensus regarding outcomes monitoring options (Fitch et al. 2001). This process included literature review, expert interviews, and a national panel of experts and stakeholders. The literature review covered published journal articles and books, unpublished documents on treatment and outcome assessment in schizophrenia, plus communication via phone and email with key informants. These informants included experts who had published on schizophrenia outcome assessment, Workgroup members, and individuals who had key unpublished work. Each informant was asked for literature they were aware of on the subject (published and unpublished) and for the names of other individuals who are knowledgeable in this area. Following several iterations, the identified documents were reviewed. A draft paper was prepared summarizing, for each domain, evidence-based treatments, common problems with treatment quality, and potential outcome indicators, instruments and assessment modalities.
A national panel was assembled that included experts in clinical treatment, quality improvement, and outcomes assessment, as well as national and regional policy-makers. In September 2005, this panel met via video conference. Prior to the conference, panel members reviewed the results of the literature review. During the conference there was, for each targeted domain, review of the evidence supporting treatments in the domain and common problems with treatment implementation. Within each domain, the panel suggested constructs that can identify (a) need for treatment, and (b) change in status associated with treatment. For each construct identified, alternative strategies and instruments were discussed that make use of administrative or billing data, medical records, patient self-assessment, clinician assessment, or interviewer-based assessments. Recommendations were made regarding alternative outcomes monitoring strategies. Panel members discussed and negotiated areas of disagreement to increase consensus regarding the recommendations. Following the conference, panel members provided additional feedback by email and phone. A document was prepared summarizing project consensus. This was circulated to the Workgroup. An iterative process of revisions led to the final consensus reported in this manuscript and endorsed by the Mental Health QUERI. The authors of this manuscript have no known conflicts of interest, and certify their responsibility for this article.
Results
Psychotic Symptoms
Symptom Severity
National guidelines concur that the treatment of psychosis includes the prescription and use of appropriate medication to control psychotic symptoms (American Psychiatric Association 2004; Kreyenbuhl et al. 2010; Miller et al. 2004). By “appropriateness,” we mean the extent to which clinicians are using effective strategies to address symptoms and side-effects (Young 1999). Almost all patients who are in treatment for schizophrenia are prescribed an antipsychotic medication (Lehman 1999). However, many patients do not receive treatment that appropriately addresses their symptoms or side-effects (Young et al. 1998). While choice of antipsychotic medication is controversial, there is agreement that symptoms and side-effects need to be managed, and there are strategies for doing so. It is useful to identify which patients are not receiving appropriate medication management for psychosis, so that quality improvement efforts can focus on their needs. The appropriateness of prescribing can be evaluated using data on the severity of psychotic symptoms in combination with data on medications prescribed (Young et al. 1998). Unfortunately, administrative data and chart review are not reliable or valid methods for assessing psychosis (Young et al. 1998). One alternative data source is patient self-assessment. Self-report measures of psychosis severity can be found in Table 1.
Medication Adherence
Another common challenge is poor medication adherence (Lacro et al. 2002). It is useful to identify which patients are poorly adherent (see Table 1 for measures), so that quality improvement efforts can implement interventions that improve adherence. There are evidence-based strategies that improve adherence, including cognitive-behavioral approaches and long-acting injectable medications (Zygmunt et al. 2002).
Treatment Resistance
Clozapine is the only antipsychotic that has superior efficacy in patients with treatment-refractory psychosis (Chakos et al. 2001; McEvoy et al. 2006). According to national guidelines, it should be used in patients who have had at least two trials of other antipsychotic medications, but who continue to have clinically significant psychotic symptoms (Kreyenbuhl et al. 2010). However, clozapine is more complex to prescribe than other medications and is underused (Busch et al. 2004; Leslie and Rosenheck 2007). It is likely that prescription data can be used to detect under-use of clozapine. Treatment-refractory patients might be identified by detecting polypharmacy or high dosages of antipsychotic agents, but this approach has not yet been validated. Another approach is to identify patients who have persistent high levels of psychotic symptoms, adequate adherence, and trials of at least two antipsychotic medications. Alternatively, there is no reason to believe that treatment resistance varies by clinic. Therefore, it should be possible to establish a benchmark for clozapine use that would reflect a reasonable prevalence of clozapine use. It has been estimated that between 25 and 40% of patients are refractory to treatment, and up to a half of these individuals have substantial improvement with clozapine (Kane and Marder 2005). High performing states have had about 15% of patients on clozapine (Covell et al. 2002).
Recommendations
Further research is needed to determine the extent to which assessment can inform efforts to improve treatment, reduce psychosis, and improve psychosis-related outcomes, as well as the optimal intervals for assessments. Adherence is clearly important and can be monitored using pharmacy data. To improve treatment of psychosis, it will be necessary to systematically monitor the severity of psychosis in patients. Self-report is a valid method for screening for psychosis (Niv et al. 2007a), and the panel recommends the BSI or BASIS-R. When psychosis assessment is combined with data on medication prescribing, it is possible to identify patients who are not improving with their current regimen. Under-use of clozapine can be detected with this approach, or by comparing prescribing rates to benchmarks.
Medication Side-Effects
Most antipsychotic prescriptions are now for second generation agents. The most common side-effect with this medication class is weight gain and related metabolic complications, such as diabetes and hyperlipidemia. These place patients at elevated risk for cardiovascular disease. Less common side-effects include extrapyramidal side-effects (especially tardive dyskinesia and akathisia), and sexual dysfunction. National panels have developed recommendations for the screening and monitoring of antipsychotic medication side-effects (American Diabetes Association 2004; American Psychiatric Association 2004; Marder et al. 2004) (see Table 1 for measures).
Recommendations
The most important side-effects of second-generation antipsychotic agents are weight gain and related medical conditions. Patients can self-monitor their weight. Clinicians should routinely calculate BMI and perform laboratory monitoring for diabetes and hyperlipidemia according to national guidelines. Weight and metabolic data can be combined with data on prescribing and other treatments. These data identify patients who are overweight or gaining weight, and who are not receiving medication changes or psychosocial interventions that improve weight (Newcomer 2007; Newcomer et al. 2008). Similarly, it is possible to identify patients with diabetes or hyperlipidemia who are not receiving needed medical services.
Patients who are prescribed first-generation agents should be assessed for extrapyramidal side-effects. Patients who are prescribed a first-generation agent, risperidone, or paliperidone should be monitored for sexual dysfunction. Extrapyramidal and sexual side-effects can be reliably evaluated by self-assessment.
Drug and Alcohol Use
Evidence-based care for schizophrenia and comorbid substance abuse consists of coordinated, simultaneous provision of mental health and substance abuse treatment (McHugo et al. 2004). To ensure the provision of evidence-based services for the dually diagnosed, it is necessary to know when patients are abusing drugs or alcohol and, more specifically, what substances are being used, the frequency and amount of use, and impairments in functioning resulting from substance use. Objective assessment of substance use, e.g., urine toxicology screens and breathalyzer data, is the gold standard for guiding treatment. Since objective data are often not available, monitoring may also be based on administrative data, clinician assessment, or self-assessment. Administrative data tend to be of limited use because rates of detection and documentation of substance abuse in routine care are low and of questionable accuracy. Also, when diagnoses are made, there is often limited documentation regarding the severity and recency of the use. Self-administered assessments may be useful in a clinical setting as they are easier to obtain than clinician-administered assessments and have better predictive utility (Wolford et al. 1999). Patients appear to be more forthcoming with self-administered assessments than during interviews. Self-report measures of alcohol and drug use can be found in Table 1.
Recommendations
Regular screening should be used to identify substance abuse and associated problems and to guide treatment. Self-administered patient reports can provide data of similar or better quality than clinician assessment while reducing clinician burden. The AUDIT or AUDIT-C can be used to screen for alcohol use disorders and the DAST can be used to screen for drug use disorders.
Depression and Suicidality
Depressive symptoms have been shown to be important determinants of quality of life among patients with schizophrenia. Some treatment guidelines support the use of antidepressant medication in this population for those who continue to experience depression after a reduction in positive psychotic symptoms (American Psychiatric Association 2004). These guideline recommendations have been made even though no pharmacological or psychosocial treatments for comorbid depression and schizophrenia have consistently proven to improve outcomes. Suicide is a common cause of premature death in schizophrenia, and individuals with schizophrenia are nine times as likely to die by suicide as the general population. Self-report measures of depression and suicidality are valid, and can be found in Table 1.
Recommendations
Routine assessment of depression and suicidality is important, though there is limited evidence to guide the frequency of these assessments. Screening and assessment for depression can be done by self-assessment using the BDI or PHQ-9. The PHQ-9 is quicker to administer. The BASIS-R suicidality items are a reasonable choice to screen for suicidality. However, further research is needed to develop and evaluate psychosocial and medication interventions that improve depression and suicidality in schizophrenia.
Family and Caregivers
Family psychoeducation is included in national treatment guidelines based on its positive effect on outcomes for patients and their family members (Kreyenbuhl et al. 2010). Family interventions have been shown to reduce relapse and hospitalizations, and have broader benefits on psychosocial adjustment, such as vocational functioning (McFarlane et al. 2000; Pilling et al. 2002; Pitschel-Walz et al. 2001). Despite mandates to implement family services and family-driven care (President’s New Freedom Commission on Mental Health 2003), research indicates that most families do not have even minimal contact with mental health clinicians (Resnick et al. 2005). To ensure that families and other caregivers have access to services and clinicians, it is necessary to know which patients are in close contact with family or caregivers. This information is not reliably found in administrative data or medical records. However, it can be reliably obtained through patient self-report. Although there is no cut-off for intensity of contact that defines an “involved caregiver,” this is sometimes defined as contact of at least several times a month. Research on the potential benefits of family involvement suggests that high priority should be given to engaging the families of patients who have severe psychotic symptoms, low medication adherence, or who report a negative family environment. Strategies for identifying the first two groups are found in the section on psychotic symptoms. There are several scales that assess family environment and relationships (see Table 1). However, research does not support limiting interventions to families with particular scores on these instruments. Clinicians may find them helpful, but none is recommended for routine assessment. Structured, long-term family services have been associated with reduced psychosis and less hospitalization. Evaluation of psychosis is described above. Hospitalizations and related crisis services can be identified using a combination of administrative data and self-report. There are also established scales to measure family outcomes, including knowledge of illness and family burden, though the validity and reliability of these instruments is uncertain.
Recommendations
Family members and other caregivers who might benefit from increased involvement in care should be identified and engaged through outreach. The presence of an involved family or caregiver can be ascertained by patient self-report. At a minimum, caregivers should be provided with information, given access to the treatment team, and given referrals to family support organizations. Evidence-based family interventions should be offered. In regard to the outcomes of family services, relevant domains for assessment include psychotic symptoms, medication compliance, and hospitalization.
Vocational Functioning
Supported employment (SE) is an evidence-based practice that substantially increases rates of competitive employment in populations with schizophrenia (Crowther et al. 2001). According to guidelines, SE should be offered to individuals who are not employed and wish to find work, and to individuals who are currently working but find their work unsatisfying or too stressful (Kreyenbuhl et al. 2010). It is often impossible to determine from administrative data or medical records which patients should be offered SE, since work status and goals are not consistently documented. While research on assessment is limited, it is likely that these domains can be assessed by self-report (see Table 1).
Recommendations
Self-report measures should be used to identify eligible patients and to assess the outcomes of SE. Employment status can be assessed by self-report, including dollars earned and hours worked. Job satisfaction can be assessed using the Indiana Job Satisfaction Scale.
Community Tenure and Housing
A key component of recovery for people with schizophrenia is successfully living in their communities; for example, not being hospitalized, incarcerated, or homeless. About 10% of people with schizophrenia are homeless (American Psychiatric Association 2004), and there is a spectrum of autonomy in housing arrangements. The Workgroup saw value in measuring the strength of an individual’s integration into their community, but there was little research or consensus regarding how to measure this construct. Therefore, the focus was restricted to housing and duration of community tenure.
Some individuals with schizophrenia require specific psychosocial services to live in the community or require long-term, structured living environments. Assertive community treatment (ACT) is effective in improving community tenure and is included in national treatment guidelines. Patients may be eligible for ACT if they are homeless, hospitalized, incarcerated, or at high risk for these outcomes. Repeated hospitalizations can be identified using billing or administrative data. Since these datasets typically do not include all hospitalizations, they can be augmented by self-report. In addition to ACT, there are other care models that improve access to housing (Young and Magnabosco 2004).
Important outcomes of ACT and housing services include housing status and satisfaction with one’s living situation. Administrative and billing data typically lack accurate and current information regarding community tenure and housing. There has been limited research on the validity of self-administered measures of housing, although one study found patient reports of housing status to be highly correlated with family and clinician reports (Cuffel et al. 1997). There are widely used interviewer-administered instruments that are likely to provide an accurate assessment of housing status, including homelessness, and stable or unstable community housing (Table 1).
Consumer-driven interventions, such as peer support or consumer outreach workers, also show promise for improving community tenure (Chinman et al. 2006). However, there is insufficient evidence to know which individuals should receive consumer-driven interventions, how to provide them, or how to characterize their effectiveness. They are not yet covered in national treatment guidelines, and the Workgroup was not able to characterize relevant needs for outcome assessment.
Recommendations
Eligibility for ACT should be determined using a combination of administrative and patient self-administered data assessing housing, hospitalizations and incarceration. The major challenge to using these data to evaluate access to ACT is the nature of the patient population being served, which may be homeless and minimally engaged in treatment. Novel approaches for identifying eligible individuals for ACT should be developed and studied. In regard to housing services, homelessness and unstable housing should be assessed by self-assessment. Risk for homelessness, while important, has not been well characterized. Satisfaction with housing should be assessed using items from the Quality of Life Interview (Lehman 1988).
Discussion
The Schizophrenia Outcomes Workgroup used literature review, expert interviews, and a national expert panel process to increase consensus regarding outcomes monitoring strategies that can be used to support quality improvement and the delivery of evidence-based treatments for schizophrenia. It is clear that outcome measurement is better developed in some domains than others. Good consensus was reached for the domains of psychotic symptoms, medication side-effects, drug and alcohol use, depression, family and caregivers, vocational functioning, community tenure, and housing. In these domains, there are well developed, validated, self-report outcomes assessment instruments, and often multiple excellent choices for routine outcomes monitoring. However, there are areas that would benefit from further research on measurement, including, for example, identifying eligible patients for clozapine or supported employment. It was not possible to develop consensus in other domains, such as negative symptoms, cognitive symptoms, and use of peer support interventions, because efficacy research is not sufficiently developed. National treatment guidelines offer limited guidance regarding treatments that should be provided to patients and, therefore, outcomes to be monitored.
Although there are high quality outcomes assessment instruments, some assessment strategies are much easier than others to implement in routine treatment settings. The most feasible strategies for obtaining outcomes data are the use of administrative or billing data, and self-administered assessments. Administrative data are typically not designed for this purpose, and often contain inaccurate or incomplete information on outcomes and limited clinical content. This is somewhat better in systems that have implemented Electronic Medical Records (EMRs). However, even with EMRs, outcomes data are inconsistently documented or encoded in fields that are readily usable for quality improvement (Young 2007). To standardize medical records data, healthcare organizations sometimes rely on clinicians to document outcomes, an approach that has not worked well. Although many clinicians could be trained to high levels of reliability in standardized instruments, they typically do not have the time or interest necessary for training, routine performance, and entry of outcomes data into medical records.
Self-administered patient assessments are another strategy for outcome monitoring. They requires less time from busy clinicians, do not require repeated trainings, and alleviate concerns about clinician gaming of assessments. Methods of self-assessment vary and can be determined by the needs of specific projects and healthcare organizations. Methods of self-assessment have included traditional paper and pencil forms, scannable forms, electronic self-entry devices, and internet access by patients. Electronic devices at clinics can include dedicated kiosks or tablet computers with touch-screen and audio interfaces. Self-entry systems can generate reliable and valid data, are well accepted by clinic populations with schizophrenia, and can be feasible to implement in clinic settings (Chinman et al. 2004, 2007).
For quality improvement, it is necessary to know which patients would benefit from specific treatments, and which of these patients have not been receiving those treatments. The Workgroup did not review methods for monitoring the utilization or appropriateness of services that patients receive. However, there was agreement that, in addition to monitoring outcomes, it will also be important to monitor the provision of effective care. For psychosocial treatments in particular, it is important to monitor the extent to which treatments demonstrate fidelity to practices that are known to improve outcomes.
This project had a number of limitations. First, the outcome strategies and instruments recommended are not intended to replace clinical decision-making in individual patients. Rather, they were developed to support organizational efforts to improve care in populations of patients. Also, the instruments reviewed are not an exhaustive list. There are likely to be other instruments with good psychometric properties that are suitable for monitoring outcomes in these domains. In any specific instance of quality-oriented outcomes monitoring, selection of specific strategies and instruments must be a function of the goals of the project and the characteristics of the healthcare system.
Healthcare organizations have succeeded in improving care by making use of quality improvement that includes routine collection of outcome data. The routine and systematic collection of data allows for identification of unmet needs, and redesign of services to improve access to effective practices and improve outcomes. There have been fewer successes with this approach in schizophrenia than in other chronic medical disorders, and there are specific challenges to collection and encoding of outcomes data in this population. However, the Workgroup believes that there is substantial potential to improve access to effective treatments in populations with schizophrenia. This begins with systematic monitoring of outcomes. Providing outcomes data to clinicians would not by itself be expected to improve the quality of care (Fihn et al. 2004; Stone et al. 2002). Rather, this data needs to be used as part of systematic projects to reorganize service delivery. Systematic research will bee needed to evaluate these efforts to improve care.
There was strong consensus regarding viable strategies for routine outcomes monitoring. There are some domains, such as psychotic symptoms, medication side-effects, and medication adherence, where there are existing instruments and data sources that are ready now for widespread implementation and use. There are other domains, such as clozapine, employment, and community reintegration where useful instruments are available, but measurement could be substantially strengthened with additional research. It was not possible, or desirable, to definitively recommend one specific instrument and measurement strategy for all quality improvement efforts. Each measurement strategy and instrument has pros and cons that must be weighed. And, to be successful, quality improvement needs to attend both to evidence and also local context when selecting measurement strategies. Therefore, when there were multiple acceptable instruments, the Workgroup did not select one choice, but left this choice to the quality improvement project that will use the data. With regard to alternative measurement strategies, patient self-report appears to be the most broadly feasible method in many domains. Self-report should be supplemented, where helpful, by clinician documentation and administrative data. Whatever path an organization selects with regard to domains, instruments, and data collection strategies, there are important opportunities to use outcomes data as part of projects to improve care for schizophrenia.
References
American Diabetes Association. (2004). Consensus development conference on antipsychotic drugs and obesity and diabetes. Diabetes Care, 27, 596–601.
American Psychiatric Association. (2004). Practice guideline for the treatment of patients with schizophrenia, second edition. American Journal of Psychiatry, 161, 1–56.
Asch, S. M., McGlynn, E. A., Hogan, M. M., Hayward, R. A., Shekelle, P., Rubenstein, L., et al. (2004). Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Annals of Internal Medicine, 141, 938–945.
Bebout, R. R., Drake, R. E., Xie, H., McHugo, G. J., & Harris, M. (1997). Housing status among formerly homeless dually diagnosed adults. Psychiatric Services, 48, 936–941.
Beck, A. T. (1993). Cognitive therapy: Past, present, and future. Journal of Consulting and Clinical Psychology, 61, 194–198.
Beck, A., & Steer, R. (1991). Manual for the beck scale for suicidal ideation. San Antonio, TX: Psychological Corporation.
Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the beck depression inventory (2nd ed.). San Antonio, TX: The Psychological Corporation.
Bohn, M. J., Babor, T. F., & Kranzler, H. R. (1995). The Alcohol Use Disorders Identification Test (AUDIT): Validation of a screening instrument for use in medical settings. Journal of Studies on Alcohol, 56, 423–432.
Brown, G. S., Lambert, M. J., Jones, E. R., & Minami, T. (2005). Identifying highly effective psychotherapists in a managed care environment. American Journal of Managed Care, 11, 513–520.
Brown, R. L., Leonard, T., Saunders, L. A., & Papasouliotis, O. (1997). A two-item screening test for alcohol and other drug problems. Journal of Family Practice, 44, 151–160.
Bryson, C. L., Au, D. H., Young, B., McDonell, M. B., & Fihn, S. D. (2007). A refill adherence algorithm for multiple short intervals to estimate refill compliance (ReComp). Medical Care, 45, 497–504.
Busch, A. B., Frank, R. G., & Lehman, A. F. (2004). The effect of a managed behavioral health carve-out on quality of care for medicaid patients diagnosed as having schizophrenia. Archives of General Psychiatry, 61, 442–448.
Bush, K., Kivlahan, D. R., McDonell, M. B., Fihn, S. D., & Bradley, K. A. (1998). The AUDIT alcohol consumption questions (AUDIT-C): An effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Archives of Internal Medicine, 158, 1789–1795.
Byerly, M. J., Nakonezny, P. A., Fisher, R., Magouirk, B., & Rush, A. J. (2006). An empirical evaluation of the Arizona sexual experience scale and a simple one-item screening test for assessing antipsychotic-related sexual dysfunction in outpatients with schizophrenia and schizoaffective disorder. Schizophrenia Research, 81, 311–316.
Byerly, M. J., Thompson, A., Carmody, T., Bugno, R., Erwin, T., Kashner, M., et al. (2007). Validity of electronically monitored medication adherence and conventional adherence measures in schizophrenia. Psychiatric Services, 58, 844–847.
Chakos, M., Lieberman, J., Hoffman, E., Bradford, D., & Sheitman, B. (2001). Effectiveness of second-generation antipsychotics in patients with treatment-resistant schizophrenia: A review and meta-analysis of randomized trials. American Journal of Psychiatry, 158, 518–526.
Chinman, M., Hassell, J., Magnabosco, J., Nowlin-Finch, N., Marusak, S., & Young, A. S. (2007). The feasibility of computerized patient self-assessment at mental health clinics. Administration and Policy In Mental Health, 34, 401–409.
Chinman, M., Young, A. S., Hassell, J., & Davidson, L. (2006). Toward the implementation of mental health consumer provider services. Journal of Behavioral Health Services and Research, 33, 176–195.
Chinman, M., Young, A. S., Schell, T., Hassell, J., & Mintz, J. (2004). Computer-assisted self-assessment in persons with severe mental illness. Journal of Clinical Psychiatry, 65, 1343–1351.
Clark, A., & Friedman, M. J. (1983). Factor structure and discriminant validity of the SCL-90 in a veteran psychiatric population. Journal of Personality Assessment, 47, 396–404.
Cole, J. D., & Kazarian, S. S. (1988). The Level of Expressed Emotion Scale: A new measure of expressed emotion. Journal of Clinical Psychology, 44, 392–397.
Covell, N. H., Jackson, C. T., Evans, A. C., & Essock, S. M. (2002). Antipsychotic prescribing practices in Connecticut’s public mental health system: Rates of changing medications and prescribing styles. Schizophrenia Bulletin, 28, 17–29.
Cramer, J. A. (1991). Overview of methods to measure and enhance patient compliance. In J. A. Cramer & B. Spilker (Eds.), Patient compliance in medical practice and clinical trials. New York, NY: Raven Press.
Crowther, R. E., Marshall, M., Bond, G. R., & Huxley, P. (2001). Helping people with severe mental illness to obtain work: Systematic review. BMJ, 322, 204–208.
Cuffel, B. J., Fischer, E. P., Owen, R. R., Jr., & Smith, G. R., Jr. (1997). An instrument for measurement of outcomes of care for schizophrenia. Issues in development and implementation. Evaluation and the Health Professions, 20, 96–108.
Dawe, S., Seinen, A., & Kavanagh, D. (2000). An examination of the utility of the AUDIT in people with schizophrenia. Journal of Studies on Alcohol, 61, 744–750.
Dawson, D. A., Grant, B. F., Stinson, F. S., & Zhou, Y. (2005). Effectiveness of the derived Alcohol Use Disorders Identification Test (AUDIT-C) in screening for alcohol use disorders and risk drinking in the US general population. Alcoholism, Clinical and Experimental Research, 29, 844–854.
Day, J. C., Bentall, R. P., Roberts, C., Randall, F., Rogers, A., Cattell, D., et al. (2005). Attitudes toward antipsychotic medication: The impact of clinical variables and relationships with health professionals. Archives of General Psychiatry, 62, 717–724.
Derogatis, L. R., & Melisaratos, N. (1983). The Brief Symptom Inventory: An introductory report. Psychological Medicine, 13, 595–605.
Eisen, S. V., Normand, S. L., Belanger, A. J., Spiro, A., III, & Esch, D. (2004). The Revised Behavior and Symptom Identification Scale (BASIS-R): Reliability and validity. Medical Care, 42, 1230–1241.
Fihn, S. D., McDonell, M. B., Diehr, P., Anderson, S. M., Bradley, K. A., Au, D. H., et al. (2004). Effects of sustained audit/feedback on self-reported health status of primary care patients. American Journal of Medicine, 116, 241–248.
Fitch, K., Bernstein, S., Aguilar, M. D., Burnand, B., LaCalle, J. R., Lazaro, P., et al. (2001). The RAND/UCLA appropriateness method user’s manual. Santa Monica, CA: RAND.
Francis, J., & Perlin, J. B. (2006). Improving performance through knowledge translation in the Veterans Health Administration. The Journal of Continuing Education in the Health Professions, 26, 63–71.
Gilmer, T. P., Dolder, C. R., Lacro, J. P., Folsom, D. P., Lindamer, L., Garcia, P., et al. (2004). Adherence to treatment with antipsychotic medication and health care costs among Medicaid beneficiaries with schizophrenia. American Journal of Psychiatry, 161, 692–699.
Gorall, D. M., Tiesel, J., & Olson, D. H. (2004). FACES-IV: Development and validation. Minneapolis, MN: Life Innovations.
Hays, R. D., Wells, K. B., Sherbourne, C. D., Rogers, W., & Spritzer, K. (1995). Functioning and well-being outcomes of patients with depression compared with chronic general medical illnesses. Archives of General Psychiatry, 52, 11–19.
Hoe, M., & Brekke, J. (2009). Testing the cross-ethnic construct validity of the Brief Symptom Inventory. Research on Social Work Practice, 19, 93–103.
Hogan, T. P., Awad, A. G., & Eastwood, R. (1983). A self-report scale predictive of drug compliance in schizophrenics: Reliability and discriminative validity. Psychological Medicine, 13, 177–183.
Institute of Medicine. (2006). Improving the quality of health care for mental and substance-use conditions: Quality chasm series. Washington, DC: National Academies Press.
Jha, A. K., Perlin, J. B., Kizer, K. W., & Dudley, R. A. (2003). Effect of the transformation of the Veterans Affairs Health Care System on the quality of care. New England Journal of Medicine, 348, 2218–2227.
Kane, J. M., & Marder, S. R. (2005). Schizophrenia: Somatic treatment. In B. J. Sadock & V. A. Sadock (Eds.), Kaplan & Sadock’s comprehensive textbook of psychiatry (pp. 1467–1476). New York, NY: Lippincott Williams & Wilkins.
Kavanagh, D. J., O’Halloran, P., Manicavasagar, V., Clark, D., Piatkowska, O., Tennant, C., et al. (1997). The Family Attitude Scale: Reliability and validity of a new scale for measuring the emotional climate of families. Psychiatry Research, 70, 185–195.
Kreisman, D. E., Simmens, S. J., & Joy, V. D. (1979). Rejecting the patient: Preliminary validation of a self-report scale. Schizophrenia Bulletin, 5, 220–222.
Kreyenbuhl, J., Buchanan, R. W., Dickerson, F. B., & Dixon, L. B. (2010). The schizophrenia patient outcomes research team (PORT): Updated treatment recommendations 2009. Schizophrenia Bulletin, 36, 94–103.
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613.
Kupersmith, J., Francis, J., Kerr, E., Krein, S., Pogach, L., Kolodner, R. M., et al. (2007). Advancing evidence-based care for diabetes: Lessons from the Veterans Health Administration. Health Affairs, 26, w156–w168.
Lacro, J. P., Dunn, L. B., Dolder, C. R., Leckband, S. G., & Jeste, D. V. (2002). Prevalence of and risk factors for medication nonadherence in patients with schizophrenia: A comprehensive review of recent literature. Journal of Clinical Psychiatry, 63, 892–909.
Law, M. R., Soumerai, S. B., Ross-Degnan, D., & Adams, A. S. (2008). A longitudinal study of medication nonadherence and hospitalization risk in schizophrenia. Journal of Clinical Psychiatry, 69, 47–53.
Lehman, A. F. (1988). A quality of life interview for the chronically mentally ill. Evaluation and Program Planning, 11, 51–62.
Lehman, A. F. (1999). Quality of care in mental health: The case of schizophrenia. Health Affairs, 18, 52–65.
Leslie, D. L., & Rosenheck, R. (2007). Eighth annual report on schizophrenia pharmacotherapy in VA (FY 2006). New Haven, CT: Northeast Program Evaluation Center, US Department of Veterans Affairs.
Lowe, B., Unutzer, J., Callahan, C. M., Perkins, A. J., & Kroenke, K. (2004). Monitoring depression treatment outcomes with the patient health questionnaire-9. Medical Care, 42, 1194–1201.
Marder, S. R., Essock, S. M., Miller, A. L., Buchanan, R. W., Casey, D. E., Davis, J. M., et al. (2004). Physical health monitoring of patients with schizophrenia. American Journal of Psychiatry, 161, 1334–1349.
Mayfield, D., McLeod, G., & Hall, P. (1974). The CAGE questionnaire: Validation of a new alcoholism screening instrument. American Journal of Psychiatry, 131, 1121–1123.
McEvoy, J. P., Lieberman, J. A., Stroup, T. S., Davis, S. M., Meltzer, H. Y., Rosenheck, R. A., et al. (2006). Effectiveness of clozapine versus olanzapine, quetiapine, and risperidone in patients with chronic schizophrenia who did not respond to prior atypical antipsychotic treatment. American Journal of Psychiatry, 163, 600–610.
McFarlane, W. R., Dushay, R. A., Deakins, S. M., Stastny, P., Lukens, E. P., Toran, J., et al. (2000). Employment outcomes in family-aided assertive community treatment. The American Journal of Orthopsychiatry, 70, 203–214.
McGahuey, C. A., Gelenberg, A. J., Laukes, C. A., Moreno, F. A., Delgado, P. L., McKnight, K. M., et al. (2000). The Arizona Sexual Experience Scale (ASEX): Reliability and validity. Journal of Sex and Marital Therapy, 26, 25–40.
McHugo, G. J., Bebout, R. R., Harris, M., Cleghorn, S., Herring, G., Xie, H., et al. (2004). A randomized controlled trial of integrated versus parallel housing services for homeless adults with severe mental illness. Schizophrenia Bulletin, 30, 969–982.
McHugo, G. J., Paskus, T. S., & Drake, R. E. (1993). Detection of alcoholism in schizophrenia using the MAST. Alcoholism, Clinical and Experimental Research, 17, 187–191.
McQueen, L., Mittman, B. S., & Demakis, J. G. (2004). Overview of the veterans health administration (VHA) Quality Enhancement Research Initiative (QUERI). Journal of the American Medical Informatics Association, 11, 339–343.
Miller, I. W., Epstein, N. B., Bishop, D. S., & Keitner, G. I. (1985). The McMaster Family Assessment Device: Reliability and validity. Journal of Marital and Family Therapy, 11, 345–356.
Miller, A. L., Hall, C. S., Buchanan, R. W., Buckley, P. F., Chiles, J. A., Conley, R. R., et al. (2004). The Texas Medication Algorithm Project antipsychotic algorithm for schizophrenia: 2003 update. Journal of Clinical Psychiatry, 65, 500–508.
Morisky, D. E., Green, L. W., & Levine, D. M. (1986). Concurrent and predictive validity of a self-reported measure of medication adherence. Medical Care, 24, 67–74.
Morlan, K. K., & Tan, S. Y. (1998). Comparison of the brief psychiatric rating scale and the brief symptom inventory. Journal of Clinical Psychology, 54, 885–894.
Newcomer, J. W. (2007). Antipsychotic medications: Metabolic and cardiovascular risk. Journal of Clinical Psychiatry, 68(Suppl 4), 8–13.
Newcomer, J. W., Campos, J. A., Marcus, R. N., Breder, C., Berman, R. M., Kerselaers, W., L’Italien, G. J., Nys, M., Carson, W. H., & McQuade, R. D. (2008). A multicenter, randomized, double-blind study of the effects of aripiprazole in overweight subjects with schizophrenia or schizoaffective disorder switched from olanzapine. Journal of Clinical Psychiatry, 69, 1046–1056.
Niv, N., Cohen, A. N., Mintz, J., Ventura, J., & Young, A. S. (2007a). The validity of using patient self-report to assess psychotic symptoms in schizophrenia. Schizophrenia Research, 90, 245–250.
Niv, N., Cohen, A. N., Sullivan, G., & Young, A. S. (2007b). The MIRECC version of the Global Assessment of Functioning Scale: Reliability and validity. Psychiatric Services, 58, 529–535.
Osterberg, L., & Blaschke, T. (2005). Adherence to medication. New England Journal of Medicine, 353, 487–497.
Pfeffer, C. R., Newcorn, J., Kaplan, G., Mizruchi, M. S., & Plutchik, R. (1988). Suicidal behavior in adolescent psychiatric inpatients. Journal of the American Academy of Child and Adolescent Psychiatry, 27, 357–361.
Pilling, S., Bebbington, P., Kuipers, E., Garety, P., Geddes, J., Orbach, G., et al. (2002). Psychological treatments in schizophrenia: I. Meta-analysis of family intervention and cognitive behaviour therapy. Psychological Medicine, 32, 763–782.
Pinninti, N., Steer, R. A., Rissmiller, D. J., Nelson, S., & Beck, A. T. (2002). Use of the Beck Scale for suicide ideation with psychiatric inpatients diagnosed with schizophrenia, schizoaffective, or bipolar disorders. Behaviour Research and Therapy, 40, 1071–1079.
Pitschel-Walz, G., Leucht, S., Bauml, J., Kissling, W., & Engel, R. R. (2001). The effect of family interventions on relapse and rehospitalization in schizophrenia—A meta-analysis. Schizophrenia Bulletin, 27, 73–92.
President’s New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. http://www.mentalhealthcommission.gov/reports/reports.htm. Accessed 7 May 2009.
Preston, N. J., & Harrison, T. J. (2003). The brief symptom inventory and the positive and negative syndrome scale: Discriminate validity between a self-reported and observational measure of psychopathology. Comprehensive Psychiatry, 44, 220–226.
Rauter, U. K., Leonard, C. E., & Swett, C. P. (1996). SCL-90-R factor structure in an acute, involuntary, adult psychiatric inpatient sample. Journal of Clinical Psychology, 52, 625–629.
Resnick, S. G., & Bond, G. R. (2001). The Indiana Job Satisfaction Scale: Job satisfaction in vocational rehabilitation for people with severe mental illness. Psychiatric Rehabilitation Journal, 25, 12–19.
Resnick, S. G., Rosenheck, R. A., Dixon, L., & Lehman, A. F. (2005). Correlates of family contact with the mental health system: Allocation of a scarce resource. Mental Health Services Research, 7, 113–121.
Rubenstein, L. V., Mittman, B. S., Yano, E. M., & Mulrow, C. D. (2000). From understanding health care provider behavior to improving health care: The QUERI framework for quality improvement. Quality Enhancement Research Initiative. Medical Care, 38, I.129–I.141.
Russell, M. (1994). New assessment tools for risk drinking during pregnancy: T-ACE, TWEAK, and others. Alcohol Health & Research World, 18, 55–61.
Russell, M., & Bigler, L. (1979). Screening for alcohol-related problems in an outpatient obstetric-gynecologic clinic. American Journal of Obstetrics and Gynecology, 134, 4–12.
Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the alcohol use disorders identification test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption—II. Addiction, 88, 791–804.
Schooler, N., Hogarty, G., & Weissman, M. M. (1979). Social Adjustment Scale II (SAS-II). In W. A. Hargreaves, C. C. Attkisson, & J. E. Sorenson (Eds.), Resource materials for community health program evaluators. Washington, DC: US Department of Health, Education and Welfare.
Selzer, M. L. (1971). The Michigan alcoholism screening test: The quest for a new diagnostic instrument. American Journal of Psychiatry, 127, 1653–1658.
Skinner, H. A. (1982). The drug abuse screening test. Addictive Behaviors, 7, 363–371.
Staley, D., & El-Guebaly, N. (1990). Psychometric properties of the drug abuse screening test in a psychiatric patient population. Addictive Behaviors, 15, 257–264.
Stone, E. G., Morton, S. C., Hulscher, M. E., Maglione, M. A., Roth, E. A., Grimshaw, J. M., et al. (2002). Interventions that increase use of adult immunization and cancer screening services: A meta-analysis. Annals of Internal Medicine, 136, 641–651.
Toland, A. M., & Moss, H. B. (1989). Identification of the alcoholic schizophrenic: Use of clinical laboratory tests and the MAST. Journal of Studies on Alcohol, 50, 49–53.
Valenstein, M., Copeland, L. A., Blow, F. C., McCarthy, J. F., Zeber, J. E., Gillon, L., et al. (2002). Pharmacy data identify poorly adherent patients with schizophrenia at increased risk for admission. Medical Care, 40, 630–639.
Ventura, J., Green, M. F., Shaner, A., & Liberman, R. P. (1993). Training and quality assurance with the Brief Psychiatric Rating Scale: “The drift busters”. International Journal of Methods in Psychiatric Research, 3, 221–244.
Veterans Health Administration. (2003). VA technology assessment program (VATAP) Short report: Outcomes measurement in schizophrenia. Number 2 in a series: Outcomes measurement in VHA mental health services. http://www.va.gov/VATAP/pubs/mho.htm. Accessed 11 May 2009.
Weiden, P. J., Kozma, C., Grogg, A., & Locklear, J. (2004). Partial compliance and risk of rehospitalization among California Medicaid patients with schizophrenia. Psychiatric Services, 55, 886–891.
WHO ASSIST Working Group. (2002). The alcohol, smoking and substance involvement screening test (ASSIST): Development, reliability and feasibility. Addiction, 97, 1183–1194.
Wolford, G. L., Rosenberg, S. D., Drake, R. E., et al. (1999). Evaluation of methods for detecting substance use disorder in persons with severe mental illness. Psychol Addict Behav, 13, 313–326.
Young, A. S. (1999). Evaluating and improving the appropriateness of treatment for schizophrenia. Harvard Review of Psychiatry, 7, 114–118.
Young, A. S. (2007). Routinely measuring schizophrenia treatment quality—Shortcomings with medical records. Medical Care, 45, 104. (author reply 104–105).
Young, A. S., & Magnabosco, J. L. (2004). Services for adults with mental illness. In B. L. Levin, J. Petrila, & K. D. Hennessy (Eds.), Mental health services: A public health perspective (pp. 177–208). New York, NY: Oxford University Press.
Young, A. S., Mintz, J., & Cohen, A. N. (2004). Using information systems to improve care for persons with schizophrenia. Psychiatric Services, 55, 253–255.
Young, A. S., Sullivan, G., Burnam, M. A., & Brook, R. H. (1998). Measuring the quality of outpatient treatment for schizophrenia. Archives of General Psychiatry, 55, 611–617.
Zygmunt, A., Olfson, M., Boyer, C. A., & Mechanic, D. (2002). Interventions to improve medication adherence in schizophrenia. American Journal of Psychiatry, 159, 1653–1664.
Acknowledgments
This work was supported by the Department of Veterans Affairs through the Mental Health Quality Enhancement Research Initiative (QUERI), the Health Services Research & Development Service QUERI (MNH 03-297), and the Desert Pacific Mental Illness Research, Education and Clinical Centers (MIRECC); and by the UCLA-RAND NIMH Partnered Research Center for Quality Care (MH082760). Any opinions expressed are only those of the authors and do not necessarily represent the views of any affiliated institutions. We thank the expert panel for this project, which consisted of: Alexander Young (chair), Richard Owen (co-chair), Lisa Dixon, Ellen Fischer, Teresa Hudson, Miklos Losonczy, Stephen Marder, Marcia Valenstein, and William Van Stone. We thank the members of the Mental Health QUERI Schizophrenia Outcomes Workgroup, which includes the authors of this paper, plus Teresa Hudson, Amy Kilbourne, Miklos Losonczy, David Smelson, and William Van Stone. We also thank Gary Bond, Shirley Glynn, Daniel Kivlahan, and Kim Mueser for contributions to the literature review.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Young, A.S., Niv, N., Chinman, M. et al. Routine Outcomes Monitoring to Support Improving Care for Schizophrenia: Report from the VA Mental Health QUERI. Community Ment Health J 47, 123–135 (2011). https://doi.org/10.1007/s10597-010-9328-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10597-010-9328-y