INTRODUCTION

Improving care for chronic illness, across many conditions, has proven to be very challenging1,2 in large part because of the conceptual and organizational changes required by a paradigm shift from acute treatment to long-term management.3,4 While there have been some successes in improving care in primary care,5,6 efforts in specialty mental health settings have been fewer and have not yielded substantial changes in long-term outcomes.7 In care for patients with severe mental illness (SMI), providers often have not adopted practices that improve outcomes.8,9

Implementing changes in clinical care is notoriously difficult.10,11 Passive approaches such as education may improve knowledge, but do not tend to induce change or improve care.12 Moreover, clinical interventions are often instituted without a baseline understanding of the context in which the change is to occur. Greater attention to existing organizational infrastructures could facilitate higher quality implementation.13,14 In particular, assessing organizational readiness for change can contribute to the tailoring of innovative adoption strategies to local structure and context.15 This paper describes a baseline assessment of readiness for change in EQUIP-2 (Enhancing QUality in Psychosis), a controlled trial of an intervention to improve mental health outpatient care for schizophrenia.13 With a recent upsurge in large-scale efforts to improve adoption of evidence-based practices in mental health (e.g., the Veterans Health Administration Uniform Mental Health Services Package)16 and chronic conditions in general, detailed case studies are necessary as examples of how to facilitate quality improvement. The goal of this paper is to provide a brief illustration of the ways in which the formative evaluation data were analyzed and used to address organizational issues prior to the implementation of quality improvement efforts, in order to enhance the probability that such efforts would be implemented favorably.

METHODS

The goals of EQUIP-2 were to increase use of the evidence-based practices of weight reduction (i.e., wellness) and supported employment (i.e., placement and support) for individuals with SMI. To assess the context and readiness for uptake of these services at the four clinics in the intervention, administrators and staff members completed an organizational readiness survey, and a subset also completed a qualitative interview.

Clinic Sites and Sampling

VA health care is managed within 21 national Veterans Integrated Service Networks (VISNs) of medical centers, veteran centers, and outpatient clinics offering primary and specialized care. Selected specialty mental health clinics in four VISNs are participating in EQUIP.

Of the 21 administrators eligible to participate at baseline, 20 (95%) consented; of the 95 eligible staff, 75 (79%) consented. Of those who consented, 16 (80%) administrators and 43 (57%) staff completed the organizational survey.

For the semi-structured interview, stratified purposeful sampling24 was used to select key stakeholders in the clinics who could provide their perspectives on organizational culture, readiness for change, and experience with quality improvement projects. All administrators (n = 10) participated in the interview, and all clinicians (n = 28) who had direct contact with patients with schizophrenia participated.

Procedures and Measures

All consented administrators and staff were asked in person after consenting to complete an online survey. Individuals who did not complete the survey in a timely manner were prompted via phone, visit, or e-mail. The 20-min electronic survey included several scales from the Organizational Readiness for Change (ORC) instrument.18 The ORC, originally developed for substance abuse treatment facilities, assesses five domains: motivation for change, resources, staff attributes, organizational climate, and training exposure and utilization. To tailor the ORC to VA specialty mental health clinics and minimize participant burden, minor adjustments were made to the wording, and two domains (resources, training exposure and utilization) were not included. These domains would have required extensive changes in order to make them relevant to mental health and VA settings. Ten subscales were utilized: program needs, training needs, and pressures for change in the “motivation for change” domain; growth and adaptability in the “staff attributes” domain; and mission, cohesion, autonomy, communication, and change in the “organizational climate” domain. Responses on each item are arranged on a Likert scale from 1 = strongly disagree to 5 = strongly agree.

Individuals who participated in the semi-structured interview component of the assessment were interviewed by the project coordinator at each site or by the lead author. The interview protocol was developed by the three authors and addressed perceptions of the needs of patients with schizophrenia, knowledge about the care targets, organizational culture in terms of decision-making and change, and experience with quality improvement projects. Interviews lasted 20–30 min on average. All interviews were digitally recorded and professionally transcribed.

Data Analysis

To analyze the ORC, subscale scores were obtained by summing responses to the items (reversing scores when necessary), dividing the sum by the number of items included (yielding an average), and multiplying by 10 in order to rescale final scores so they ranged from 10 to 50. Means for each scale were examined across sites and in comparison to normative data. In addition, 25th and 75th percentiles were calculated for further comparisons.

Subscale scores above 30 indicate areas of strength, which translates to the idea of “readiness” in that the respondents generally agree that the organization has the attribute in a given subscale. Scores below 30 (which are unusual) indicate areas of weakness that might need attention prior to change efforts. The standard deviations (SDs) of the ORC scores are also illuminating in that they indicate the level of consensus on any given subscale; SDs above 9 indicate considerable variability in responses and prompt questions about why the subscale topic is perceived differently across respondents (D. Simpson, personal communication, February 10, 2009).

Analysis of the interview data was conducted using Atlas.ti. Transcripts were coded using content analysis with codes deductively derived from the interview guide topics. Results within each interview topic were examined to determine areas that required subcoding. Results were reviewed collectively by the three authors, and narratives that speak to organizational predisposition are reported below.

RESULTS

Provider Background

Nineteen men and 24 women completed the staff version of the ORC. Approximately one-quarter (28%) of these respondents were psychiatrists, and the remaining respondents were other clinicians. Half (51%) had worked at the VA for 11 or more years, 9% for 5–10 years, and 40% for 4 years or less.

Organizational Readiness for Change

As depicted in Figure 1, scores on the staff ORC generally fell within the 25th and 75th percentile norms for the instrument, indicating moderately favorable conditions for change in terms of motivation for change, staff attributes, and organizational climate.

Figure 1
figure 1

Staff ORC scores.

Full size image

As depicted in Table 1, scores were consistent, with few high standard deviations. The staff attributes domain seemed to be the area of greatest strength across sites, with staff reporting high levels of agreement with regard to staff growth and adaptability. The program needs subscale (see Fig. 2) had the most inconsistency, with SDs of 9 or more in three of the four sites.

Figure 2
figure 2

ORC program needs subscale.

Full size image
Table 1 Staff Organizational Readiness for Change Subscale Mean Scores and Standard Deviation by Clinic
Full size table

Scores were generally moderate for site A, indicating good structure and functioning, with high consistency across respondents, especially in the organizational climate scale. Site B scores tended to have higher standard deviations than site A, with high variability in scores on program needs and cohesion, communication, and change, indicating less consensus in these areas. Site C also displayed high variability in perceptions of program needs, but was otherwise consistent. Sites B and C had the lowest mean scores on program needs, and site C had the lowest scores for training needs, pressures for change, autonomy, mission, communication, and change. All scores were moderate and consistent for site D, which reported the lowest adaptability and cohesion.

Key Stakeholder Interviews

Interviews were analyzed with reference to the ORC findings to provide more detail about areas of organizational strengths and weaknesses. Site A had particularly high and consistent scores in the staff attributes domain, indicating strength with regard to opportunities for professional growth and adaptability to change; it also had a high autonomy score. This autonomy was evident in the interviews, where respondents described being able to make changes as necessary and to manage their own workloads and activities. Site B’s particularly high adaptability and cohesion scores were seen as strengths; staff also indicated little need for additional training. These findings were consistent with the interviews, where respondents described “team efforts.” One respondent stated, “We don’t always get somebody higher up for advice…In any area where we could do better, we do it.” Site C had a consistent and low change score. This subscale emphasizes an attitude of trying new things (e.g., “You are encouraged here to try new and different techniques.”). A lower score indicates only moderate agreement with that statement, and therefore a sense of little autonomy over initiating change. Accordingly, this issue was investigated in the interviews at this site, and we found that staff perceived “constant changes” to be occurring, but mostly from the top down, i.e., changes that were being imposed on them.

Site D interviews were explored for discussions related to greater reported program and training needs than the other sites, and lower scores regarding adaptability and cohesion. There was a sense in the interviews that this site was, as one respondent stated, “moving toward making changes,” but that there had been, as another stated, “a very large amount of inertia.” Interviews revealed that this clinic was somewhat fragmented at baseline; some suggested that decisions were made behind closed doors and then imparted to staff, and others described how small changes could be made by their clinical team, but “major changes” were not possible.

DISCUSSION

This study of organizational readiness provides insight into the complexity of engaging in organizational change18 and evidence-based quality improvement. Each intervention site had different strengths and weaknesses, but these would likely not have been revealed without the triangulated ORC and semi-structured interview data.

Implementation strategies at each site, in the site-level controlled trial, were tailored based on the findings from the baseline assessment. Sites A and B were both ready for change, so site A served as the lead implementation site, given its adaptability and emphasis on professional growth and trying new techniques. The site developed testimonials for the other sites on the successes of their implementation strategies. At site B, a strong sense of cohesion was built on by having the opinion leader talk to the staff as a whole about implementation and having frequent meetings with all staff where the project was discussed and problems were addressed.

Sites C and D required more tailoring of the context of the intervention to give it its best chance of acceptance. In site C, training needs were addressed by heightening awareness of gaps in care through use of opinion leaders and educational programs; changing leadership was compensated for by keeping the remaining team the same and by maintaining a consistent message about the mission and goals of the intervention targets; autonomy was encouraged by allowing clinicians to design aspects of their interventions (wherever possible given the evidence base) around the VISN-established care targets.

In site D, adaptability issues were addressed by bringing the opinion leader and key staff on board first and providing direct guidance on the specifics of implementing the intervention; efforts were made to link study goals with administration’s goals in order to address mission issues. Cohesion was enhanced by building teams within the clinic, and as with site C, autonomy was encouraged by allowing clinicians to design the implementation of their interventions.

Overall, intervention sites were moderately ready to change, and may have been identified by VISN leadership because they were perceived to be ready to change (sites A and B) and/or in need of change but also in need of additional support (sites C and D). The latter sites required more flexibility from the coordinating site as to how to implement the intervention, whereas the former sites could implement the interventions closer to the original intent. As noted, half (51%) of the intervention sites’ respondents had worked at the VA for 11 years or more, and the response rate was relatively low among staff (57%). This could suggest that intervention site staff in general were entrenched in their routines and/or perhaps had seen change efforts that had not worked or that had not been sustained and were skeptical about engaging in the project. The low response rate may indicate resistance on the part of the intervention sites to the presence of the project. Despite numerous targeted efforts to increase the response rate, those who consented but refused to complete the instrument were dogmatic in their refusal, usually due to the perception that the instrument would take too much time to complete.

The program needs subscale reflected the most variability in responses, with high standard deviations in three of the four sites. This variability may be due to differences in educational and professional backgrounds such that respondents had differing emphases on the areas in which their clinics needed additional guidance. Training needs, program needs, and change were the domains with generally lower scores, and staff attributes scores remained high. The latter finding could be because these questions focused on perceptions of one’s own professional motivation (growth) and personality style (adaptability), which might be more proximal and stable than most of the other subscales, which call for evaluation of the clinic and therefore might be more distal and subject to fluctuation depending on daily work dynamics.

Next Steps in Organizational Assessment and Implementation

EQUIP-2 falls within a burgeoning number of studies that assess barriers and facilitators to implementation and then focus on refining implementation strategies during intervention.19 Staff expectations, perceptions, and attitudes may encourage or inhibit adoption of evidence-based practices,20 and are thus critical to investigate and address at the outset of a quality improvement project through formative evaluation methods.21 Also critical to investigate are the larger systemic and structural factors that may affect attitudes towards and experiences of change. For example, in a system such as the VA, change from the “top” is expected by staff, and typically the changes are not voluntary or optional.

Key stakeholder perspectives will be gathered mid- and post-intervention, and organizational readiness will be assessed again post-intervention to see if readiness changed over time. This long-term perspective on adoption of innovation has been recommended by others who have noted that change toward evidence-based care in mental health care is a slow and uneven process, warranting a longitudinal perspective.22,23 Issues such as organizational vision and commitment may affect the long-term sustainability of innovations, so ongoing assessment of these issues could be critical.24 With a longitudinal perspective, this study will be able to explore associations between patient outcomes and organizational readiness, as has been demonstrated by others.25 Whitley and colleagues26 suggest that implementation of illness management and recovery requires strong leadership, an organizational culture that embraces innovation, effective training, and committed staff. Lin and colleagues1 suggest that employees’ perceptions of an organization’s orientation, activities, and support of quality management are associated with their perceptions of whether implementation of quality improvement activities will lead to improved patient outcomes.

Limitations

This study has limitations in that the sample sizes are not large and there is variability across sites, so generalizability is limited. Data from administrators was particularly limited, so insight is not provided here as to the ways in which administrators and staff may differ in their perceptions of organizational readiness, as operationalized by the ORC and as explored in the interviews. Additionally, taking the average survey response from a group of employees is a limited way in which to assess organizational readiness, at least in part because employees likely differ in their awareness of and contribution to readiness. This study was not designed to test variable contributions to readiness, but an investigation that tests this idea would be clinically and empirically valuable.

Conclusion

One of the main points of this quality improvement effort was the need to acknowledge from the outset that clinical settings, though they may be similar in many ways, are never completely equivalent despite best efforts to have them equivalent from an empirical standpoint. Accordingly, we designed the study to explore the ways in which the sites were not equivalent, i.e., to assess in some depth each site’s organizational climate, with more emphasis on the intervention sites because we wanted to maximize uptake of the intervention in order to improve quality of care.

Organizational change, though difficult to achieve, can occur. Adoption of evidence-based care in specialty mental health is critical for the improvement of patient outcomes.27 The onus is on implementation researchers to continue to identify factors that facilitate successful adoption of appropriate and effective clinical practices. One step toward the identification of these factors is thorough assessment of organizational readiness for change, so subsequent intervention efforts are more carefully attuned to the strengths and barriers present in each site. Ideally, with locally tailored implementation strategies, adoption of evidence-based practices will increasingly become the norm, and patients seeking services will receive the care that they need in order to have optimal health outcomes.