Abstract
This article describes the importance and role of 4 stages of formative evaluation in our growing understanding of how to implement research findings into practice in order to improve the quality of clinical care. It reviews limitations of traditional approaches to implementation research and presents a rationale for new thinking and use of new methods. Developmental, implementation-focused, progress-focused, and interpretive evaluations are then defined and illustrated with examples from Veterans Health Administration Quality Enhancement Research Initiative projects. This article also provides methodologic details and highlights challenges encountered in actualizing formative evaluation within implementation research.
Similar content being viewed by others
References
QUERI. HSRD special solicitation for service directed projects (SDP) on implementation of research into practice (posted 2003). Available at: http://www.hsrd.research.va.gov/for_researchers/funding/solicitations/. Accessed August 10, 2005.
Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality enhancement research initiative (QUERI): a collaboration between research and clinical practice. Med Care. 2000;38(suppl 1):117–25.
McQueen L, Mittman BS, Demakis JG. Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc. 2004;11:339–43.
QUERI. QUERI program description (posted July 2003). Available at: http://www1.va.gov/hsrd/queri/. Accessed August 10, 2005.
Cook TD, Campbell DT. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company; 1979.
Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin Company; 1963.
Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.
Dzewaltowski D, Estabrooks P, Glasgow R, Klesges L. Workgoup to evaluate and enhance the reach and dissemination of health promotion interventions (RE-AIM). Available at: http://www.re-aim.org/2003/whoweare.html. Accessed July 9, 2005.
Glasgow RE, Lichtenstein E, Marcus A. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Pub Health. 2003; 93:1261–7.
Stead M, Hastings G, Eadie D. The challenge of evaluating complex interventions: a framework for evaluating media advocacy. Health Educ Res. 2002;17:351–64.
Zapka J, Goins KV, Pbert L, Ockene JK. Translating efficacy research to effectiveness studies in practice: lessons from research to promote smoking cessation in community health centers. Health Promot Pract. 2004;5:245–55.
Ovretveit J, Gustafson D. Using research to inform quality programmes. BMJ. 2003;326:759–61.
Hulscher M, Laurant M, Grol R. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2002;12:40–6.
Solberg LI. Guideline implementation: what the literature doesn’t tell us. Jt Comm J Qual Improv. 2000;26:525–37.
Mittman B. Creating the evidence base for quality improvement collaboratives. Ann Int Med. 2004;140:897–901.
Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17:204–20.
Rossi P, Freeman H. Evaluation: A Systematic Approach. Newbury Park: Sage Publications; 1993.
Bhola HS. Evaluating “Literacy for Development” Projects, Programs and Campaigns: Evaluation Planning, Design and Implementation, and Utilization of Evaluation Results. Hamburg, Germany: UNESCO Institute for Education; DSE (German Foundation for International Development), xii; 1990.
Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.
Altman DG. A framework for evaluating community-based heart disease prevention programs. Soc Sci Med. 1986;22:479–87.
Havas S, Anliker J, Damron D, Feldman R, Langenberg P. Uses of process evaluation in the Maryland WIC 5-a-day promotion program. Health Educ Behav. 2000;27:254–63.
Evans RI, Raines BE, Owen AE. Formative evaluation in school-based health promotion investigations. Prev Med. 1989;18:229–34.
Patton MQ. Utilization-Focused Evaluation: The New Century Text. 3rd edn. Thousand Oaks, CA: Sage Publications; 1997.
Patton MQ. Qualitative Research & Evaluation Methods. 3rd edn. Newbury Park, CA: Sage Publications; 2001.
Walshe K, Freeman T. Effectiveness of quality improvement: learning from evaluations. Qual Saf Health Care. 2002;11:85–7.
Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.
Forsetlund L, Talseth KO, Bradley P, Nordheim L, Bjorndal A. Many a slip between cup and lip. Process evaluation of a program to promote and support evidence-based public health practice. Eval Rev. 2003;27:179–209.
Guide for Implementing Evidence-Based Clinical Practice and Conducting Implementation Research. Available at: http://www1.va.gov/hsrd/queri/implementation. Accessed August 10, 2005.
van Bokhoven MA, Kok G, van der Weijden T. Designing a quality improvement intervention: a systematic approach. Qual Saf Health Care. 2003;3:215–20.
Rogers EM. Diffusion of Innovations. 4th edn. New York: Free Press; 1995.
Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.
Sales A. Organizational readiness for evidence-based health care interventions. Available at: www.measurementsexperts.org/instrument;instrument_reviews.asp?detail=53. Accessed August 10, 2005.
Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.
Brown K, Gerhardt M. Formative evaluation: an integrative practice model and case study. Personnel Psychol. 2002;55:951f.
Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.
Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.
Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.
Rycroft-Malone J, Kitson A, Harvey G, et al. Ingredients for change: revisiting a conceptual framework. Qual Saf Health Care. 2002;11:174–80.
McCormack B, Kitson A, Harvey G, et al. Getting evidence into practice: the meaning of ‘context’. J Adv Nurs. 2002;38:94–104.
Bradley E, Holmboe E, Mattera J, et al. A qualitative study of increasing β-blocker use after myocardial infarction: why do some hospitals succeed? J Am Med Assoc. 2001;285:2604–11.
Kimmel B. Bridging the gap between knowledge and practice — the veterans administration pathway, Newsletter of the National Council of the University Research Administrators 2003-4;35.
Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.
Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomized controlled trial be? BMJ. 2004;328:1561–3.
Kirchhoff K, Dille C. Issues in intervention research: maintaining integrity. Appl Nurs Res. 1994;7:32–8.
Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurs Res. 2004;53:63–6.
Boyd NR, Windsor RA. A formative evaluation in maternal and child health practice: the partners for life nutrition education program for pregnant women. Matern Child Health J. 2003;7:137–43.
Stetler C, Creer E, Effken J. Evaluating a redesign program: challenges and opportunities. In Kelly K., ed. Series on Nursing Administration, Vol. 8. St. Louis: Mosby Year Book; 1996.
Effken E, Stetler C. Impact of organizational redesign. J Nurs Admin. 1997;27:23–32.
Krumholz H, Herrin J. Quality improvement: the need is there but so are the challenges. Am J Med. 2000;109:501–3.
Willenbring ML, Hagedorn H. Implementing evidence-based practices in opioid agonist therapy clinics. In: Roberts A., Yeager K., eds. Evidence-Based Practice Manual: Research and Outcome Measures in Health and Human Services 2004. New York: Oxford University Press; 2004:340–7.
Willenbring ML, Hagedorn H, Poster AC, Kenny M. Variations in evidence-based clinical practices in nine United States Veterans Administration opioid agonist therapy clinics. Drug Alcohol Dependence. 2004;75:97–106.
LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.
La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.
Nazareth I, Freemantle N, Duggan C, Mason J, Haines A. Evaluation of a complex intervention for changing professional behaviour: the evidence based out reach (EBOR) Trial. J Health Serv Res Policy. 2002;7:230–8.
Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use. J Biomed Inform. 2003;36:218–27.
Sanson-Fisher RW, Grimshaw JM, Eccles MP. The science of changing providers’ behaviour: the missing link in evidence-based practice. Med J Aust. 2004;180:205–6.
Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME—PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice. BMC Health Serv Res. 2003;3:22.
Yin R. Case Study Research: Design and Methods. Thousand Oaks: Sage; 1994.
Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9.
Sharp ND, Pineros SL, Hsu C, Starks H, Sales AE. A qualitative study to identify barriers and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) Northwest Network. Worldviews Evidence-Based Nurs. 2004;1:129–39.
Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.
Litchman J, Roumanis S, Radford M, et al. Can practice guidelines be transported effectively to different settings? Results from a multi-center interventional study. J Comm J Qual Improv. 2001;27:42–53.
Author information
Authors and Affiliations
Corresponding author
Additional information
An earlier version of this article was included in materials to support the State of the Art of Implementation Conference. The conference was conducted by the Department of Veterans Affairs, Office of Research and Development, Health Services Research and Development Service, and was held in Washington, DC, from August 30 to September 1 2004.
The work reported here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
Rights and permissions
About this article
Cite this article
Stetler, C.B., Legro, M.W., Wallace, C.M. et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 21 (Suppl 2), S1–S8 (2006). https://doi.org/10.1007/s11606-006-0267-9
Issue Date:
DOI: https://doi.org/10.1007/s11606-006-0267-9