Skip to main content
Log in

The role of formative evaluation in implementation research and the QUERI experience

  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

This article describes the importance and role of 4 stages of formative evaluation in our growing understanding of how to implement research findings into practice in order to improve the quality of clinical care. It reviews limitations of traditional approaches to implementation research and presents a rationale for new thinking and use of new methods. Developmental, implementation-focused, progress-focused, and interpretive evaluations are then defined and illustrated with examples from Veterans Health Administration Quality Enhancement Research Initiative projects. This article also provides methodologic details and highlights challenges encountered in actualizing formative evaluation within implementation research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. QUERI. HSRD special solicitation for service directed projects (SDP) on implementation of research into practice (posted 2003). Available at: http://www.hsrd.research.va.gov/for_researchers/funding/solicitations/. Accessed August 10, 2005.

  2. Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality enhancement research initiative (QUERI): a collaboration between research and clinical practice. Med Care. 2000;38(suppl 1):117–25.

    Google Scholar 

  3. McQueen L, Mittman BS, Demakis JG. Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc. 2004;11:339–43.

    Article  PubMed  Google Scholar 

  4. QUERI. QUERI program description (posted July 2003). Available at: http://www1.va.gov/hsrd/queri/. Accessed August 10, 2005.

  5. Cook TD, Campbell DT. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company; 1979.

    Google Scholar 

  6. Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin Company; 1963.

    Google Scholar 

  7. Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.

    Google Scholar 

  8. Dzewaltowski D, Estabrooks P, Glasgow R, Klesges L. Workgoup to evaluate and enhance the reach and dissemination of health promotion interventions (RE-AIM). Available at: http://www.re-aim.org/2003/whoweare.html. Accessed July 9, 2005.

  9. Glasgow RE, Lichtenstein E, Marcus A. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Pub Health. 2003; 93:1261–7.

    Article  Google Scholar 

  10. Stead M, Hastings G, Eadie D. The challenge of evaluating complex interventions: a framework for evaluating media advocacy. Health Educ Res. 2002;17:351–64.

    Article  PubMed  Google Scholar 

  11. Zapka J, Goins KV, Pbert L, Ockene JK. Translating efficacy research to effectiveness studies in practice: lessons from research to promote smoking cessation in community health centers. Health Promot Pract. 2004;5:245–55.

    Article  PubMed  Google Scholar 

  12. Ovretveit J, Gustafson D. Using research to inform quality programmes. BMJ. 2003;326:759–61.

    Article  PubMed  Google Scholar 

  13. Hulscher M, Laurant M, Grol R. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2002;12:40–6.

    Article  Google Scholar 

  14. Solberg LI. Guideline implementation: what the literature doesn’t tell us. Jt Comm J Qual Improv. 2000;26:525–37.

    CAS  PubMed  Google Scholar 

  15. Mittman B. Creating the evidence base for quality improvement collaboratives. Ann Int Med. 2004;140:897–901.

    PubMed  Google Scholar 

  16. Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17:204–20.

    Article  Google Scholar 

  17. Rossi P, Freeman H. Evaluation: A Systematic Approach. Newbury Park: Sage Publications; 1993.

    Google Scholar 

  18. Bhola HS. Evaluating “Literacy for Development” Projects, Programs and Campaigns: Evaluation Planning, Design and Implementation, and Utilization of Evaluation Results. Hamburg, Germany: UNESCO Institute for Education; DSE (German Foundation for International Development), xii; 1990.

    Google Scholar 

  19. Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.

    Google Scholar 

  20. Altman DG. A framework for evaluating community-based heart disease prevention programs. Soc Sci Med. 1986;22:479–87.

    Article  CAS  PubMed  Google Scholar 

  21. Havas S, Anliker J, Damron D, Feldman R, Langenberg P. Uses of process evaluation in the Maryland WIC 5-a-day promotion program. Health Educ Behav. 2000;27:254–63.

    Article  CAS  PubMed  Google Scholar 

  22. Evans RI, Raines BE, Owen AE. Formative evaluation in school-based health promotion investigations. Prev Med. 1989;18:229–34.

    Article  CAS  PubMed  Google Scholar 

  23. Patton MQ. Utilization-Focused Evaluation: The New Century Text. 3rd edn. Thousand Oaks, CA: Sage Publications; 1997.

    Google Scholar 

  24. Patton MQ. Qualitative Research & Evaluation Methods. 3rd edn. Newbury Park, CA: Sage Publications; 2001.

    Google Scholar 

  25. Walshe K, Freeman T. Effectiveness of quality improvement: learning from evaluations. Qual Saf Health Care. 2002;11:85–7.

    Article  CAS  PubMed  Google Scholar 

  26. Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.

    Google Scholar 

  27. Forsetlund L, Talseth KO, Bradley P, Nordheim L, Bjorndal A. Many a slip between cup and lip. Process evaluation of a program to promote and support evidence-based public health practice. Eval Rev. 2003;27:179–209.

    Article  PubMed  Google Scholar 

  28. Guide for Implementing Evidence-Based Clinical Practice and Conducting Implementation Research. Available at: http://www1.va.gov/hsrd/queri/implementation. Accessed August 10, 2005.

  29. van Bokhoven MA, Kok G, van der Weijden T. Designing a quality improvement intervention: a systematic approach. Qual Saf Health Care. 2003;3:215–20.

    Article  Google Scholar 

  30. Rogers EM. Diffusion of Innovations. 4th edn. New York: Free Press; 1995.

    Google Scholar 

  31. Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.

    CAS  Google Scholar 

  32. Sales A. Organizational readiness for evidence-based health care interventions. Available at: www.measurementsexperts.org/instrument;instrument_reviews.asp?detail=53. Accessed August 10, 2005.

  33. Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.

    PubMed  Google Scholar 

  34. Brown K, Gerhardt M. Formative evaluation: an integrative practice model and case study. Personnel Psychol. 2002;55:951f.

  35. Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.

    Google Scholar 

  36. Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.

    PubMed  Google Scholar 

  37. Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.

    Google Scholar 

  38. Rycroft-Malone J, Kitson A, Harvey G, et al. Ingredients for change: revisiting a conceptual framework. Qual Saf Health Care. 2002;11:174–80.

    Article  CAS  PubMed  Google Scholar 

  39. McCormack B, Kitson A, Harvey G, et al. Getting evidence into practice: the meaning of ‘context’. J Adv Nurs. 2002;38:94–104.

    Article  PubMed  Google Scholar 

  40. Bradley E, Holmboe E, Mattera J, et al. A qualitative study of increasing β-blocker use after myocardial infarction: why do some hospitals succeed? J Am Med Assoc. 2001;285:2604–11.

    Article  CAS  Google Scholar 

  41. Kimmel B. Bridging the gap between knowledge and practice — the veterans administration pathway, Newsletter of the National Council of the University Research Administrators 2003-4;35.

  42. Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: http://www.hsrd.research.va.gov/publications/internal/pm_v9_n1.pdf. Accessed September 9, 2005.

    Google Scholar 

  43. Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomized controlled trial be? BMJ. 2004;328:1561–3.

    Article  PubMed  Google Scholar 

  44. Kirchhoff K, Dille C. Issues in intervention research: maintaining integrity. Appl Nurs Res. 1994;7:32–8.

    Article  CAS  PubMed  Google Scholar 

  45. Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurs Res. 2004;53:63–6.

    Article  PubMed  Google Scholar 

  46. Boyd NR, Windsor RA. A formative evaluation in maternal and child health practice: the partners for life nutrition education program for pregnant women. Matern Child Health J. 2003;7:137–43.

    Article  PubMed  Google Scholar 

  47. Stetler C, Creer E, Effken J. Evaluating a redesign program: challenges and opportunities. In Kelly K., ed. Series on Nursing Administration, Vol. 8. St. Louis: Mosby Year Book; 1996.

    Google Scholar 

  48. Effken E, Stetler C. Impact of organizational redesign. J Nurs Admin. 1997;27:23–32.

    CAS  Google Scholar 

  49. Krumholz H, Herrin J. Quality improvement: the need is there but so are the challenges. Am J Med. 2000;109:501–3.

    Article  CAS  PubMed  Google Scholar 

  50. Willenbring ML, Hagedorn H. Implementing evidence-based practices in opioid agonist therapy clinics. In: Roberts A., Yeager K., eds. Evidence-Based Practice Manual: Research and Outcome Measures in Health and Human Services 2004. New York: Oxford University Press; 2004:340–7.

    Google Scholar 

  51. Willenbring ML, Hagedorn H, Poster AC, Kenny M. Variations in evidence-based clinical practices in nine United States Veterans Administration opioid agonist therapy clinics. Drug Alcohol Dependence. 2004;75:97–106.

    Article  Google Scholar 

  52. LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.

    PubMed  Google Scholar 

  53. La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.

  54. Nazareth I, Freemantle N, Duggan C, Mason J, Haines A. Evaluation of a complex intervention for changing professional behaviour: the evidence based out reach (EBOR) Trial. J Health Serv Res Policy. 2002;7:230–8.

    Article  PubMed  Google Scholar 

  55. Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use. J Biomed Inform. 2003;36:218–27.

    Article  PubMed  Google Scholar 

  56. Sanson-Fisher RW, Grimshaw JM, Eccles MP. The science of changing providers’ behaviour: the missing link in evidence-based practice. Med J Aust. 2004;180:205–6.

    PubMed  Google Scholar 

  57. Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME—PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice. BMC Health Serv Res. 2003;3:22.

    Article  PubMed  Google Scholar 

  58. Yin R. Case Study Research: Design and Methods. Thousand Oaks: Sage; 1994.

    Google Scholar 

  59. Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9.

    Article  PubMed  Google Scholar 

  60. Sharp ND, Pineros SL, Hsu C, Starks H, Sales AE. A qualitative study to identify barriers and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) Northwest Network. Worldviews Evidence-Based Nurs. 2004;1:129–39.

    Article  Google Scholar 

  61. Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.

    Google Scholar 

  62. Litchman J, Roumanis S, Radford M, et al. Can practice guidelines be transported effectively to different settings? Results from a multi-center interventional study. J Comm J Qual Improv. 2001;27:42–53.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Cheryl B. Stetler PhD, RN, FAAN.

Additional information

An earlier version of this article was included in materials to support the State of the Art of Implementation Conference. The conference was conducted by the Department of Veterans Affairs, Office of Research and Development, Health Services Research and Development Service, and was held in Washington, DC, from August 30 to September 1 2004.

The work reported here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stetler, C.B., Legro, M.W., Wallace, C.M. et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 21 (Suppl 2), S1–S8 (2006). https://doi.org/10.1007/s11606-006-0267-9

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-006-0267-9

Key words

Navigation