Information technology and medical missteps: Evidence from a randomized trial

Abstract

We analyze the effect of a decision support tool designed to help physicians detect and correct medical “missteps”. The data comes from a randomized trial of the technology on a population of commercial HMO patients. The key findings are that the new information technology lowers average charges by 6% relative to the control group. This reduction in resource utilization was the result of reduced in-patient charges (and associated professional charges) for the most costly patients. The rate at which identified issues were resolved was generally higher in the study group than in the control group, suggesting the possibility of improvements in care quality along measured dimensions and enhanced diffusion of new protocols based on new clinical evidence.

Introduction

In 1987, Nobel Laureate Robert Solow famously remarked, “you can see the computer age everywhere but in the productivity statistics.” (Solow, 1987, p. 36). Solow's aphorism neatly summarized the state of knowledge in the late 1980s and early 1990s. Since that time, however, economists have been able to identify measurable economic effects of the revolution in information technology (IT). The emerging consensus from this research is that the effect of IT varies depending on the design of organizations and the nature of production processes. IT complements the work of people engaged in non-routine problem solving and communication while it substitutes for lower-skill tasks involving the sorts of explicit rules that are relatively easy to program into computers.¹

Studying the effect of IT on work processes involving non-routine problem solving and communication is hard—in large part because the inherent complexity of these processes make it difficult to identify meaningful performance measures that are also directly related to specific IT innovations. The search for good performance indicators and cleanly demarcated innovations has moved economists away from the analysis of aggregate productivity and technology data towards more narrowly focused studies.² The added institutional knowledge made possible by the limited scope of these studies also helps analysts address the selection problems created by the non-random distribution of new innovations across organizations and work places.³

In this paper, we also analyze the effects of an IT enabled innovation in a narrowly defined production process characterized by non-routine problem solving and communication. The information technology we study is a decision support tool designed to notify physicians about potential medical “errors” as well as deviations from evidence-based clinical practice guidelines. Our approach is closest in spirit to Athey and Stern's (2002) study of emergency medical services. Like Athey and Stern, we focus on the introduction of a discrete innovation that altered the handling of information in a health care setting and we assess the efficacy of the innovation by tracking health-related outcomes. Our econometric approach, however, differs from theirs in that we use a randomized controlled trial to identify the effect of the new technology.⁴

Although we focus on a specific production process, the results we report have broad implications for management and economic issues in health care. A large and influential body of research suggests that preventable medical errors have a substantial effect on the cost and quality of medical care.⁵ In response to these findings, a number of high-profile public and private initiatives have called for major new investments in information technology and decision support tools to reduce the incidence of errors and increase compliance with evidence-based treatment guidelines (President's Information Technology Advisory Committee, 2004; Institute of Medicine Committee on Quality of Health Care in America, 2001).⁶ Economists who have examined these issues generally agree that new information technologies and decision support tools – perhaps combined with novel incentive arrangements – will likely have a substantial influence on both errors and efficiency in the delivery of health care, yet economic studies concerning the efficacy these interventions have been scarce (Newhouse, 2002).⁷

The data in this study comes from a randomized trial of a physician decision support technology introduced to a population of commercial HMO patients. We find that the intervention reduced resource utilization: average charges were 6% lower in the study group than in the control group. These savings were the result of reduced in-patient charges (and associated professional charges) for the most costly patients.

The importance of IT-based decision support systems for physicians extends beyond resource utilization: patients, providers, payers and policy-makers want to know whether this type of technology improves care quality. Decision support might improve quality if the system reminded physicians to do something beneficial for their patients that they had already intended to do but somehow forgot. Alternatively, decision support might improve quality if it provided useful new information to physicians in a form that was easy to incorporate into their daily practice and routines. This latter avenue of action is especially important because of the problem of physician information overload. In medicine, the number and variety of diseases and treatments and the rapid growth of new knowledge threaten to overwhelm the information processing capacities of individual doctors. Failure to keep abreast of this flood of information can cause physicians to overlook important new treatments or protocols that may improve care quality (Frank, 2004, Phelps, 2000, IMCQHCA, 2000, IMCQHCA, 2001).

Although the experiment was not designed to analyze the source of missteps or the mechanisms by which the technology influenced physicians, we can learn something about quality by comparing the rate at which identified issues were resolved in study and control groups.⁸ Under plausible assumptions, a higher rate of resolution in the study relative to the control group can be interpreted as an improvement in care quality—at least along measured dimensions. Our findings generally point towards higher resolution rates in the study group, although measurement issues discussed below require that we present this conclusion cautiously. The increase in resolution rates was especially large for a new treatment protocol that emerged from the results of a widely publicized clinical trial in the year 2000, the year before our study began. Computer generated messages suggesting that a patient appeared to be a good candidate for the new protocol were triggered quite frequently in our study, suggesting that it takes some time for physicians to incorporate even widely promoted new protocols into their treatment of patients. More importantly we found that the resolution rates in the study group were double those in the control group. This result suggests that the IT system may have been more effective than conventional communication channels in disseminating new knowledge to physicians. We discuss the economic and behavioral implications of these results below.

The plan of the paper is as follows: Section 2 describes the setting of the trial and the decision support technology. Section 3 presents the data analysis. Section 4 concludes and discusses new research questions raised by the study.