Markov decision process applied to the control of hospital elective admissions

https://doi.org/10.1016/j.artmed.2009.07.003Get rights and content

Summary

Objective

To present a decision model for elective (non-emergency) patient admissions control for distinct specialties on a periodic basis. The purpose of controlling patient admissions is to promote a more efficient utilization of hospital resources, thereby preventing idleness or excessive use of these resources, while considering their relative importance.

Methods

The patient admission control is modeled as a Markov decision process. A hypothetical prototype is implemented, applying the value iteration algorithm.

Results

The model is able to generate an optimal admission control policy that maintains resource consumption close to the desired levels of utilization, while optimizing the established deviation costs.

Conclusion

This is a complex model due to its stochastic dynamic and dimensionality. The model has great potential for application, and requires the development of customized solution methods.

Introduction

There are several aspects involved in the need for controlling patient admissions. Admissions may be scheduled to satisfy different and sometimes contradictory goals, such as sustaining a high utilization of available hospital capacity (possibly resulting in some bottlenecks); or smooth throughput for a minimum length of patient stay (possibly resulting in some idleness). As mentioned by Kusters and Groot [1], controlling patients’ admissions is the key activity which allows a hospital to balance the demand for patient facilities against the availability of these resources. Choosing the “right” patients from the waiting list in order to reach this balance is not a simple task. It is unlikely that a single model capable of generating an optimal solution and dealing with the complexity of this problem exists. Since the final decision shall take into account a variety of factors, it can only be made by a human decision maker. However, it is possible to provide this decision maker with support. In this study we described and tested a model aimed at providing precisely this type of support.

Unlike Adan and Vissers [2], we do not consider the admission process as a daily occurrence. Rather, we wish to plan the number of admissions within fixed periods of time (e.g., one week or fifteen days). While controlling the number of patients admitted during these fixed periods of time, we aim at sustaining stabilized, desirable (target) levels of the average use of key hospital resources (such as exams, follow-up consultations, in-patient admissions, and surgical interventions), preventing idleness or excessive use of these resources. In other words, we developed a decision model that helps hospital managers plan at the strategic and tactical level, rather than at operational level. At the strategic level (input), top hospital managers must set the desirable levels of use, as well as the relative importance of each hospital resource. At the tactical level (output), the decision model sets, at the beginning of each planning period, an admission policy that, depending on the number of patients being served, their specialties and the pattern of resource consumption, informs the number of admissions to be achieved for each specialty during the next period. At the operational level, if the recommended number of admissions is always adhered to throughout each period, over the long-run, the hospital will be stabilized in relation to the target level for resource use and the relative importance of each resource.

According to Adan and Vissers [2], patients’ admissions to a hospital can be divided into two types: non-scheduled and scheduled. Non-scheduled admissions, also called emergency admissions, concern patients that are immediately admitted as a consequence of medical conditions. Scheduled admissions, also called elective admissions, are selected from a waiting list for an admission date. In this study we focused on scheduled patient admissions. In particular, we analyzed the procedures for elective patient admission in a hospital that does not provide emergency care. Tertiary hospitals, such as rehabilitation services, fit this type of elective care. However, the model developed in this paper can also be applied, with modifications, to hospitals that provide emergency care.

Four studies represented an essential contribution to the development of our model, as follows: 1) a survey on theoretical models of admission planning in Gemmel and Van Dierdonck [3], which included, in detail, the studies of Groot [4], and Roth and Van Dierdonck [5]; 2) the original idea of treatment patterns combined with Markov models in order to estimate hospital resource utilization, from Kapadia et al. [6], [7]; 3) Adan and Vissers [2]’s interesting cost function aimed at stabilizing hospital resource utilization at desirable levels—the authors translated the admission planning problem into a mathematical model in the form of a linear integer program; and 4) Markov processes in Puterman [8]—Markov models suit the stochastic characteristics behind patients’ dynamic throughout hospital services.

There is rather extensive literature on Markov models applied to describe the stochastic dynamics of patients. We included a brief summary of the referred studies that, despite their distinct focus, applied the Markov theory. Smallwood et al. [9] and Kao [10] formed groups of patients admitted to the same specialty and with similar arrival rates at the hospital; next, a distinct and independent semi-Markov model was processed for each one of these groups. The authors developed the necessary formulation to estimate performance parameters, such as the average number of patients in a given department and the expected resource utilization. Hershey et al. [11] and Côté and Stein [12] also modeled semi-Markovian processes, but they considered the flow of one group of patients. The study of Hershey et al. [11] was focused on modeling a hospital as being formed by capacitated facility units, representing transitory states in a semi-Markov process. Côté and Stein [12] went further and introduced the Erlang distribution for governing the transition probabilities among states. Navarro [13] applied proportion calculations in order to estimate the transition probabilities among hospital facilities represented by discrete time Markov chains. Through this Markov chain, he obtained performance parameters, taking the hospital as a closed system comprised of recurrent states and a unique group of patients. Hincapié et al. [14] performed a longitudinal follow-up of groups of patients’ arrivals at the hospital, and estimated the transition probabilities among hospital facilities; then, the authors developed a discrete time Markov chain to obtain performance parameters. Weiss et al. [15] presented a model similar to that of Smallwood et al. [9] and Kao [10], and proposed an iterative methodology for testing the validity of Markovian characteristic assumptions.

In the present study, we propose a new approach, modeling the control of patients’ admissions as a Markov decision process (MDP). MDPs have proven to be useful as models for sequential decision problems with stochastic characteristics and the Markovian property, i.e., future states and decisions are independent from past states and decisions, given the knowledge of the present situation of the system. At each decision instant in an MDP, the system's state is observed and one action is adopted. Based on this information [observed state; adopted action] we can calculate the probabilities of reaching any possible system state in the next decision instant, as well as the expected cost to be incurred until the next decision instant.

The decision process involved in admission planning is repeated over and over during hospital activities. At the tactical level, the decisions concerning the number of patients to be admitted can be made in equally spaced periods (e.g., one week or fifteen days). We assume that it is always possible to count the number of patients during the latest period and classify them into distinct treatment patterns. Kapadia et al. [6], [7] showed us that it is possible to estimate the probabilities of admission of patients to a hospital in any treatment pattern, and that it is also possible, at the beginning of a period, to determine the transition probabilities among the treatment patterns for patients being served. This data allows us to estimate the consumption of each given resource throughout the next period. In this way we defined the elements for the stochastic dynamics related to the MDP.

The motivation for modeling the elective hospital admission system using an MDP was the identification of system characteristics geared towards the concepts of MDPs, namely: (1) a sequential decision process that operates under uncertainty, generating stochastic dynamics; (2) a possibility of observing the system's state at decision instants equally spaced over time (discrete time decisions); and finally, (3) Markovian characteristics can be assumed and modeled (the future is dependent only on the present state and the decision taken).

The proposed model is presented within this context in the following sections. Section 2: brief introduction of the MDPs background; Section 3: presentation of the hospital characteristics as they are considered in this study; Section 4: description of MDP elements—state space, action space, probabilistic dynamics, and cost function; Section 5: presentation of the implementation of a small-sized hypothetical model as an example; Section 6: discussion of practical considerations on the model. Our conclusions are presented in Section 7.

Section snippets

Markov decision process background

MDPs are deeply and comprehensively explored in Puterman [8]. In this section we briefly present the concepts applied in this paper. MDPs can generally be defined by the n-tuple (X, A, P, R), where: X is the set of states; A is the set of actions applicable depending on the states; P represents the transition probabilities among states, depending on an observed state and on an adopted action at a decision instant; and R determines the expected cost related to the observed state and the adopted

Hospital elements

When designing the model for patients’ admissions control, the hospital elements must be defined in a systematic framework. Here, we present the definitions for the three basic elements considered in the model: patients’ demand, treatment patterns, and hospital resources.

Markov decision process elements

Considering the definition in the last section, the corresponding MDP elements are presented in this section.

Model implementation

In this section we present the implementation of a small-sized hypothetical model. We describe the example design (Table 1, Table 2) and show some results to illustrate the correct working of the model (Table 3, Table 4).

We apply the value iteration algorithm (see Section 2) in order to obtain an optimal admission policy. We compare the optimal policy with two non-optimal policies: the “greedy” policy, and the “fixed” policy. Given an observed state, the greedy admission policy prescribes an

Discussion

Rather than the number of admission on a daily basis (operational level), we looked for the number of admissions within successive planning periods (strategic/tactical levels). In this context, with the aim of stabilizing average hospital resources utilization at desirable levels, preventing idleness or excessive use while considering the relative importance of the resources, we modeled the control of patients’ admissions as a Markov decision process.

Based on the results described in the

Conclusion

Controlling the admission of patients into hospitals with limited resources is a traditional and common problem faced by health care systems. Modeling this control as an MDP is a new approach, which may lead to more effective decisions involving the balance between admission of elective patients and utilization of available hospital resources. Combined with an efficient solution method for large dimension MDPs, this method has great potential for application.

References (22)

  • R. Smallwood et al.

    A medical service requirements model for health system design

    Proceedings of the IEEE

    (1969)
  • Cited by (44)

    • Integrated testing strategies can be optimal for chemical risk classification

      2017, Mathematical Biosciences
      Citation Excerpt :

      They formulate the problem as a discrete time acyclic Markov Decision Process and solve it by value iteration. Finally, the transition probabilities do not emerge from the imprecision of measurements of needle’s position, turning angle and velocity but instead are a consequence of the stochastic environment in which the needle operates [18]. develops a Markov Decision Problem for the optimal control of hospital admissions for non-emergency patients.

    • An MDP model for walk-in patient admission management in primary care clinics

      2015, International Journal of Production Economics
      Citation Excerpt :

      Schütz and Kolisch (2012) classify the problems of capacity allocation and appointment scheduling into three levels: capacity allocation or advanced scheduling, appointment scheduling, and short-term decisions on the day of service. According to this classification, most of those studies have applied MDP to model the capacity allocation problems, i.e., determining how many patients should be admitted and how the available capacity should be allocated among different types of patients (Gupta and Wang, 2008; Patrick et al., 2008; Nunes et al., 2009; Ayvaz and Huh, 2010; Liu et al., 2010; Patrick, 2012; Sauré et al., 2012; Schütz and Kolisch, 2012; Gocgun and Puterman, 2014). Different from these studies, our study focuses on short-term decisions, i.e., determining whether or not to admit a walk-in patient and when a walk-in patient should be seen.

    • Approximate Linear Program for Multi-Type Inpatient Allocation Admission Scheduling Problem

      2023, IEEE International Conference on Automation Science and Engineering
    View all citing articles on Scopus
    View full text