Guiding the design and selection of interventions to influence the implementation of evidence-based practice: an experimental simulation of a complex intervention trial
Introduction
Clinical and health services research is continually producing new findings that may contribute to effective and efficient patient care. However, despite the considerable resources devoted to this area, a consistent finding is that the transfer of research findings into practice is unpredictable and can be a slow and haphazard process. Studies in the UK, USA and the Netherlands suggest that about 30–40% of the patients do not receive care according to current scientific evidence and about 20–25% of care provided is not needed or potentially harmful (e.g. Ketley & Woods, 1993; Grol, 2001; Schuster, McGlynn, & Brook, 1998).
Extensive resources have been devoted to reducing this variation, and hence inappropriate care, in trials of interventions to encourage health professionals to implement evidence-based practice. Over the last decade a large number of implementation intervention studies have been published and reviewed (Bero et al., 1998; Oxman, Thomson, Davis, & Haynes, 1995; Grimshaw et al., 2001). Overall, there has been variable success in effectively influencing health care professionals’ behaviour, and little progress has been made in understanding why any particular implementation intervention succeeds or fails. The substantial heterogeneity within interventions, targeted behaviours, and study settings, makes generalising results of implementation trials problematic; a position aggravated by not drawing on theoretical models specifically directed at behaviour change. While the imperative of reducing variation in clinical practice by encouraging evidence-based practice remains, research conducted to date provides little guidance on the design or selection of implementation interventions for further evaluation in trials.
A possible solution to this problem is to have more systematic methods of developing implementation interventions, optimising service level trials and enhancing the generalisability of research findings. This approach accords with the framework for evaluating complex interventions suggested by the Medical Research Council (MRC, 2000). The MRC framework includes 5 ordered stages: Theory, Modelling, Exploratory Trial, Definitive randomised controlled trial, and Long-term Implementation. To date, implementation research has focused on the definitive randomised controlled trial stage. By drawing on the 3 prior phases, it should be possible to enhance an understanding of how the interventions operate and to concentrate resources on interventions with greater likelihood of success of definitive randomised controlled trials than is presently the case.
One approach to conducting the Exploratory Trial stages of the MRC framework is to conduct Intervention Modelling Experiments. In such experiments, key elements of an intervention are manipulated in a manner that simulates the ‘real-world’ as much as possible, but the measured outcome is an interim endpoint—a proxy for the clinical behaviour that will be the definitive randomised controlled trial outcome. Using interim endpoints, which are easier and cheaper to measure than actual behaviour, means that an Intervention Modelling Experiment functions as a resource-efficient tool to enable intervention design modification, while providing a scientific rationale for intervention selection. An intervention should successfully change the experiment's proxy outcome before being considered eligible for testing on actual behaviour in a definitive randomised controlled trial. Of course an intervention that successfully influences a proxy outcome in a modelling experiment is not guaranteed to successfully influence behaviour—the experiment may not manage to replicate the exact ecological conditions of real life or may investigate only short-term effects. Nevertheless, an intervention that influences a modelling experiment's proxy outcome is more likely to influence the represented behaviour in a service-level trial than an intervention that does not.
Possible interim endpoints can be identified from the large body of research utilising psychological theories to predict and change health behaviour and long-term health outcomes. For example, behaviour change is unlikely to occur if motivation is lacking (Fishbein et al., 2001). Using empirically supported psychological theories to identify interim endpoints should also further an understanding of the interventions themselves. If the interventions achieve their effects by influencing predictive cognitions derived from these models, then this information can guide the future design and development of implementation interventions, thereby increasing the chance of successful definitive randomised controlled trials.
However, the validity of using psychological models in relation to clinician behaviour has yet to be established. It is possible that the uptake of evidence on effective clinical practice may differ from the health-related behaviours and outcomes generally the subject of studies using psychological models. If so, then cognitions drawn from these models could not be used as modelling experiment outcomes. It also may not be possible to capture ‘real-world’ effects of interventions on clinical decision-making at an experimental level. Although modelling experiments in other fields have demonstrated that short-term laboratory effects have been replicated in ‘real-world’ studies of long-term health outcomes (Paul, 1966; Bandura, 1969; Mathews, Gelder, & Johnston, 1981), the validity of using modelling experiments in relation to clinician's implementation behaviour has yet to be established.
Whilst the intended use of a modelling experiment is to optimise the subsequent design of an intervention, we chose, at this exploratory stage, to examine the feasibility of the method by replicating a recently completed trial, the North–East X-ray Utilisation Study (NEXUS: Eccles et al., 2001). The idea was to ‘backward engineer’ a definitive randomised controlled trial that tested two commonly used implementation interventions which have a variable record of success. The NEXUS trial evaluated the relative effectiveness of audit & feedback and educational reminder messages (reminders) in changing general practitioners’ (GP) radiology ordering behaviour for lumber spine and knee X-rays within a 2×2 factorial cluster randomised controlled trial design. The educational reminder messages were based on the Royal College of Radiologists’ guidelines and were provided on the report of every X-ray ordered over a 12-month period. The audit and feedback (which was comparative and of practice-level referrals over 6 months) was delivered to individual GPs at the start of the intervention period and again 6 months later. The study found that reminders led to a significant reduction of approximately 20% in X-ray requests, whilst audit/feedback led to a non-significant reduction of around 1% in X-ray requests (Fig. 1).
The first aim of this study was to examine whether it was possible to design a modelling experiment to reflect a definitive randomised controlled trial. It was acknowledged that reproducing the NEXUS results would not, in itself, test the validity of the methodology, given that the purpose of a modelling experiment is to determine an intervention's eligibility for taking to the definitive randomised controlled trial stage, not guaranteeing its success at that stage. Nevertheless, being able to reproduce both types of interventions at this experimental level would be crucial for the validity of this methodology. Another critical issue for the validity of the modelling experiment method is being able to identify interim endpoints to proxy clinical behaviour. Therefore, the second aim of this study was to examine the applicability of psychological models to identify interim endpoints to clinical decision-making.
The theory of planned behaviour (Ajzen, 1991) and social cognitive theory (Bandura (1997), Bandura (2000)) are psychological models which have been successfully used to predict variation in many different behaviours in many different populations (e.g. Norman & Conner, 1993; Conner & Sparks 1996; Godin & Kok 1996; Cox et al., 1998; Albarracin, Johnson, Fishbein, & Muellerleile, 2001; Armitage & Conner, 2001; Hardeman et al., 2002).
The theory of planned behaviour proposes that the strength of an individual's intention to engage in behaviour, and the degree of control they feel that they have over the behaviour (perceived behavioural control) are the proximal determinants of engaging in it (see Fig. 2). Intention is defined in terms of a conscious plan to exert effort to perform the behaviour. Intention strength is posited as determined by attitudes towards the behaviour, subjective norms and perceived behavioural control, all of which, in turn, are based upon salient beliefs about the behaviour. Attitudes towards the behaviour are proposed to arise from a combination of beliefs about its consequences (behavioural beliefs) and evaluations of those consequences (outcome evaluations). Subjective norms are based on perceptions of the views about the behaviour of other individuals or groups (normative beliefs), and the strength of the individual's desire to gain approval of these groups (motivation to comply). Perceived behavioural control is a function of beliefs about factors likely to facilitate or inhibit the behaviour, including factors such as patient preferences or resource constraints on health professional practice.
In social cognitive theory, it is proposed that behaviour is determined by self-efficacy cognitions, outcome expectations, impediments and proximal goals (Fig. 2). It is proposed that behaviour change and maintenance are directly linked with the modification of self-efficacy cognitions, which is usually the only variable employed when utilising this theory. Self-efficacy cognitions are beliefs about capabilities to organise and execute courses of action required to produce given attainments and are a function of learning and experience. Self-efficacy is expected to vary depending on the task and context, affecting the acquisition, inhibition and disinhibition of behaviour by influencing affective states, choice of tasks and situations, persistence, and effort investment (Bandura, 1997).
The third aim of this study was to explore whether it was possible to use psychological models to further our understanding of the interventions themselves—of how they achieve their effects. To explore the specific cognitions influenced by the interventions, measures of decision difficulty, and variables derived from the theory of planned behaviour and social cognitive theory were employed as predictors.
In view of the importance of the interim endpoint in the modelling experiment methodology, this study used two different measures to proxy GPs’ referral behaviour. One was a theoretically derived measure, behavioural intention. This measure was chosen because there is considerable evidence supporting intention as the single best predictor of subsequent health-related behaviour (Ajzen, 1991; Randall & Wolff, 1994; Conner & Norman, 1996). The second outcome measure, behavioural simulation, was designed to simulate GPs’ referral behaviour by asking GPs to make decisions about who they would refer from a set of scenarios depicting patients with back pain.
To achieve the objectives of this study, four questions were explored in the context of an intervention modelling experiment relating to the referral of patients for lumbar spine X-ray:
- 1.
Do audit and feedback or educational reminder messages influence GPs’ behavioural intention?
- 2.
Do audit and feedback or educational reminder messages influence GPs’ simulated behaviour?
- 3.
Do audit and feedback or educational reminder messages influence GPs’ cognitions (attitude, subjective norm, perceived behavioural control, self-efficacy and decision difficulty)?
- 4.
Can cognitive variables derived from psychological models predict GPs’ behavioural intention or simulated behaviour?
Section snippets
Design
This study replicated the design of the NEXUS trial, and was a 2×2 factorial randomised control trial (Fig. 1). Data collection was by two postal questionnaire surveys, two months apart at baseline and post intervention. The baseline survey was used to recruit trial participants and to generate data for the interventions (see below).
Following the baseline survey, respondents were randomised twice—to receive or not to receive audit and feedback and reminders. This produced the four groups of
Results
The descriptive statistics for all measures are reported in Table 1.
Discussion
This study examined whether it was possible to design an intervention modelling experiment to reflect a real-world, definitive randomised controlled trial, testing the effect of audit and feedback and educational message reminders on GPs’ X-ray ordering. The presentation format of the NEXUS interventions was reproduced in this study. In NEXUS, the interventions were individually tailored, based on participants’ referrals during a set period. In this study, GPs’ referral decisions from the
Conclusion
The results of this study show that it is possible to reproduce implementation interventions at an experimental level, although care needs to taken when addressing delivery differences in the ‘real-world’. The results also suggest that psychological theories may help isolate mediators of clinical decision-making, thereby enhancing the potential of intervention modelling experiment methodology. The application of psychological models can help pinpoint the further refinements which may be
Acknowledgements
We are grateful to Tayside Research Network (TayReN) for their support and assistance in this study.
References (35)
The theory of planned behaviour
Organizational Behaviour and Human Decision Processes
(1991)- et al.
Effect of Audit & Feedback and educational reminder messages on primary care radiology referrals
Lancet
(2001) - et al.
Impact of clinical trials on clinical practiceexample of thrombolysis for acute myocardial infarction
Lancet
(1993) - et al.
Theories of reasoned action and planned behavior as models of condom usea meta-analysis
Psychological Bulletin
(2001) - et al.
Social cognition models and health behavioura structured review
Psychology & Health
(2001) Principles of behaviour modification
(1969)Self-efficacythe exercise of control
(1997)Health promotion from the perspective of Social Cognitive Theory
- et al.
The moderator–mediator variable distinction in social psychological researchconceptual, strategic, and statistical considerations
Journal of Personality and Social Psychology
(1986) - et al.
Closing the gap between research and practicean overview of systematic reviews of interventions to promote implementation of research findings by health care professionals
British Medical Journal
(1998)
Can psychological models bridge the gap between clinical guidelines and clinicians’ behaviour? A randomised controlled trial of an intervention to influence dentists’ intention to implement evidence-based practice
British Dental Journal
The theory of planned behaviour and health behaviours
Take five, a nutrition education intervention to increase fruit and vegetable intakesimpact on consumer choice and nutrient intakes
British Journal of Nutrition
GPOWERa priori, post-hoc, and compromise power analyses for MS-DOS (Computer program)
Factors influencing behaviour and behaviour change
Cited by (85)
Implementation of repairs in dental practice: Randomized behavior simulation trial
2023, Journal of the American Dental AssociationPsychometric evaluation of a measure of factors influencing hand hygiene behaviour to inform intervention
2019, Journal of Hospital InfectionA primary care Web-based Intervention Modeling Experiment replicated behavior changes seen in earlier paper-based experiment
2016, Journal of Clinical EpidemiologyCitation Excerpt :The UK Medical Research Council framework for developing and evaluating complex interventions proposes more and better theoretical and exploratory work before a full-scale trial as a means of improving intervention development [6]. Intervention Modeling Experiments (IMEs) are one way of doing this exploratory work [7] with some of the present study's authors (D.B., M.P.E., J.J.F., and N.B.P.) involved in their development. In an IME, key elements of the intervention are delivered, using a randomized design, in a manner that approximates the real world but where the measured outcome is generally an interim outcome, a proxy for the behavior of interest.
The development and implementation of a theory-informed, integrated mother-child intervention in rural Uganda
2015, Social Science and Medicine