Guiding the design and selection of interventions to influence the implementation of evidence-based practice: an experimental simulation of a complex intervention trial

https://doi.org/10.1016/j.socscimed.2004.08.072Get rights and content

Abstract

A consistent finding in health services research is the report of uneven uptake of research findings. Implementation trials have a variable record of success in effectively influencing clinicians’ behaviour. A more systematic approach may be to conduct Intervention Modelling Experiments before service-level trials, examining intervention effects on ‘interim endpoints’ representing clinical behaviour, derived from empirically supported psychological theories. The objectives were to: (1) Design Intervention Modelling Experiments by backward engineering a ‘real-world’ randomised controlled trial (NEXUS); (2) examine the applicability of psychological theories to clinical decision-making; (3) explore whether psychological theories can illuminate how interventions achieve their effects.

A 2×2 factorial randomised controlled trial was designed with pre- and post-intervention data collection by postal questionnaire surveys. The first survey was used to generate feedback data and the interventions were delivered in the second survey. General medical practitioners (GPs) in England and Scotland participated. First survey respondents were randomised twice to receive or not audit and feedback and educational reminder messages.

The main outcome measures included behavioural intention (general plan to refer for lumbar X-rays) and simulated behaviour (specific, scenario-based, decisions to refer for lumbar X-ray). Predictors were attitude, subjective norm, perceived behavioural control (theory of planned behaviour), self-efficacy (social cognitive theory) and decision difficulty.

Both interventions significantly influenced simulated behaviour, but neither influenced behavioural intention. There were no interaction effects. All theoretically derived cognitions significantly predicted simulated behaviour. Only subjective norm was not predictive of behavioural intention. The effect of audit and feedback on simulated behaviour was mediated through perceived behavioural control.

The results of this study suggest that Intervention Modelling Experiments, using psychological models to help isolate mediators of clinical decision-making, may be a means of developing more potent interventions, and selecting implementation interventions with a greater likelihood of success in a service-level randomised controlled trial.

Introduction

Clinical and health services research is continually producing new findings that may contribute to effective and efficient patient care. However, despite the considerable resources devoted to this area, a consistent finding is that the transfer of research findings into practice is unpredictable and can be a slow and haphazard process. Studies in the UK, USA and the Netherlands suggest that about 30–40% of the patients do not receive care according to current scientific evidence and about 20–25% of care provided is not needed or potentially harmful (e.g. Ketley & Woods, 1993; Grol, 2001; Schuster, McGlynn, & Brook, 1998).

Extensive resources have been devoted to reducing this variation, and hence inappropriate care, in trials of interventions to encourage health professionals to implement evidence-based practice. Over the last decade a large number of implementation intervention studies have been published and reviewed (Bero et al., 1998; Oxman, Thomson, Davis, & Haynes, 1995; Grimshaw et al., 2001). Overall, there has been variable success in effectively influencing health care professionals’ behaviour, and little progress has been made in understanding why any particular implementation intervention succeeds or fails. The substantial heterogeneity within interventions, targeted behaviours, and study settings, makes generalising results of implementation trials problematic; a position aggravated by not drawing on theoretical models specifically directed at behaviour change. While the imperative of reducing variation in clinical practice by encouraging evidence-based practice remains, research conducted to date provides little guidance on the design or selection of implementation interventions for further evaluation in trials.

A possible solution to this problem is to have more systematic methods of developing implementation interventions, optimising service level trials and enhancing the generalisability of research findings. This approach accords with the framework for evaluating complex interventions suggested by the Medical Research Council (MRC, 2000). The MRC framework includes 5 ordered stages: Theory, Modelling, Exploratory Trial, Definitive randomised controlled trial, and Long-term Implementation. To date, implementation research has focused on the definitive randomised controlled trial stage. By drawing on the 3 prior phases, it should be possible to enhance an understanding of how the interventions operate and to concentrate resources on interventions with greater likelihood of success of definitive randomised controlled trials than is presently the case.

One approach to conducting the Exploratory Trial stages of the MRC framework is to conduct Intervention Modelling Experiments. In such experiments, key elements of an intervention are manipulated in a manner that simulates the ‘real-world’ as much as possible, but the measured outcome is an interim endpoint—a proxy for the clinical behaviour that will be the definitive randomised controlled trial outcome. Using interim endpoints, which are easier and cheaper to measure than actual behaviour, means that an Intervention Modelling Experiment functions as a resource-efficient tool to enable intervention design modification, while providing a scientific rationale for intervention selection. An intervention should successfully change the experiment's proxy outcome before being considered eligible for testing on actual behaviour in a definitive randomised controlled trial. Of course an intervention that successfully influences a proxy outcome in a modelling experiment is not guaranteed to successfully influence behaviour—the experiment may not manage to replicate the exact ecological conditions of real life or may investigate only short-term effects. Nevertheless, an intervention that influences a modelling experiment's proxy outcome is more likely to influence the represented behaviour in a service-level trial than an intervention that does not.

Possible interim endpoints can be identified from the large body of research utilising psychological theories to predict and change health behaviour and long-term health outcomes. For example, behaviour change is unlikely to occur if motivation is lacking (Fishbein et al., 2001). Using empirically supported psychological theories to identify interim endpoints should also further an understanding of the interventions themselves. If the interventions achieve their effects by influencing predictive cognitions derived from these models, then this information can guide the future design and development of implementation interventions, thereby increasing the chance of successful definitive randomised controlled trials.

However, the validity of using psychological models in relation to clinician behaviour has yet to be established. It is possible that the uptake of evidence on effective clinical practice may differ from the health-related behaviours and outcomes generally the subject of studies using psychological models. If so, then cognitions drawn from these models could not be used as modelling experiment outcomes. It also may not be possible to capture ‘real-world’ effects of interventions on clinical decision-making at an experimental level. Although modelling experiments in other fields have demonstrated that short-term laboratory effects have been replicated in ‘real-world’ studies of long-term health outcomes (Paul, 1966; Bandura, 1969; Mathews, Gelder, & Johnston, 1981), the validity of using modelling experiments in relation to clinician's implementation behaviour has yet to be established.

Whilst the intended use of a modelling experiment is to optimise the subsequent design of an intervention, we chose, at this exploratory stage, to examine the feasibility of the method by replicating a recently completed trial, the North–East X-ray Utilisation Study (NEXUS: Eccles et al., 2001). The idea was to ‘backward engineer’ a definitive randomised controlled trial that tested two commonly used implementation interventions which have a variable record of success. The NEXUS trial evaluated the relative effectiveness of audit & feedback and educational reminder messages (reminders) in changing general practitioners’ (GP) radiology ordering behaviour for lumber spine and knee X-rays within a 2×2 factorial cluster randomised controlled trial design. The educational reminder messages were based on the Royal College of Radiologists’ guidelines and were provided on the report of every X-ray ordered over a 12-month period. The audit and feedback (which was comparative and of practice-level referrals over 6 months) was delivered to individual GPs at the start of the intervention period and again 6 months later. The study found that reminders led to a significant reduction of approximately 20% in X-ray requests, whilst audit/feedback led to a non-significant reduction of around 1% in X-ray requests (Fig. 1).

The first aim of this study was to examine whether it was possible to design a modelling experiment to reflect a definitive randomised controlled trial. It was acknowledged that reproducing the NEXUS results would not, in itself, test the validity of the methodology, given that the purpose of a modelling experiment is to determine an intervention's eligibility for taking to the definitive randomised controlled trial stage, not guaranteeing its success at that stage. Nevertheless, being able to reproduce both types of interventions at this experimental level would be crucial for the validity of this methodology. Another critical issue for the validity of the modelling experiment method is being able to identify interim endpoints to proxy clinical behaviour. Therefore, the second aim of this study was to examine the applicability of psychological models to identify interim endpoints to clinical decision-making.

The theory of planned behaviour (Ajzen, 1991) and social cognitive theory (Bandura (1997), Bandura (2000)) are psychological models which have been successfully used to predict variation in many different behaviours in many different populations (e.g. Norman & Conner, 1993; Conner & Sparks 1996; Godin & Kok 1996; Cox et al., 1998; Albarracin, Johnson, Fishbein, & Muellerleile, 2001; Armitage & Conner, 2001; Hardeman et al., 2002).

The theory of planned behaviour proposes that the strength of an individual's intention to engage in behaviour, and the degree of control they feel that they have over the behaviour (perceived behavioural control) are the proximal determinants of engaging in it (see Fig. 2). Intention is defined in terms of a conscious plan to exert effort to perform the behaviour. Intention strength is posited as determined by attitudes towards the behaviour, subjective norms and perceived behavioural control, all of which, in turn, are based upon salient beliefs about the behaviour. Attitudes towards the behaviour are proposed to arise from a combination of beliefs about its consequences (behavioural beliefs) and evaluations of those consequences (outcome evaluations). Subjective norms are based on perceptions of the views about the behaviour of other individuals or groups (normative beliefs), and the strength of the individual's desire to gain approval of these groups (motivation to comply). Perceived behavioural control is a function of beliefs about factors likely to facilitate or inhibit the behaviour, including factors such as patient preferences or resource constraints on health professional practice.

In social cognitive theory, it is proposed that behaviour is determined by self-efficacy cognitions, outcome expectations, impediments and proximal goals (Fig. 2). It is proposed that behaviour change and maintenance are directly linked with the modification of self-efficacy cognitions, which is usually the only variable employed when utilising this theory. Self-efficacy cognitions are beliefs about capabilities to organise and execute courses of action required to produce given attainments and are a function of learning and experience. Self-efficacy is expected to vary depending on the task and context, affecting the acquisition, inhibition and disinhibition of behaviour by influencing affective states, choice of tasks and situations, persistence, and effort investment (Bandura, 1997).

The third aim of this study was to explore whether it was possible to use psychological models to further our understanding of the interventions themselves—of how they achieve their effects. To explore the specific cognitions influenced by the interventions, measures of decision difficulty, and variables derived from the theory of planned behaviour and social cognitive theory were employed as predictors.

In view of the importance of the interim endpoint in the modelling experiment methodology, this study used two different measures to proxy GPs’ referral behaviour. One was a theoretically derived measure, behavioural intention. This measure was chosen because there is considerable evidence supporting intention as the single best predictor of subsequent health-related behaviour (Ajzen, 1991; Randall & Wolff, 1994; Conner & Norman, 1996). The second outcome measure, behavioural simulation, was designed to simulate GPs’ referral behaviour by asking GPs to make decisions about who they would refer from a set of scenarios depicting patients with back pain.

To achieve the objectives of this study, four questions were explored in the context of an intervention modelling experiment relating to the referral of patients for lumbar spine X-ray:

  • 1.

    Do audit and feedback or educational reminder messages influence GPs’ behavioural intention?

  • 2.

    Do audit and feedback or educational reminder messages influence GPs’ simulated behaviour?

  • 3.

    Do audit and feedback or educational reminder messages influence GPs’ cognitions (attitude, subjective norm, perceived behavioural control, self-efficacy and decision difficulty)?

  • 4.

    Can cognitive variables derived from psychological models predict GPs’ behavioural intention or simulated behaviour?

Section snippets

Design

This study replicated the design of the NEXUS trial, and was a 2×2 factorial randomised control trial (Fig. 1). Data collection was by two postal questionnaire surveys, two months apart at baseline and post intervention. The baseline survey was used to recruit trial participants and to generate data for the interventions (see below).

Following the baseline survey, respondents were randomised twice—to receive or not to receive audit and feedback and reminders. This produced the four groups of

Results

The descriptive statistics for all measures are reported in Table 1.

Discussion

This study examined whether it was possible to design an intervention modelling experiment to reflect a real-world, definitive randomised controlled trial, testing the effect of audit and feedback and educational message reminders on GPs’ X-ray ordering. The presentation format of the NEXUS interventions was reproduced in this study. In NEXUS, the interventions were individually tailored, based on participants’ referrals during a set period. In this study, GPs’ referral decisions from the

Conclusion

The results of this study show that it is possible to reproduce implementation interventions at an experimental level, although care needs to taken when addressing delivery differences in the ‘real-world’. The results also suggest that psychological theories may help isolate mediators of clinical decision-making, thereby enhancing the potential of intervention modelling experiment methodology. The application of psychological models can help pinpoint the further refinements which may be

Acknowledgements

We are grateful to Tayside Research Network (TayReN) for their support and assistance in this study.

References (35)

  • I. Ajzen

    The theory of planned behaviour

    Organizational Behaviour and Human Decision Processes

    (1991)
  • M. Eccles et al.

    Effect of Audit & Feedback and educational reminder messages on primary care radiology referrals

    Lancet

    (2001)
  • D. Ketley et al.

    Impact of clinical trials on clinical practiceexample of thrombolysis for acute myocardial infarction

    Lancet

    (1993)
  • D. Albarracin et al.

    Theories of reasoned action and planned behavior as models of condom usea meta-analysis

    Psychological Bulletin

    (2001)
  • C.J. Armitage et al.

    Social cognition models and health behavioura structured review

    Psychology & Health

    (2001)
  • A. Bandura

    Principles of behaviour modification

    (1969)
  • A. Bandura

    Self-efficacythe exercise of control

    (1997)
  • A. Bandura

    Health promotion from the perspective of Social Cognitive Theory

  • R.M. Baron et al.

    The moderator–mediator variable distinction in social psychological researchconceptual, strategic, and statistical considerations

    Journal of Personality and Social Psychology

    (1986)
  • L. Bero et al.

    Closing the gap between research and practicean overview of systematic reviews of interventions to promote implementation of research findings by health care professionals

    British Medical Journal

    (1998)
  • Bond, S. (1999). The reassurance value to general practitioners and patients of lumbar spine radiography for low back...
  • D. Bonetti et al.

    Can psychological models bridge the gap between clinical guidelines and clinicians’ behaviour? A randomised controlled trial of an intervention to influence dentists’ intention to implement evidence-based practice

    British Dental Journal

    (2003)
  • M. Conner et al.

    The theory of planned behaviour and health behaviours

  • D.N. Cox et al.

    Take five, a nutrition education intervention to increase fruit and vegetable intakesimpact on consumer choice and nutrient intakes

    British Journal of Nutrition

    (1998)
  • F. Faul et al.

    GPOWERa priori, post-hoc, and compromise power analyses for MS-DOS (Computer program)

    (1992)
  • M. Fishbein et al.

    Factors influencing behaviour and behaviour change

  • Cited by (85)

    • A primary care Web-based Intervention Modeling Experiment replicated behavior changes seen in earlier paper-based experiment

      2016, Journal of Clinical Epidemiology
      Citation Excerpt :

      The UK Medical Research Council framework for developing and evaluating complex interventions proposes more and better theoretical and exploratory work before a full-scale trial as a means of improving intervention development [6]. Intervention Modeling Experiments (IMEs) are one way of doing this exploratory work [7] with some of the present study's authors (D.B., M.P.E., J.J.F., and N.B.P.) involved in their development. In an IME, key elements of the intervention are delivered, using a randomized design, in a manner that approximates the real world but where the measured outcome is generally an interim outcome, a proxy for the behavior of interest.

    View all citing articles on Scopus
    View full text