Elsevier

Social Science & Medicine

Volume 75, Issue 12, December 2012, Pages 2299-2306
Social Science & Medicine

Review
Realist randomised controlled trials: A new approach to evaluating complex public health interventions

https://doi.org/10.1016/j.socscimed.2012.08.032Get rights and content

Abstract

Randomized trials of complex public health interventions generally aim to identify what works, accrediting specific intervention ‘products’ as effective. This approach often fails to give sufficient consideration to how intervention components interact with each other and with local context. ‘Realists’ argue that trials misunderstand the scientific method, offer only a ‘successionist’ approach to causation, which brackets out the complexity of social causation, and fail to ask which interventions work, for whom and under what circumstances. We counter-argue that trials are useful in evaluating social interventions because randomized control groups actually take proper account of rather than bracket out the complexity of social causation. Nonetheless, realists are right to stress understanding of ‘what works, for whom and under what circumstances’ and to argue for the importance of theorizing and empirically examining underlying mechanisms. We propose that these aims can be (and sometimes already are) examined within randomized trials. Such ‘realist’ trials should aim to: examine the effects of intervention components separately and in combination, for example using multi-arm studies and factorial trials; explore mechanisms of change, for example analysing how pathway variables mediate intervention effects; use multiple trials across contexts to test how intervention effects vary with context; draw on complementary qualitative and quantitative data; and be oriented towards building and validating ‘mid-level’ program theories which would set out how interventions interact with context to produce outcomes. This last suggestion resonates with recent suggestions that, in delivering truly ‘complex’ interventions, fidelity is important not so much in terms of precise activities but, rather, key intervention ‘processes’ and ‘functions’. Realist trials would additionally determine the validity of program theory rather than only examining ‘what works’ to better inform policy and practice in the long-term.

Highlights

► ‘Realists’ argue that RCTs ignore the complexity of causation and fail to ask which interventions work, for whom and when. ► We disagree but believe RCTs should examine for whom and when interventions work. ► To this end, RCTs could aim to examine intervention mechanisms of change and how intervention effects vary with context. ► “Realist” RCTs should examine the validity of intervention theory to better inform policy and practice in the long term.

Introduction

In this paper, we outline problems with the way complex public health interventions are sometimes evaluated using randomized controlled trials (RCTs) before examining ‘realist’ critiques of and proposed alternatives to RCTs. Realism in evaluation represents a paradigm through which the world is seen as an open system of dynamic structures, mechanisms and contexts that intricately influence the change phenomena that evaluations aim to capture (Kazi, 2003). Realistic evaluators argue that RCTs fail to test hypotheses rooted in theory and embrace a crude notion of causality based on comparison groups and statistical association rather than understanding mechanisms. They argue that evaluators must develop a priori theories about how, for whom and under what conditions interventions will work and then use observational data to examine how context and intervention mechanism interact to generate outcomes. While we dispute the realists' rejection of experimental designs in the social sciences (Pawson & Tilley, 1997), we agree with their arguments concerning the need for evaluation: to examine how, why and for whom interventions work; to give more attention to context; and to focus on the elaboration and validation of program theory. Some previous authors (Blackwood, O'Halloran et al., 2010) have argued for a synergistic, rather than oppositional, relationship between realist and randomized evaluation:

The RCT can be used to ascertain whether, all other things being equal, a particular causal mechanism (intervention) is efficacious [i.e. effective under optimum conditions], while realistic evaluation can establish what effect the interaction of other mechanisms operating in the open contexts studied has upon its effectiveness, and identify which mechanisms promote, and which inhibit that effectiveness (Blackwood et al., 2010, p. 519).

We would go further than this, however, to propose that RCTs themselves could contribute to a realist approach to evaluation. We examine the extent to which some RCTs are already embracing many of these issues and, bringing together some of these existing innovations alongside our own ideas, sketch out what ‘realist RCTs’ might look like. We argue that it is possible to benefit from the insights provided by realist evaluation without relinquishing the RCT as the best means of examining intervention causality.

Section snippets

Current approaches to evaluating complex public health interventions

RCTs aim to generate minimally biased estimates of intervention effects by ensuring that intervention and control groups are not systematically different from each other in terms of measured and/or unmeasured characteristics. RCTs may randomly allocate individuals or ‘clusters’ of individuals, such as schools or villages; a method that should ensure that the groups are similar other than differences that occur due to chance. Random allocation is widely regarded as ethical if there is

The ‘realist’ critique of, and alternative to, RCTs

The realist tradition provides a critique of, and apparent alternative to, the RCT tradition, but we argue it might be more useful in informing how RCTs could be modified to be more useful in the evaluation of complex public health interventions. Realism asserts that the objects of knowledge exist independently of our minds and that unobservable objects and structures can exert causal influence (Bhaskar, 1975); in contrast to a positivist tradition, which would limit science to directly

Problems with the ‘realist’ position

One might question Pawson and Tilley's claim that the use of RCTs inevitably embodies a set of strong epistemological and ontological commitments, such as the positivistic account of scientific knowledge and the ‘successionist’ theory of causality (Bonell et al., 2003). However, we will not address this broader point here. Instead, firstly, we will concentrate on Pawson and Tilley's critique of the use of counterfactuals to examine causality. Pawson and Tilley are incorrect in arguing that

A synthesis: realist RCTs of complex public health interventions

What might a ‘realist’ program of RCTs look like? We set out some suggestions below illustrated by cases in which RCTs are already employing these approaches, and we suggest how these may be further enhanced.

First, realist RCTs would place emphasis on understanding the effects of intervention components separately as well as in combination. For example, this could involve use of ‘multi-arm studies’ with various combinations of intervention components in each arm. Factorial trials (Montgomery,

Conclusions

This paper has scrutinized apparent tensions between RCT and realist approaches to evaluating complex public health interventions and proposed opportunities for synergy between the two. Some argue against anything other than an approach to complex interventions being experimentally evaluated and those found to be effective being replicated with maximum fidelity, because we have so little understanding of key causal mechanisms (Kemple & Willner, 2008). However, given the uncertain and mixed

References (72)

  • J. Berrueta-Clement et al.

    Changed lives: The effects of the Perry Preschool Program on youths through age 19

    (1984)
  • R.A. Bhaskar

    A realist theory of science

    (1975)
  • B. Blackwood et al.

    On the problems of mixing RCTs with qualitative research: the case of the MRC framework for the evaluation of complex healthcare interventions

    Journal of Research in Nursing

    (2010)
  • C. Blakely et al.

    The fidelity-adaptation debate: implications for the implementation of public sector social programs

    American Community Psychology

    (1987)
  • C. Bonell et al.

    Sexual health interventions should be subject to experimental evaluation

  • C. Bonell et al.

    Alternatives to randomisation in the evaluation of public-health interventions: design challenges and solutions

    Journal of Epidemiology and Community Health

    (2011)
  • C. Bonell et al.

    Trials of health interventions and empirical assessment of generalizability: suggested framework and systematic review

    British Medical Journal

    (2006)
  • G.J. Botvin et al.

    Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population

    Journal of the American Medical Association

    (1995)
  • G.E.P. Box

    Evolutionary operation: a method for increasing industrial productivity

    Applied Statistics

    (1958)
  • S.M. Breitenstein et al.

    Implementation fidelity in community-based interventions

    Research in Nursing & Health

    (2010)
  • R.F. Catalano et al.

    Positive youth development in the United States: research findings on evaluations of positive youth development programs

    Prevention & Treatment

    (2002)
  • Center for the Study and Prevention of Violence
  • J.P. Connell et al.

    Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems

    (1998)
  • Craig P, Dieppe P, Macintyre S, Michie S, Nazareth U, Petticrew M. Developing and evaluating complex interventions: new...
  • P. Craig et al.

    Developing and evaluating complex interventions: the new Medical Research Council guidance

    British Medical Journal

    (2008)
  • U.A. Dane et al.

    Program integrity in primary and early secondary prevention: are implementation effects out of control?

    Clinical Psychology Review

    (1998)
  • A. Dangour et al.

    A factorial-design cluster randomised controlled trial investigating the cost-effectiveness of a nutrition supplement and an exercise programme on pneumonia incidence, walking capacity and body mass index in older people living in Santiago, Chile: the CENEX study protocol

    Nutrition Journal

    (2007)
  • T.J. Dishion et al.

    When interventions harm

    American Psychologist

    (1999)
  • J.A. Durlack

    Why intervention implementation is important

    Journal of Prevention and Intervention in the Community

    (1998)
  • J.A. Durlak et al.

    Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation

    American Journal of Community Psychology

    (2008)
  • J.S. Eccles et al.

    Community based programs for youth: lessons learned from general development research and from experimental and quasi-experimental evaluations

  • A. Fagan et al.

    Bridging science to practice: achieving prevention program implementation fidelity in the community youth development study

    American Journal of Community Psychology

    (2008)
  • D.P. Farrington et al.

    School-based programs to reduce bullying and victimization

    Campbell Systematic Reviews

    (2010)
  • B.R. Flay et al.

    Effects of 2 prevention programs on high-risk behaviors among African American youth: a randomized trial

    Archives of Pediatric and Adolescent Medicine

    (2004)
  • F. Gardner et al.

    Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: outcomes and mechanisms of change

    Journal of Child Psychology and Psychiatry

    (2006)
  • R.E. Glasgow et al.

    Evaluating the overall impact of health promotion programs: using the RE-AIM framework for decision making and to consider complex issues

    Health Education Research

    (2006)
  • Cited by (371)

    • Complex interventions in frail older adults

      2024, Archives of Gerontology and Geriatrics
    View all citing articles on Scopus
    View full text