Skip to main content

SUPPORT Tools for evidence-informed health Policymaking (STP) 16: Using research evidence in balancing the pros and cons of policies

Abstract

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.

In this article, we address the use of evidence to inform judgements about the balance between the pros and cons of policy and programme options. We suggest five questions that can be considered when making these judgements. These are: 1. What are the options that are being compared? 2. What are the most important potential outcomes of the options being compared? 3. What is the best estimate of the impact of the options being compared for each important outcome? 4. How confident can policymakers and others be in the estimated impacts? 5. Is a formal economic model likely to facilitate decision making?

About STP

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The series is intended to help such people ensure that their decisions are well informed by the best available research evidence. The SUPPORT tools and the ways in which they can be used are described in more detail in the Introduction to this series [1]. A glossary for the entire series is attached to each article (see Additional File 1). Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website http://www.support-collaboration.org. Feedback about how to improve the tools in this series is welcome and should be sent to: STP@nokc.no.

Scenario

You work in the Ministry of Health. The Minister of Health has asked you to present a summary of the expected benefits, harms and costs of an important change in health policy that is being considered.

Background

In this article, we suggest five questions that policymakers and those who support them can ask when considering how to ensure that judgements about the pros and cons of health policy and programme options are well-informed by research evidence. Such questions can be asked, for instance, in scenarios, such as the one described above.

Research alone does not make decisions [2]. Judgements are always required, including judgements about what evidence to use, how to interpret that evidence, and our confidence in the evidence. More importantly, decisions about options require judgements about whether the anticipated desirable consequences outweigh the undesirable consequences (see Figure 1) [3]. In addition to making judgements about how big the impacts are likely to be, decision-making processes require judgements about how important the impacts are, the resources that are required to implement the option [4], and the extent to which the option is a priority relative to other ways in which those resources might be used.

Figure 1
figure 1

Balancing the pros and cons of health policies and programmes. Decisions about health policy or programme options require judgements about whether the desirable consequences of an option are worth the undesirable consequences

It would be simple to make a decision if an option was expected to have large benefits with few downsides and little cost, if we were confident about the evidence and the importance of the benefits, and if the option was a clear priority. Unfortunately, this is rarely the case. More often the expected impacts and costs are uncertain, and complex and difficult judgements must be made.

The questions we propose here do not reduce the need for judgements. However, more systematic considerations and discussions of these questions could help to ensure that important considerations are not overlooked and that judgements are well informed. These could also help to resolve disagreements or at least help to provide clarification. If these judgements are made transparently they could help others to understand the reasoning behind health policy decisions.

Preparing and using a balance sheet (as explained in Table 1 and addressed in the first four questions discussed below) can facilitate well-informed decision making. Sometimes using a formal economic model, such as a cost-effectiveness analysis, can also be helpful. This latter issue is addressed in the fifth question discussed in this article. The considerations we suggest here are based on the work of the GRADE Working Group [5]. Although the Group's focus has been primarily on clinical practice guidelines, their approach to decisions about clinical interventions can also be applied to policies and programmes [6].

Table 1 The pros and cons of balance sheets

Questions to consider

The following five questions can be used to guide the use of evidence to inform judgements about the pros and cons of health policy and programme options:

  1. 1.

    What are the options that are being compared?

  2. 2.

    What are the most important potential outcomes of the options being compared?

  3. 3.

    What is the best estimate of the impact of the options being compared for each important outcome?

  4. 4.

    How confident can policymakers and others be in the estimated impacts?

  5. 5.

    Is a formal economic model likely to facilitate decision making?

The first four questions are intended to guide the use of balance sheets in policy decision making. Answering the final question can help to ensure that the scarce resources used in full economic analyses are applied where they are needed most.

Ideally, balance sheets (and economic models) should be constructed by researchers or technical support staff together with policymakers. They should also be based on systematic reviews for the same reasons described elsewhere that highlight the importance of systematic reviews in general [7]. We will not consider the many detailed judgements that must be made when constructing a balance sheet as these have been addressed elsewhere [8]. Policymakers are rarely, if ever, in a position where they are required to make all such judgements themselves. Yet even in instances where there is competent technical support to prepare a balance sheet, it is important that policymakers know what to look for and what questions to ask. This ensures that balance sheets can be used judiciously to inform the decisions for which policymakers are accountable.

1. What are the options that are being compared?

When using a balance sheet such as the one shown in Table 2, the first consideration is the need to identify what options are being compared. Often this is not as straightforward as it sounds (see Table 3, for example). Those preparing a balance sheet must decide on both the option being considered and the comparative option. Typically, the comparison is the status quo. However, the status quo is likely to vary from setting to setting. Decisions need to be made, therefore, about which characteristics of the status quo are:

Table 2 Should the licensing of tobacco retailers be conditional on not selling tobacco to minors?
Table 3 What is being compared? Case example: The licensing of tobacco retailers
  • Crucial - such that research with a comparison without those same characteristics would be excluded

  • Important but not crucial - such that research with a comparison without those same characteristics would be included, but with less confidence that the results would be the same in the chosen setting, and

  • Unimportant - such that we would be confident that the results are likely be the same in the chosen setting

These same judgements also need to be made about the options being considered: which of their characteristics are crucial, important or unimportant in terms of affecting the likely impacts?

2. What are the most important potential outcomes of the options being compared?

Policymakers, in general, are motivated by the desire to serve the people they represent and should be interested primarily in the impacts of policy and programme options on outcomes that are important to those affected (see, for example, Table 4). These include health outcomes, access to - or utilisation of - health services, unintended effects (harms), and resource use (costs or savings) (see Figure 1). Other often important consequences include the distribution and equity of benefits and costs [9], and spillover effects to other sectors. Ethical consequences such as those related to a reduction in people's autonomy, may also be important.

Table 4 What are the most important outcomes? Case example: The licensing of tobacco retailers

Being explicit about which outcomes are important can help to ensure that the important consequences of an option are not overlooked. It can also help to ensure that unimportant consequences are not given undue weight. This is particularly important for surrogate outcomes - i.e. outcomes that are not important in and of themselves. They are considered important because they are believed to reflect important outcomes. For example, people do not typically regard their blood pressure as an important concern. What makes the issue of blood pressure important is its association with strokes, heart attacks and death, all of which are very much of importance to people. So when considering options targeted at hypertension (or other cardiovascular risk factors), decisions should be based on the impacts of these options on important outcomes (cardiovascular disease). Evidence of impacts on blood pressure alone is only a form of indirect evidence of the impacts on cardiovascular disease.

3. What is the best estimate of the impact of the options being compared for each important outcome?

Deciding whether the desirable impacts of an option are worth the undesirable impacts requires an estimate of how large these different impacts (and their economic consequences) will be. Ideally, this should take the form of a comparison between what could be expected for every important outcome if an option were to be implemented, and what could be expected if it were not - or what could be expected if a different option were implemented instead (see Table 5, for example). It is also useful to know how precise each estimate is - i.e. what the 'confidence interval' is for each estimate (this is explained further in Table 6).

Table 5 What are the best estimates of the impacts? Case example: The licensing of tobacco retailers
Table 6 Confidence intervals

It is important that decision makers recognise the difference between estimates of effect that are presented as relative effects, and those that are presented as absolute effects. Patients, health professionals, and people making decisions about health policies and programmes are more likely to decide to use an intervention if its effects are reported as a relative effect than if they are reported as an absolute effect [10]. For example, a study reported that 61% of a sample of health professionals in Australia agreed to implement a colorectal cancer screening programme that would reduce the rate of deaths from bowel cancer by 17% (the relative risk reduction). In comparison, only 24% of the health professionals agreed to implement a programme that produced an absolute reduction in deaths from bowel cancer of 0.4% (the absolute risk reduction) [11]. Both estimates were, in fact, from the same programme (for an explanation of the difference between relative and absolute effects see Table 4 in Article 10 of this series [9]).

4. How confident can policymakers and others be in the estimated impacts?

Six factors can lower our confidence in estimates of the impacts of a policy or programme [12]:

  • A weak study design

  • Other study limitations

  • Imprecision

  • Inconsistent results

  • Indirectness of the evidence

  • Publication bias

An assessment of these factors is inevitably technical. Policymakers do not need to have a detailed understanding of these factors or how they are assessed. But both policymakers and their technical support staff can still benefit from understanding why it is important to consider these factors.

Studies in which a programme is randomly assigned reduce the risk of unknown or unmeasured differences between the groups being compared. This gives greater confidence that impacts are attributable to the programme and not some other factor [13–15]. Study designs that do not use random assignment can account only for differences that are measured. For example, a study in which communities are randomly assigned to a programme or policy option, such as the licensing of tobacco retailers, would provide more compelling evidence of the impacts of an option than a study would if it compared communities that had decided themselves whether to implement a particular option. This is because communities that decide to implement an option are likely to differ from those that do not in ways that could have an impact on the outcomes of interest (in this case, smoking prevalence). It would therefore be impossible to know whether the differences in outcomes were due to the policy or programme option or due to those other differences between the communities.

Other study limitations can affect both randomised and non-randomised impact evaluations. Incomplete data or the unreliable measurement of outcomes, for instance, may increase the risk of an estimate being biased, and therefore lower confidence in the derived estimates.

Imprecision (as indicated by a wide confidence interval) also lowers the confidence with which chance can be ruled out as a factor shaping any observed differences in outcomes between compared groups, and consequently our confidence in an estimated effect. (Table 6 explains the concept of confidence intervals in further detail)

If different studies of the same policy or programme option have inconsistent results and there is no compelling explanation for such differences, there will also be less confidence in knowing the expected impacts arising from implementing the option.

There are several ways in which studies might not be directly relevant to a particular question, and therefore result in less confidence in the results. As noted above, if an indirectly relevant outcome (such as blood pressure) is measured in place of an important outcome (cardiovascular disease), there will be less confidence in the impacts on the important outcome (for which the indirect outcome is a surrogate). If only indirect comparisons are provided, confidence will also be lower. We would be less confident in studies of an option that lacked head-to-head comparisons, for example, between the option compared to a control (with no intervention) and studies of a different option compared to a control. Other ways in which evidence can be indirect include differences between a study and the setting of interest in:

  • The characteristics of the population

  • The option being considered, or

  • The status quo or comparison option

Studies that find statistically significant effects are often more likely to be published than those that do not [16]. When such 'publication bias' appears likely, confidence in estimates from published studies alone may also be lowered. Publication bias should be considered in instances where there are a number of small studies, especially if these are industry-sponsored, or if the investigators are known to share other similar conflicts of interest.

In summary, assessments of the 'quality' or robustness of evidence, and confidence in estimates of the likely impacts of options, depend on a consideration of all of the factors noted above. Although there are no fixed rules for assessing these factors, judgements related to the quality of evidence that explicitly address each factor help to reduce the likelihood of important factors being overlooked. They also help to reduce the probability of biased assessments of the evidence (see Table 7, for example). Using a systematic and transparent approach, such as the GRADE approach (see Table 8), makes it easier to inspect the judgements made [5].

Table 7 How confident are we in the estimated impacts? Case example: The licensing of tobacco retailers
Table 8 The GRADE system for assessing the quality of evidence

5. Is a formal economic model likely to facilitate decision making?

Formal economic models, such as cost-effectiveness analyses and cost-utility analyses, can help to inform judgements about the balance between the desirable and undesirable consequences of an option [17]. Economic models can be valuable for complex decision making and for testing how sensitive a decision is to key estimates or assumptions. A model, though, is only as good as the data on which it is based. When estimates of benefits, harms or resource use come from low-quality evidence, the results will necessarily be highly speculative (an example is provided in Table 9).

Table 9 Is a formal economic model likely to help? Case example: The licensing of tobacco retailers

A full economic model is more likely to help to inform a decision when there is:

  • A large difference in the resources consumed between the compared options

  • Large capital investments are required, such as the construction of new facilities

  • Uncertainty about whether the net benefits are worth the incremental costs

  • Good quality evidence regarding resource consumption

An economic model can also be used to clarify information needs by exploring the sensitivity of an analysis to a range of plausible estimates.

Unfortunately, published cost-effectiveness analyses, particularly those undertaken for drugs, have a high probability of being flawed or biased. They are also specific to a particular setting which may differ in important ways from the setting of interest [18]. Policymakers may thus consider developing their own formal economic models. To do this, they must have the necessary expertise and resources.

Conclusion

Policy decisions are informed by assessments of the balance between the pros and cons of options. As we have recommended, these should be done systematically and transparently. When the net benefit (i.e. the difference between the desirable and undesirable consequences) is large in relation to the costs, we are more confident about a decision. When the net benefit is small in relation to the costs, we are less confident.

Generally, the less confident we are about the likely impacts of an option, the less confident we will be when deciding what to do. There are exceptions to this: firstly, we may have so little confidence about the impacts of something that it is easy to decide not to do it.

Secondly, even if there is little confidence in the benefits of a particular option it may be easy to decide to do something simply because there is little or no risk of harm, it doesn't cost much, and it might do some good. Many types of health information could be categorised as such. Policymakers, though, should be cautious about assuming that seemingly harmless polices and programmes cannot do harm [19]. Even something as simple as providing health information can, in fact, be deadly [20]. This is demonstrated by the advice given to mothers in many countries for nearly 50 years, that babies should sleep on their front. The seemingly harmless advice caused tens of thousands of deaths from sudden infant death syndrome [21].

Finally, despite important uncertainty about the likely impacts of a policy or programme, it may be easy to come to a decision that something that is promising should only be done in the context of a well-designed evaluation of its impacts [22].

Even when we are confident about the impacts of a policy or programme, it may not be a priority to implement it. The extent to which we are confident is a critical factor for deciding on what to do and the extent to which doing something is a priority. Other additional factors (such as those described in Table 10) may also determine whether policy or programme implementation is a priority or not.

Table 10 Factors that can determine the importance of implementing health policies and programmes

Resources

Useful documents and further reading

  • Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ, and the GRADE Working Group. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336:924-6.

  • Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, and the GRADE Working Group. What is 'quality of evidence' and why is it important to clinicians? BMJ 2008; 336:995-8.

  • Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Vist GE, Schunemann HJ, and the GRADE Working Group. Incorporating considerations of resource use. BMJ 2008; 336:1170-3.

Links to websites

  • SUPPORT Summaries: http://www.support-collaboration.org/index.htm - Concise summaries of the pros and cons of health policies and programmes for low- and middle-income countries based on systematic reviews.

  • GRADE Working Group: http://www.gradeworkinggroup.org - The Grading of Recommendations Assessment, Development and Evaluation (or GRADE) Working Group has developed an approach to grading the quality of evidence and the strength of healthcare recommendations.

References

  1. Lavis JN, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). Introduction. Health Res Policy Syst. 2009, 7 (Suppl 1): I1-10.1186/1478-4505-7-S1-I1.

    Article  PubMed Central  PubMed  Google Scholar 

  2. Chalmers I: If evidence-informed policy works in practice, does it matter if it doesn't work in theory?. Evidence & Policy. 2005, 1: 227-42.

    Article  Google Scholar 

  3. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, Schunemann HJ: Going from evidence to recommendations. BMJ. 2008, 336: 1049-51. 10.1136/bmj.39493.646875.AE.

    Article  PubMed Central  PubMed  Google Scholar 

  4. Oxman AD, Fretheim A, Lavis JN, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 12. Finding and using research evidence about resource use and costs. Health Res Policy Syst. 2009, 7 (Suppl 1): S12-10.1186/1478-4505-7-S1-S12.

    Article  PubMed Central  PubMed  Google Scholar 

  5. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ: GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008, 336: 924-6. 10.1136/bmj.39489.470347.AD.

    Article  PubMed Central  PubMed  Google Scholar 

  6. The SUPPORT Collaboration: How SUPPORT Summaries are prepared. 2009, [http://www.support-collaboration.org/summaries/methods.htm]

    Google Scholar 

  7. Lavis JN, Oxman AD, Grimshaw J, Johansen M, Boyko JA, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 7. Finding systematic reviews. Health Res Policy Syst. 2009, 7 (Suppl 1): S7-10.1186/1478-4505-7-S1-S7.

    Article  PubMed Central  PubMed  Google Scholar 

  8. Guyatt G, Oxman A, Kunz R, Vist G, Brozek J, Norris S: GRADE guidelines: 1. Introduction - GRADE evidence profiles and Summary of Findings tables. J Clin Epidemiol.

  9. Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP). 10. Taking equity into consideration when assessing the findings of a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S10-10.1186/1478-4505-7-S1-S10.

    Article  PubMed Central  PubMed  Google Scholar 

  10. Akl E, Oxman A, Herrin J, Vist G, Terrenato I, Sperati F: Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database of Systematic Reviews.

  11. Adily A, Ward J: Evidence based practice in population health: a regional survey to inform workforce development and organisational change. J Epidemiol Community Health. 2004, 58: 455-60. 10.1136/jech.2003.012278.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  12. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ: What is "quality of evidence" and why is it important to clinicians?. BMJ. 2008, 336: 995-8. 10.1136/bmj.39490.551019.BE.

    Article  PubMed Central  PubMed  Google Scholar 

  13. Kleijnen J, Gotzsche P, Kunz R, Oxman A, Chalmers I: So what's so special about randomisation. Non-Random Reflections on Health Care Research: On the 25th Anniversary Of Archie Cochrane's Effectiveness and Efficiency. Edited by: Chalmers I, Maynard A. 1997, London: BMJ Publishers, 93-106.

    Google Scholar 

  14. Kunz R, Oxman AD: The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ. 1998, 317: 1185-90.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  15. Savedoff WD, Levine R, Birdsall N: When will we ever learn? Improving lives through impact evaluation. 2006, Washington DC, Center for Global Development, [http://www.cgdev.org/files/7973_file_WillWeEverLearn.pdf]

    Google Scholar 

  16. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K: Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009, 1: MR000006-

    PubMed  Google Scholar 

  17. Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A, Vist GE, Schunemann HJ, GRADE Working Group: Incorporating considerations of resources use into grading recommendations. BMJ. 2008, 336: 1170-3. 10.1136/bmj.39504.506319.80.

    Article  PubMed Central  PubMed  Google Scholar 

  18. Friedberg M, Saffran B, Stinson TJ, Nelson W, Bennett CL: Evaluation of conflict of interest in economic analyses of new drugs used in oncology. JAMA. 1999, 282: 1453-7. 10.1001/jama.282.15.1453.

    Article  CAS  PubMed  Google Scholar 

  19. Macintyre S, Petticrew M: Good intentions and received wisdom are not enough. J Epidemiol Community Health. 2000, 54: 802-3. 10.1136/jech.54.11.802.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  20. Chalmers I: Invalid health information is potentially lethal. BMJ. 2001, 322: 998-10.1136/bmj.322.7292.998.

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  21. Gilbert R, Salanti G, Harden M, See S: Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol. 2005, 34: 874-87. 10.1093/ije/dyi088.

    Article  PubMed  Google Scholar 

  22. Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP). 17. Dealing with insufficient research evidence. Health Res Policy Syst. 2009, 7 (Suppl 1): S17-10.1186/1478-4505-7-S1-S17.

    Article  PubMed Central  PubMed  Google Scholar 

  23. Eddy DM: Comparing benefits and harms: the balance sheet. JAMA. 1990, 263: 2493-10.1001/jama.263.18.2493. 2498, 2501.

    Article  CAS  PubMed  Google Scholar 

  24. Stead LF, Lancaster T: Interventions for preventing tobacco sales to minors. Cochrane Database Syst Rev. 2005, 1: CD001497-

    PubMed  Google Scholar 

  25. Kapiriri L, Norheim OF, Martin DK: Priority setting at the micro-, meso- and macro-levels in Canada, Norway and Uganda. Health Policy. 2007, 82: 78-94. 10.1016/j.healthpol.2006.09.001.

    Article  PubMed  Google Scholar 

  26. Aaserud M, Trommald M, Oxman AD, Innvaer S: [Evaluation of reimbursement applications for new drugs]. Tidsskr Nor Laegeforen. 2002, 122: 2619-23.

    PubMed  Google Scholar 

Download references

Acknowledgements

Please see the Introduction to this series for acknowledgements of funders and contributors. In addition, we would like to acknowledge Benjamin Djulbegovic and two anonymous reviewers for helpful comments on an earlier version of this article.

This article has been published as part of Health Research Policy and Systems Volume 7 Supplement 1, 2009: SUPPORT Tools for evidence-informed health Policymaking (STP). The full contents of the supplement are available online at http://www.health-policy-systems.com/content/7/S1

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Andrew D Oxman.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

ADO prepared the first draft of this article. JNL, AF and SL contributed to drafting and revising it.

Electronic supplementary material

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Oxman, A.D., Lavis, J.N., Fretheim, A. et al. SUPPORT Tools for evidence-informed health Policymaking (STP) 16: Using research evidence in balancing the pros and cons of policies. Health Res Policy Sys 7 (Suppl 1), S16 (2009). https://doi.org/10.1186/1478-4505-7-S1-S16

Download citation

  • Published:

  • DOI: https://doi.org/10.1186/1478-4505-7-S1-S16

Keywords