Elsevier

Health Policy

Volume 80, Issue 2, February 2007, Pages 314-327
Health Policy

Panning for gold: An evidence-based tool for assessment of performance indicators in primary health care

https://doi.org/10.1016/j.healthpol.2006.03.011Get rights and content

Abstract

It is important that debate occurs between theorists, policy makers, clinicians and service end-users to develop agreement over suitable and appropriate indicators for primary health care. A formal accounting of the relative strengths and weaknesses of any proposed indicator will enable sector commentators from a variety of viewpoints to discuss the relative merits of individual indicators, to understand the political and pragmatic reasons for their inclusion in any set of indicators and to trace the likely organisational impact of any given indicator.

This paper details the development of an indicator appraisal tool that combines the assessment of scientific evidence with contextual considerations from the perspective of both the policy environment and the primary health care sector. The use of the tool is discussed in the context of the proposed national implementation of a set of performance indicators in New Zealand.

Introduction

The past several decades have seen considerable effort directed towards improving systems of accountability and quality within health care systems within a number of countries. More recently increased emphasis on primary health care funding and service delivery have produced a greater focus on the development of performance indicators in primary health care. A proliferation of performance measures and ad hoc indicator sets, often lacking a sound theoretical basis for their selection and development, have led many authors to provide definitions and criteria to help in the development of evidence-based indicator sets suitable for a primary health care setting [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].

The adoption of evidence and consensus-based indicators for primary health care in various national frameworks, however, has not been without controversy and criticism, and debate about their use and value [20], [21], [22]. Virtually any proposed indicator set has met with criticism, generally in one or more of three broad strands: suitability for purpose, technical qualities and impact upon individuals or organizations [21]. Critics often occupy particular roles in the primary health care sector, and tensions between primary health care and policy communities appear to leave a Hobson's choice between alternatives of organisational paralysis or pragmatic implementation of controversial proposals.

Conceptually, the rationale for the introduction of performance indicators assumes that their presence in an organisation will foster a change in the quality of processes within that organisation, which will ultimately produce better outcomes at either a population or cost saving level. This model is predicated on the assumption that the introduced performance indicators possess the necessary attributes to provide a measure that reliably predicts either health outcomes or economic outcomes of value. These necessary attributes include the supporting evidence of best practice and established validity pertaining to the indicator, and the likelihood of this evidence being generalisable to the setting and context within which the indicator will be implemented and from which data would be collected and analysed.

Primary health care, however, is composed of both measurable and unmeasurable elements, within which exist areas of good and bad practice. Performance indicators are inevitably chosen on the basis that they focus on discrete measurable areas of practice, and can thus offer only a partial view of quality and performance within a primary care organisation. Performance indicators tend to operate at the level of single issues (i.e. they examine a few aspects of activity at a detailed level), thus inevitably assuming away complexity, and inviting clinicians and organisations to act as agents of change at the margins of practice. The alternative, of attempting to cover every important part of primary health care by means of performance indicators, would result in an unmanageable proliferation of measures.

Thus, care is needed in the choice of indicators and the manner of their implementation. Further, it is important that debate occurs between theorists, policy makers, clinicians and service end-users to develop agreement over suitable and appropriate indicators for primary health care. We believe that a formal accounting of the relative strengths and weaknesses of any proposed indicator will enable sector commentators from a variety of viewpoints to discuss the relative merits of individual indicators, to understand the political and pragmatic reasons for their inclusion in any set of indicators and to trace the likely organisational impact of any given indicator.

To this end we have developed a schema of performance criteria for the appraisal of indicators, which combines the assessment of scientific evidence, including the technical merits of individual indicators, with contextual considerations from the perspective of both the policy environment and the primary health care sector. The value of the tool is to inform debate about the place and value of any given indicator. We have then applied this tool to the assessment of a set of proposed national performance indicators for primary health care in New Zealand.

This paper details the development of the indicator appraisal tool and discusses its use to assess the quality of the proposed national indicator set for primary health care in New Zealand.

Section snippets

Terminology

A proliferation of terms such as performance indicators, quality indicators, health outcomes, benchmarking and clinical governance are used variably and sometimes interchangeably in the published literature. In this paper, the term performance indicator is defined, as “a measurable element of practice performance, for which there is evidence or consensus that it can be used to assess the quality, and hence the change in quality, of care provided” [10]. A necessary requirement of a good

Background

A major restructuring of the health sector in New Zealand, including primary health care, commenced in early 2001, guided by the New Zealand Health Strategy, and the Primary Health Care Strategy [23], [24]. A key facet of this restructuring was the formation of new not-for-profit bodies known as Primary Health Organisations (PHOs), to co-ordinate the provision of a combination of “first line” general practice services and population health services to an enrolled population [24].

The Primary

Construction of the theoretical framework

The construction of the theoretical framework involved three concurrent streams of work. Firstly, a review of the New Zealand and international literature on the development and use of performance indicators in health care was carried out. Secondly, interviews with a range of key stakeholders from the New Zealand primary health care sector were conducted to gain a broad perspective on the role and potential impact of performance indicators on New Zealand primary health care. Thirdly, designing

Results

Various attributes of performance indicators were identified from the review of the literature and the analysis of the key informant interviews [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].

These attributes were grouped under three overarching headings and subheadings as follows:

  • 1.

    The rationale for the choice of indicator. This includes:

    • a.

      The stated purpose for the introduction of the indicator.

    • b.

      The relevance of the indicator to current

Discussion

The introduction of performance indicators appears to offer the potential to promote quality improvement initiatives in primary health care. However, the choice of indicators and the manner of their implementation is vitally important if the introduction of performance indicators into an organisational setting is to maintain a focus on a strategic objective of performance measurement within the context of continuous quality improvement. The theoretical framework and sieve tool offer a way to

Conclusions

We have demonstrated that it is possible to construct and use a theoretical framework and a tool to enable the selection and critique of sets of performance indicators for primary health care, taking into account the specific attributes of individual indicators as well as health sector imperatives and scientific evidence. This framework and tool has applicability to other health care systems.

Acknowledgments

We are indebted to the people who offered their time and expertise for key informant interviews.

We gratefully acknowledge the assistance provided by Jon Foley of the Ministry of Health, the Clinical Performance Indicator Advisory Group of the Performance Management Programme and the intellectual input and collegial support provided by members of the academic research team working on the HRC performance indicators project: Dr. Kevin Dew, Dr. Sue Crengle, Prof. Philippa Howden-Chapman, Dr.

References (49)

  • P.M. Schyve et al.

    Using clinical performance data to stimulate quality improvement

  • J.E. Ibrahim

    Performance indicators from all perspectives

    International Journal for Quality in Health Care

    (2001)
  • M. Lawrence et al.

    Indicators of quality in health care

    European Journal of General Practice

    (1997)
  • J. Mant

    Process versus outcome indicators in the assessment of quality of health care

    International Journal for Quality in Health Care

    (2001)
  • M.N. Marshall et al.

    Measuring general practice. A demonstration project to develop and test a set of primary care clinical quality indicators

    (2003)
  • A.J. McColl et al.

    Population health outcome indicators for the NHS. A feasibility study

    (1993)
  • A. McColl et al.

    Performance indicators for primary care groups: an evidence based approach

    British Medical Journal

    (1998)
  • A. McColl et al.

    Knowledge and information for clinical governance

    British Medical Journal

    (2000)
  • J. Roski et al.

    Performance measurement for ambulatory care: moving towards a new agenda

    International Journal for Quality in Health Care

    (2001)
  • H.R. Rubin et al.

    From a process of care to a measure: the development and testing of a quality indicator

    International Journal for Quality in Health Care

    (2001)
  • M. McKee et al.

    Measuring performance in the NHS

    British Medical Journal

    (1998)
  • T. Freeman

    Using performance indicators to improve health care quality in the public sector: a review of the literature

    Health Services Management Research

    (2002)
  • D. Blumenthal et al.

    A report card on continuous quality improvement

    Milbank Quarterly

    (1998)
  • Cited by (25)

    • Are burn centers in German-speaking countries prepared to respond to a burn disaster? Survey-based study

      2020, Burns
      Citation Excerpt :

      The study was based on a cross-sectional descriptive survey questionnaire directed to all burn centers in German-speaking countries, i.e., Germany, Austria, and Switzerland. The self-administered survey questionnaire has been proved to be an effective assessment tool in different healthcare contexts [13]. The survey questionnaire was initially directed to the directors of the burn units as they were assumed to be the most informed professionals in their field [14].

    • Painting by numbers: A guide for systematically developing indicators of performance at any level of health care

      2012, Health Policy
      Citation Excerpt :

      The criteria and parameters chosen for inclusion in the Sieve were guided by the literature and empirical work [4]. The development of the Sieve and other aspects of this work have been published previously [6]. The Sieve deconstructs existing indicators to assess their strengths and weaknesses.

    • Patient satisfaction in the peruvian health services: Validation and application of the HEALTHQUAL scale

      2020, International Journal of Environmental Research and Public Health
    View all citing articles on Scopus
    View full text