Panning for gold: An evidence-based tool for assessment of performance indicators in primary health care
Introduction
The past several decades have seen considerable effort directed towards improving systems of accountability and quality within health care systems within a number of countries. More recently increased emphasis on primary health care funding and service delivery have produced a greater focus on the development of performance indicators in primary health care. A proliferation of performance measures and ad hoc indicator sets, often lacking a sound theoretical basis for their selection and development, have led many authors to provide definitions and criteria to help in the development of evidence-based indicator sets suitable for a primary health care setting [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].
The adoption of evidence and consensus-based indicators for primary health care in various national frameworks, however, has not been without controversy and criticism, and debate about their use and value [20], [21], [22]. Virtually any proposed indicator set has met with criticism, generally in one or more of three broad strands: suitability for purpose, technical qualities and impact upon individuals or organizations [21]. Critics often occupy particular roles in the primary health care sector, and tensions between primary health care and policy communities appear to leave a Hobson's choice between alternatives of organisational paralysis or pragmatic implementation of controversial proposals.
Conceptually, the rationale for the introduction of performance indicators assumes that their presence in an organisation will foster a change in the quality of processes within that organisation, which will ultimately produce better outcomes at either a population or cost saving level. This model is predicated on the assumption that the introduced performance indicators possess the necessary attributes to provide a measure that reliably predicts either health outcomes or economic outcomes of value. These necessary attributes include the supporting evidence of best practice and established validity pertaining to the indicator, and the likelihood of this evidence being generalisable to the setting and context within which the indicator will be implemented and from which data would be collected and analysed.
Primary health care, however, is composed of both measurable and unmeasurable elements, within which exist areas of good and bad practice. Performance indicators are inevitably chosen on the basis that they focus on discrete measurable areas of practice, and can thus offer only a partial view of quality and performance within a primary care organisation. Performance indicators tend to operate at the level of single issues (i.e. they examine a few aspects of activity at a detailed level), thus inevitably assuming away complexity, and inviting clinicians and organisations to act as agents of change at the margins of practice. The alternative, of attempting to cover every important part of primary health care by means of performance indicators, would result in an unmanageable proliferation of measures.
Thus, care is needed in the choice of indicators and the manner of their implementation. Further, it is important that debate occurs between theorists, policy makers, clinicians and service end-users to develop agreement over suitable and appropriate indicators for primary health care. We believe that a formal accounting of the relative strengths and weaknesses of any proposed indicator will enable sector commentators from a variety of viewpoints to discuss the relative merits of individual indicators, to understand the political and pragmatic reasons for their inclusion in any set of indicators and to trace the likely organisational impact of any given indicator.
To this end we have developed a schema of performance criteria for the appraisal of indicators, which combines the assessment of scientific evidence, including the technical merits of individual indicators, with contextual considerations from the perspective of both the policy environment and the primary health care sector. The value of the tool is to inform debate about the place and value of any given indicator. We have then applied this tool to the assessment of a set of proposed national performance indicators for primary health care in New Zealand.
This paper details the development of the indicator appraisal tool and discusses its use to assess the quality of the proposed national indicator set for primary health care in New Zealand.
Section snippets
Terminology
A proliferation of terms such as performance indicators, quality indicators, health outcomes, benchmarking and clinical governance are used variably and sometimes interchangeably in the published literature. In this paper, the term performance indicator is defined, as “a measurable element of practice performance, for which there is evidence or consensus that it can be used to assess the quality, and hence the change in quality, of care provided” [10]. A necessary requirement of a good
Background
A major restructuring of the health sector in New Zealand, including primary health care, commenced in early 2001, guided by the New Zealand Health Strategy, and the Primary Health Care Strategy [23], [24]. A key facet of this restructuring was the formation of new not-for-profit bodies known as Primary Health Organisations (PHOs), to co-ordinate the provision of a combination of “first line” general practice services and population health services to an enrolled population [24].
The Primary
Construction of the theoretical framework
The construction of the theoretical framework involved three concurrent streams of work. Firstly, a review of the New Zealand and international literature on the development and use of performance indicators in health care was carried out. Secondly, interviews with a range of key stakeholders from the New Zealand primary health care sector were conducted to gain a broad perspective on the role and potential impact of performance indicators on New Zealand primary health care. Thirdly, designing
Results
Various attributes of performance indicators were identified from the review of the literature and the analysis of the key informant interviews [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].
These attributes were grouped under three overarching headings and subheadings as follows:
- 1.
The rationale for the choice of indicator. This includes:
- a.
The stated purpose for the introduction of the indicator.
- b.
The relevance of the indicator to current
- a.
Discussion
The introduction of performance indicators appears to offer the potential to promote quality improvement initiatives in primary health care. However, the choice of indicators and the manner of their implementation is vitally important if the introduction of performance indicators into an organisational setting is to maintain a focus on a strategic objective of performance measurement within the context of continuous quality improvement. The theoretical framework and sieve tool offer a way to
Conclusions
We have demonstrated that it is possible to construct and use a theoretical framework and a tool to enable the selection and critique of sets of performance indicators for primary health care, taking into account the specific attributes of individual indicators as well as health sector imperatives and scientific evidence. This framework and tool has applicability to other health care systems.
Acknowledgments
We are indebted to the people who offered their time and expertise for key informant interviews.
We gratefully acknowledge the assistance provided by Jon Foley of the Ministry of Health, the Clinical Performance Indicator Advisory Group of the Performance Management Programme and the intellectual input and collegial support provided by members of the academic research team working on the HRC performance indicators project: Dr. Kevin Dew, Dr. Sue Crengle, Prof. Philippa Howden-Chapman, Dr.
References (49)
- et al.
Defining quality of care
Social Science & Medicine
(2000) - et al.
Simplifying the evaluation of primary care programmes
Social Science & Medicine
(1989) - et al.
Influenza uptake in the elderly: results from a rapid assessment of the effectiveness of new government policy in England for the winters 2000/2001 and 2001/2002
Vaccine
(2003) Ten year durability and success of an organised program to increase influenza and pneumococcal vaccination rates among high risk adults
American Journal of Medicine
(1998)Performance indicators: a precautionary tale for unit managers
- et al.
What makes a good performance indicator? Devising primary care performance indicators for New Zealand
New Zealand Medical Journal
(2004) Outomes management and performance indicators
- Gold L, Parry D, Raftery J, Stevens A, Walshe K, Woolley M. An assessment of potential indicators of clinical...
- et al.
Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes
British Medical Journal
(1999) - et al.
Quality of care indicators for population-based primary care in New Zealand
New Zealand Medical Journal
(2002)
Using clinical performance data to stimulate quality improvement
Performance indicators from all perspectives
International Journal for Quality in Health Care
Indicators of quality in health care
European Journal of General Practice
Process versus outcome indicators in the assessment of quality of health care
International Journal for Quality in Health Care
Measuring general practice. A demonstration project to develop and test a set of primary care clinical quality indicators
Population health outcome indicators for the NHS. A feasibility study
Performance indicators for primary care groups: an evidence based approach
British Medical Journal
Knowledge and information for clinical governance
British Medical Journal
Performance measurement for ambulatory care: moving towards a new agenda
International Journal for Quality in Health Care
From a process of care to a measure: the development and testing of a quality indicator
International Journal for Quality in Health Care
Measuring performance in the NHS
British Medical Journal
Using performance indicators to improve health care quality in the public sector: a review of the literature
Health Services Management Research
A report card on continuous quality improvement
Milbank Quarterly
Cited by (25)
Are burn centers in German-speaking countries prepared to respond to a burn disaster? Survey-based study
2020, BurnsCitation Excerpt :The study was based on a cross-sectional descriptive survey questionnaire directed to all burn centers in German-speaking countries, i.e., Germany, Austria, and Switzerland. The self-administered survey questionnaire has been proved to be an effective assessment tool in different healthcare contexts [13]. The survey questionnaire was initially directed to the directors of the burn units as they were assumed to be the most informed professionals in their field [14].
Painting by numbers: A guide for systematically developing indicators of performance at any level of health care
2012, Health PolicyCitation Excerpt :The criteria and parameters chosen for inclusion in the Sieve were guided by the literature and empirical work [4]. The development of the Sieve and other aspects of this work have been published previously [6]. The Sieve deconstructs existing indicators to assess their strengths and weaknesses.
Private hospitals quality assessment: Evidence from Jordan using HEALTHQUAL model
2024, Quality Management JournalQuality of Care Indicators for Hospital Physical Therapy Units: A Systematic Review
2022, Physical TherapyCommon Features of Selection Processes of Health System Performance Indicators in Primary Healthcare: A Systematic Review
2022, International Journal of Health Policy and ManagementPatient satisfaction in the peruvian health services: Validation and application of the HEALTHQUAL scale
2020, International Journal of Environmental Research and Public Health