Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity

Abstract

Paradoxical effects of bias and prevalence on the kappa coefficient are examined using the concepts of sensitivity and specificity. Results that appear paradoxical when viewed as a 2 × 2 table of frequencies do not appear paradoxical when viewed as a pair of sensitivity and specificity measures where each observer is treated as a predictor of the other observer. An adjusted kappa value can be obtained from these sensitivity/specificity measures but simulation studies indicate that it would result in substantial overestimation of reliability when bias or prevalence effects are observed. It is suggested that investigators concentrate on obtaining populations with trait prevalence near 50% rather than searching for statistical indices to rescue or excuse inefficient experiments.

Introduction

The kappa coefficient, first described by Cohen in 1960 [1], is commonly used to assess agreement between two or more observers. In its simplest form, it is used to analyze the case where two observers classify a number of cases according to whether some finding (e.g., a tumor or other disease process) is present or absent. The results may be arranged in the four cells of a 2 × 2 table as shown below.

Cell a indicates the proportion of cases where both observers agree that the finding is present. Cell b indicates the proportion of cases where observer #2 claims that the finding is present while observer #1 claims that it is not. Cell c indicates the alternative form of disagreement (where observer #1 claims that the finding is present and observer #2 claims that it is not) and cell d indicates the proportion of cases where both observers agree that the finding is absent. When these are expressed as proportions, a + b + c + d = 1.0.

The kappa coefficient (κ) is obtained by calculating the proportion of observed agreements (p_o) and expected agreements (p_e) as follows:

$$\text{p}{}_{\mathrm{o}}\text{=a+d}$$

$$\text{p}{}_{\mathrm{e}}\text{=}\text{a}\text{+}\text{b}\text{a}\text{+}\text{c}\text{+}\text{b}\text{+}\text{d}\text{c}\text{+}\text{d}$$

$$\text{κ}\text{=}\text{p}{}_{\mathrm{o}}\text{−}\text{p}{}_{\mathrm{e}}\text{1−}\text{p}{}_{\mathrm{e}}$$

Although the kappa coefficient is quite simple and is widely used to test interrater agreement, a number of authors have described seeming paradoxes associated with the effects of marginal proportions 2, 3, 4, 5, 6, 7, 8. Feinstein and Cicchetti [3] and Byrt et al. [6] have presented the examples reproduced in Table 1.

The first paradox is shown in Examples 1 and 2. Both examples have an identical proportion of agreements (0.85) but the kappa coefficient is substantially lower in Example 2. This effect is attributed to the higher prevalence seen in that example. The second paradox is shown in Examples 3 and 4. Again, the proportion of agreements is the same (0.60) but the kappa coefficient calculated for Example 4 is higher, apparently because that example shows a greater discrepancy between the two observers in marginal proportions.

Examples presented by Lantz and Nebenzahl [7] are shown in Table 2. In all three examples, the two observers agree on 90% of the cases, but the kappa coefficients are quite different.

These problems have been explained in terms of bias and prevalence effects. Prevalence effects occur when the overall proportion of positive results is substantially different from 50%. Bias effects occur when the two observers differ on the proportion of positive results. Byrt et al. [6] and Lantz and Nebenzahl [7] have independently examined an adjustment of kappa for prevalence and bias. Their equation for Prevalence and Bias Adjusted Kappa (PABAK) can be presented as:

$$\text{PABAK}\text{=2p}{}_{\mathrm{o}}\text{−1=}\text{κ}\text{1−PI}{}^2\text{+BI}{}^2\text{+PI}{}^2\text{−BI}{}^2\text{.}$$

This equation uses a bias index (BI) and a prevalence index (PI) defined as:

$$\text{BI=b−c}$$

$$\text{PI=a−d}$$

Use of the bias index is equivalent to replacing cells b and c by their average ([b + c]/2) while use of the prevalence index is equivalent to replacing cells a and d by their average ([a + d]/2) and calculating kappa in the usual fashion.