Abstract
The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.
Article PDF
Similar content being viewed by others
References
Cox, D. R., & Hinkley, D. V. (1974).Theoretical statistics. London: Chapman and Hall.
Davison, A. C., & Hinkley, D. V. (1997).Bootstrap methods and their application. Cambridge: Cambridge University Press.
Efron, B. (1979). Bootstrap methods: Another look at the jackknife.Annals of Statistics,7, 1–26.
Efron, B. (1982).The jackknife, the bootstrap and other resampling plans (CBMS-NSF Regional Conference Series in Applied Mathematics). Philadelphia: Society for Industrial and Applied Mathematics.
Efron, B. (1987). Better bootstrap confidence intervals.Journal of the American Statistical Association,82, 171–200.
Efron, B. (1988). Bootstrap confidence intervals: Good or bad?Psychological Bulletin,104, 293–296.
Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation.American Statistician,37, 36–48.
Efron, B., & Tibshirani, R. (1991). Statistical data analysis in the computer age.Science,253, 390–395.
Efron, B., & Tibshirani, R. J. (1993).An introduction to the bootstrap. New York: Chapman and Hall.
Finney, D. J. (1952).Probit analysis (2nd ed.). Cambridge: Cambridge University Press.
Finney, D. J. (1971).Probit analysis. (3rd ed.). Cambridge: Cambridge University Press.
Foster, D. H., & Bischof, W. F. (1987). Bootstrap variance estimators for the parameters of small-sample sensory-performance functions.Biological Cybernetics,57, 341–347.
Foster, D. H., & Bischof, W. F. (1991). Thresholds from psychometric functions: Superiority of bootstrap to incremental and probit variance estimators.Psychological Bulletin,109, 152–159.
Foster, D. H., & Bischof, W. F. (1997). Bootstrap estimates of the statistical accuracy of thresholds obtained from psychometric functions.Spatial Vision,11, 135–139.
Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals (with discussion).Annals of Statistics,16, 927–953.
Hill, N. J. (2001a, May).An investigation of bootstrap interval coverage and sampling efficiency in psychometric functions. Poster presented at the Annual Meeting of the Vision Sciences Society, Sarasota, FL.
Hill, N. J. (2001b).Testing hypotheses about psychometric functions: An investigation of some confidence interval methods, their validity, and their use in assessing optimal sampling strategies. Forthcoming doctoral dissertation, Oxford University.
Hill, N. J., & Wichmann, F. A. (1998, April).A bootstrap method for testing hypotheses concerning psychometric functions. Paper presented at the Computers in Psychology, York, U.K.
Hinkley, D. V. (1988). Bootstrap methods.Journal of the Royal Statistical Society B,50, 321–337.
Kendall, M. K., & Stuart, A. (1979).The advanced theory of statistics: Vol. 2. Inference and relationship. New York: Macmillan.
Lam, C. F., Mills, J. H., & Dubno, J. R. (1996). Placement of observations for the efficient estimation of a psychometric function.Journal of the Acoustical Society of America,99, 3689–3693.
Maloney, L. T. (1990). Confidence intervals for the parameters of psychometric functions.Perception & Psychophysics,47, 127–134.
McKee, S. P., Klein, S. A., & Teller, D. Y. (1985). Statistical properties of forced-choice psychometric functions: Implications of probit analysis.Perception & Psychophysics,37, 286–298.
Rasmussen, J. L. (1987). Estimating correlation coefficients: Bootstrap and parametric approaches.Psychological Bulletin,101, 136–139.
Rasmussen, J. L. (1988). Bootstrap confidence intervals: Good or bad. Comments on Efron (1988) and Strube (1988) and further evaluation.Psychological Bulletin,104, 297–299.
Strube, M. J. (1988). Bootstrap type I error rates for the correlation coefficient: An examination of alternate procedures.Psychological Bulletin,104, 290–292.
Treutwein, B. (1995, August).Error estimates for the parameters of psychometric functions from a single session. Poster presented at the European Conference of Visual Perception, Tübingen.
Treutwein, B., & Strasburger, H. (1999, September).Assessing the variability of psychometric functions. Paper presented at the European Mathematical Psychology Meeting, Mannheim.
Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit.Perception & Psychophysics,63, 1293–1313.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wichmann, F.A., Hill, N.J. The psychometric function: II. Bootstrap-based confidence intervals and sampling. Perception & Psychophysics 63, 1314–1329 (2001). https://doi.org/10.3758/BF03194545
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03194545