Abstract
Over the past decade, functional Magnetic Resonance Imaging (fMRI) has emerged as a powerful new instrument to collect vast quantities of data about activity in the human brain. A typical fMRI experiment can produce a three-dimensional image related to the human subject's brain activity every half second, at a spatial resolution of a few millimeters. As in other modern empirical sciences, this new instrumentation has led to a flood of new data, and a corresponding need for new data analysis methods. We describe recent research applying machine learning methods to the problem of classifying the cognitive state of a human subject based on fRMI data observed over a single time interval. In particular, we present case studies in which we have successfully trained classifiers to distinguish cognitive states such as (1) whether the human subject is looking at a picture or a sentence, (2) whether the subject is reading an ambiguous or non-ambiguous sentence, and (3) whether the word the subject is viewing is a word describing food, people, buildings, etc. This learning problem provides an interesting case study of classifier learning from extremely high dimensional (105 features), extremely sparse (tens of training examples), noisy data. This paper summarizes the results obtained in these three case studies, as well as lessons learned about how to successfully apply machine learning methods to train classifiers in such settings.
Article PDF
Similar content being viewed by others
References
Aguirre, G. K., Zarahn, E., & D'Esposito, M. (1998). An area within human ventral cortex sensitive to building stimuli: Evidence and implications. Neuron, 21, 373–383.
Anderson, J. R. et al. (2004). An information-processing model of the BOLD response in symbol manipulation tasks. Psychonomic Bulletin and Review (in press).
Battig, W. F., & Montague, W. E. (1968). Category norms for verbal items in 56 categories: A replication and extension of the Connecticut norms. Journal of Experimental Psychology Monograph, 80:3, 1–46.
Blankertz, B., Curio, G., & Mller, K. R. (2002). Classifying single trial EEG: Towards brain computer interfacing. Advances in Neural Inf. Proc. Systems (NIPS 2001), 14, 157–164.
Bly, B. M. (2001). When you have a General Linear Hammer, every fMRI time-series looks like independent identically distributed nails. Concepts and Methods in NeuroImaging Workshop, NIPS.
Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Journal of data Mining and Knowledge Discovery, 2:2, 121–167.
Caviness, V. S., Kennedy, D. N., Bates, J., & Makris, N. J. (1996). MRI-based parcellation of human neocortex: An anatomically specified method with estimate of reliability. Journal of Cognitive Neuroscience, 8, 566–588.
Chao, L., Haxby, J. V., & Martin, A. (1999). Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature Neuroscience, 2, 913–919.
Chao, L., Weisberg, J., & Martin, A. (2002). Experience-dependent modulation of category-related cortical activity. Cerebral Cortex, 12, 545–551.
Cover, T., & Thomas, J. (1991). Elements of information theory. Wiley and Sons.
Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI) "brain reading": Detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage, 19, 261–270.
Eddy, W. et al. (1998). The challenge of functional magnetic resonance imaging. Journal of Computational and Graphical Statistics, 8:3, 545–558.
Friston, K. J. et al. (1995a). Statistical parametric maps in functional imaging: A general linear approach. Human Brain Mapping, 2, 189–210
Friston, K. J. et al. (1995b). Analysis of fMRI time-series revisited. NeuroImage, 2, 45–53.
Genovese, C. (1999). Statistical inference in functional magnetic resonance imaging. CMU Statistics Tech Report 674.
Goutte, C., Toft, P., Rostrup, E., Nielsen, F. A., & Hansen, L. K. (1998). On clustering fMRi time series. Technical Report IMM-REP-1998-11.
Haxby, J. et al. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293, 2425–2430.
Hojen-Sorensen, P., Hansen, L. K., & Rasmussen, C. E. (1999). Bayesian modeling of fMRI time series. NIPS*99. Denver.
Ishai, A., Ungerleider, L. G., Martin, A., Schouten, J. L., & Haxby, J. V. (1999). Distributed Representation of Objects in the Human Ventral Visual Pathway. Proc. Nat. Acad. Sci. USA, 96, 9379–9384.
Joachims, T. (2001).Astatistical learning model of text classification with support vector machines. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM.
Just, M. A., Carpenter, P.A., & Varma, S. (1999). Computational modeling of high-level cognition and brain function. Human Brain Mapping, 8, 128–136.
Keller, T. A., Just, M. A., & Stenger, V. A. (2001). Reading span and the time-course of cortical activation in sentence-picture verification. Annual Convention of the Psychonomic Society, Orlando, FL.
Kjems, U., Hansen, L., Anderson, J., Frutiger, S., Muley, S., Sidtis, J., Rottenberg, D., & Strother, S. C. (2002). The quantitative evalutation of functional neuroimaging experiments: Mutual information learning curves, NeuroImage, 15, 772–786.
Mason, R., Just, M., Keller, T., & Carpenter, P. (2004). Ambiguity in the Brain: What brain imaging reveals about the processing of syntactically ambiguous sentences, {tiJounal of Experimental Psychology: Learning, Memory, and Cognition} (in press).
McKeown, M. J. et al. (1998). Analysis of fMRI data by blind separation into independent spatial components. {tiHuman Brain Mapping}, 6:3, 160–188.
Menon, R. S., Luknowsky, D. L., & Gati, J. S. (1998). Mental chronometry using latency-resolved functional magnetic resonance imaging. {tiProc. Natl. Acad. Sci (U.S.A.)}, 95, 10902–10907.
Mitchell, T. M. (1997). {atMachine learning}. McGraw-Hill.
Mitchell, T. M., Hutchinson, R., Just, M., Niculescu, S. R., Pereira, F., & Wang, X. (2003). Classifying instantaneous cognitive states from fMRI data. In {tiProceedings of the 2003 Americal Medical Informatics Association Annual Symposium}. Washington D.C.
Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134.
Ng, A. Y., & Jordan, M. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and Naive Bayes., 14.Neural Information Processing Systems
Penny,W. (2001). Mixture models with adaptive spatial priors. Concepts and Methods in NeuroImagingworkshop at NIPS*01, Vancouver, British Columbia, Canada.
Rademacher, J., Galaburda, A. M., Kennedy, D. N., Filipek, P. A., & Caviness, V. S. (1992). Human cerebral cortex: Localization, parcellation, and morphometry with magnetic resonance imaging. Journal of Cognitive Neuroscience, 4, 352–374.
Strother S. C., Anderson, J., Hansen, L., Kjems, U., Kustra, R., Siditis, J., Frutiger, S., Muley, S., LaConte, S., & Rottenberg, D. (2002). The quantitative evaluation of functional neuroimaging experiments: The NPAIRS data analysis framework. Neuroimage, 15, 747–771.
Talairach, J., & Tournoux, P. (1988). Co-planar stereotaxic atlas of the human brain. Thieme, New York.
Wagner, A. D. et al. (1998). Building memories: Remembering and forgetting of verbal experiences as predicted by brain activity. Science, 281, 1188–1191.
Wang, X., Hutchinson, R., & Mitchell, T. M. (2003). Training fMRI classifiers to detect cognitive states across multiple human subjects. In Proceedings of the 2003 Conference on Neural Information Processing Systems, Vancouver.
Yang, Y. (1999). An evaluation of statistical approaches to text categorization. Journal of Information Retrieval, 1:1/2, 67–88.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Mitchell, T.M., Hutchinson, R., Niculescu, R.S. et al. Learning to Decode Cognitive States from Brain Images. Machine Learning 57, 145–175 (2004). https://doi.org/10.1023/B:MACH.0000035475.85309.1b
Issue Date:
DOI: https://doi.org/10.1023/B:MACH.0000035475.85309.1b