Elsevier

Neuropsychologia

Volume 50, Issue 4, March 2012, Pages 458-469
Neuropsychologia

Multi-voxel patterns of visual category representation during episodic encoding are predictive of subsequent memory

https://doi.org/10.1016/j.neuropsychologia.2011.09.002Get rights and content

Abstract

Successful encoding of episodic memories is thought to depend on contributions from prefrontal and temporal lobe structures. Neural processes that contribute to successful encoding have been extensively explored through univariate analyses of neuroimaging data that compare mean activity levels elicited during the encoding of events that are subsequently remembered vs. those subsequently forgotten. Here, we applied pattern classification to fMRI data to assess the degree to which distributed patterns of activity within prefrontal and temporal lobe structures elicited during the encoding of word–image pairs were diagnostic of the visual category (Face or Scene) of the encoded image. We then assessed whether representation of category information was predictive of subsequent memory. Classification analyses indicated that temporal lobe structures contained information robustly diagnostic of visual category. Information in prefrontal cortex was less diagnostic of visual category, but was nonetheless associated with highly reliable classifier-based evidence for category representation. Critically, trials associated with greater classifier-based estimates of category representation in temporal and prefrontal regions were associated with a higher probability of subsequent remembering. Finally, consideration of trial-by-trial variance in classifier-based measures of category representation revealed positive correlations between prefrontal and temporal lobe representations, with the strength of these correlations varying as a function of the category of image being encoded. Together, these results indicate that multi-voxel representations of encoded information can provide unique insights into how visual experiences are transformed into episodic memories.

Highlights

► We used pattern classification to assess category representation during encoding. ► Category representation was robust in prefrontal and temporal lobe structures. ► Strength of representations was predictive of subsequent memory. ► Prefrontal and temporal representations were correlated, suggesting interactivity.

Introduction

For more than a decade, functional neuroimaging studies of human memory have considered how neural responses elicited during encoding relate to later memory outcomes. Most frequently, this has been addressed through univariate analysis of functional magnetic resonance imaging (fMRI) data, testing for individual voxels (or clusters of voxels) that show greater mean activity during the encoding of items that will be later remembered relative to items that will be forgotten—a subsequent memory effect (Brewer et al., 1998, Wagner et al., 1998b). Such studies have regularly implicated lateral prefrontal cortex and the medial temporal lobe in successful memory formation (for reviews, see Blumenfeld and Ranganath, 2007, Kim, 2011, Paller and Wagner, 2002, Spaniol et al., 2009). These observations are complemented by neuropsychological investigations that demonstrate the necessity of prefrontal (Shimamura, 1995, Wheeler et al., 1995) and medial temporal lobe structures for episodic memory (Cohen and Eichenbaum, 1994, Scoville and Milner, 1957, Squire, 1992). Despite the obvious importance of these structures for event memory, there remains considerable ambiguity regarding the cognitive processes and neural mechanisms that are reflected by greater fMRI activation during the encoding of subsequently remembered items. One approach that offers the potential for new insight into these processes is multi-voxel pattern analysis (MVPA). By considering distributed patterns of neural activity, MVPA represents a highly sensitive method for fMRI data analysis and is ideally suited for assessing the similarities or differences between neural states across events (Norman, Polyn, Detre, & Haxby, 2006).

To date, only a handful of studies have applied MVPA to evaluate distributed patterns of neural activity that give rise to episodic encoding success (for review, see Rissman & Wagner, in press). In one recent study, Watanabe et al. (2011) demonstrated that multi-voxel patterns within the medial temporal lobe could be used to classify individual stimuli as subsequently remembered vs. forgotten. Two additional studies used multivariate approaches to consider more subtle questions about how neural pattern similarity across repetitions of a stimulus, or across different stimuli, relate to later memory. In one study, Xue et al. (2010) found that neural pattern similarity across repeated presentations of a stimulus was positively associated with later memory for that stimulus (c.f., Wagner, Maril, & Schacter, 2000). In another study, Jenkins and Ranganath (2010) found that when an encoding event was associated with patterns of neural activity that were relatively dissimilar to neighboring events, that event was more likely to be later associated with successful memory for its temporal context.

An alternative, and to our knowledge unexplored, application of MVPA to the study of episodic encoding success is to consider how the neural representation of stimulus features during encoding relates to later memory. That is, does the strength with which an event is represented positively relate to later memory for that event? If so, are representations in some neural structures more predictive of later memory success than representations in other structures? In the present study, we addressed these questions by using MVPA to (a) measure the neural representation of the visual category of an encoded stimulus, and (b) assess how representational strength within prefrontal and temporal lobe structures relates to subsequent memory outcomes.

We scanned subjects as they formed memories for arbitrary associations between cue words and images of either well-known people (Faces) or well-known locations (Scenes). Using a subset of the encoding data, we trained an MVPA classifier to discriminate fMRI activity patterns associated with Face vs. Scene trials. This classifier was then used to index the relative manifestation of these category-selective activity patterns on each of the remaining encoding trials, and this process was iteratively repeated until all trials had been a part of both the training and testing sets. By performing these pattern classification analyses on the data from each of a set of anatomically defined regions of interest (ROIs) within prefrontal cortex and the temporal lobes, we assessed how classifier-based evidence for target information (Face vs. Scene representations) related to the likelihood that subjects would later recall the relevant Face/Scene image when probed with its associated cue word. We predicted that the degree to which encoding trials were associated with category-specific activity patterns would be an indicator of stimulus representational strength at encoding, and hence a predictor of subsequent memory. We also assessed whether this putative relationship between representational strength and subsequent memory differed across prefrontal and temporal lobe structures.

On the one hand, the representation of visual categories, such as Faces and Scenes, has been most clearly established within temporal lobe structures (Epstein and Kanwisher, 1998, Haxby et al., 2001, Kanwisher et al., 1997, Puce et al., 1996, Weiner and Grill-Spector, 2010), and there is some evidence for category-selective subsequent memory effects in temporal lobe areas (e.g., Kirchhoff et al., 2000, Prince et al., 2009). On the other hand, prefrontal cortex is regularly implicated in successful episodic encoding and, while visual category representation in prefrontal cortex has not been well defined through fMRI studies, recordings from individual neurons in monkeys have provided compelling evidence for category-level representations of visual stimuli in lateral prefrontal cortex (e.g., Freedman, Riesenhuber, Poggio, & Miller, 2001). Thus, while we predicted that category representation would be most robust within temporal lobe structures, we also anticipated that category representation would be observed in prefrontal cortex and closely tied to subsequent memory outcomes. Moreover, consistent with the view that prefrontal cortex operates upon the products of—and potentially influences—posterior representations (e.g., Miller & Cohen, 2001), we asked whether, on a trial-by-trial basis, the strength of representations in temporal lobe structures was correlated with the strength of representations within prefrontal structures.

Section snippets

Subjects

Eighteen subjects (10 female) participated in the study. All were right-handed native English speakers between the ages of 18 and 27 yrs. Subjects received $20/h for their participation. Informed consent was obtained according to procedures approved by the Stanford Institutional Review Board.

Materials and procedure

The experiment consisted of alternating blocks of encoding and retrieval, all conducted during fMRI scanning. During encoding, subjects viewed nouns (cues; e.g., ‘flag’ ‘couch’) presented above grayscale

Behavioral results

Subjects were able to recall the target image category (either “specific” or “general” memory for the image) on the majority of retrieval trials (M = 79.2%); hereinafter ‘Remembered’ items. ‘Forgotten’ items corresponded to trials on which subjects responded “Don’t Know” (M = 13.2%) or responded with the incorrect category (M = 5.7%). Trials for which subjects failed to respond (M = 2.0%) were excluded from subsequent memory analyses. The percentage of items Remembered did not differ for Face vs. Scene

Discussion

The present study yielded three main findings. First, during encoding of words paired with images of Faces or Scenes, MVPA revealed that information highly diagnostic of visual category was present in distributed patterns of activity in temporal and prefrontal structures. Second, representation of category information during encoding was positively associated with subsequent memory outcomes. Notably, in prefrontal cortex, both trial-by-trial variation and individual differences in

Acknowledgements

This work was supported by grants from the National Institute of Mental Health (5R01-MH080309 and 5R01-MH076932 to A.D.W.), and the National Eye Institute (EY019624-02 to B.A.K.).

References (75)

  • K. Sergerie et al.

    A face to remember: Emotional expression modulates prefrontal activity during memory formation

    Neuroimage

    (2005)
  • J. Spaniol et al.

    Event-related fMRI studies of episodic encoding and retrieval: Meta-analyses using activation likelihood estimation

    Neuropsychologia

    (2009)
  • N.B. Turk-Browne et al.

    Linking implicit and explicit memory: Common encoding factors and shared representations

    Neuron

    (2006)
  • T. Watanabe et al.

    Prediction of subsequent recognition performance using brain activity in the medial temporal lobe

    Neuroimage

    (2011)
  • K.S. Weiner et al.

    Sparsely-distributed organization of face and limb activations in human ventral temporal cortex

    Neuroimage

    (2010)
  • C.R. Wilson et al.

    Functional localization within the prefrontal cortex: Missing the forest for the trees?

    Trends in Neurosciences

    (2010)
  • T. Awipi et al.

    Content-specific source encoding in the human medial temporal lobe

    Journal of Experimental Psychology: Learning, Memory, and Cognition

    (2008)
  • J.T. Baker et al.

    Neural correlates of verbal memory encoding during semantic and structural processing tasks

    NeuroReport

    (2001)
  • M. Bar et al.

    Top-down facilitation of visual recognition

    Proceedings of the National Academy of Sciences of the United States of America

    (2006)
  • R.S. Blumenfeld et al.

    Prefrontal cortex and long-term memory encoding: An integrative review of findings from neuropsychology and neuroimaging

    Neuroscientist

    (2007)
  • J.B. Brewer et al.

    Making memories: Brain activity that predicts how well visual experience will be remembered

    Science

    (1998)
  • N.J. Cohen et al.

    Memory, amnesia, and the hippocampal system

    (1994)
  • S.M. Courtney et al.

    An area specialized for spatial working memory in human frontal cortex

    Science

    (1998)
  • J.B. Demb et al.

    Semantic encoding and retrieval in the left inferior prefrontal cortex: A functional MRI study of task difficulty and process specificity

    The Journal of Neuroscience

    (1995)
  • J. Duncan

    An adaptive coding model of neural function in prefrontal cortex

    Nature Reviews: Neuroscience

    (2001)
  • R. Epstein et al.

    A cortical representation of the local visual environment

    Nature

    (1998)
  • D.J. Freedman et al.

    Categorical representation of visual stimuli in the primate prefrontal cortex

    Science

    (2001)
  • D.J. Freedman et al.

    A comparison of primate prefrontal and inferior temporal cortices during visual categorization

    The Journal of Neuroscience

    (2003)
  • K.J. Friston et al.

    Functional connectivity: The principal-component analysis of large (PET) data sets

    Journal of Cerebral Blood Flow and Metabolism

    (1993)
  • A.J. Golby et al.

    Material-specific lateralization in the medial temporal lobe and prefrontal cortex during memory encoding

    Brain

    (2001)
  • C.L. Grady et al.

    Neural correlates of the episodic encoding of pictures and words

    Proceedings of the National Academy of Sciences of the United States of America

    (1998)
  • J.V. Haxby et al.

    Distributed and overlapping representations of faces and objects in ventral temporal cortex

    Science

    (2001)
  • S.M. Hayes et al.

    The effect of scene context on episodic object recognition: Parahippocampal cortex mediates memory encoding and retrieval success

    Hippocampus

    (2007)
  • L.J. Jenkins et al.

    Prefrontal and medial temporal lobe activity at encoding predicts temporal context memory

    The Journal of Neuroscience

    (2010)
  • M.K. Johnson et al.

    fMRI evidence for an organization of prefrontal cortex by both type of process and type of information

    Cerebral Cortex

    (2003)
  • N. Kanwisher et al.

    The fusiform face area: A module in human extrastriate cortex specialized for face perception

    The Journal of Neuroscience

    (1997)
  • C. Kaul et al.

    The gender of face stimuli is represented in multiple regions in the human brain

    Frontiers in Human Neuroscience

    (2011)
  • Cited by (83)

    • Representing the Good and Bad: fMRI signatures during the encoding of multisensory positive, negative, and neutral events

      2022, Cortex
      Citation Excerpt :

      In contrast to the results for emotional stimuli, the MVPA results revealed that similarity was higher for subsequent hits than misses for neutral events in both visual and auditory ROIs. These findings are consistent with prior MVPA studies demonstrating that category-selective neural patterning in sensory processing regions is predictive of subsequent memory for neutral information (e.g., face or scene information in ventral temporal cortex; Kuhl et al., 2012). Here, similarity was computed as the difference between within-category (i.e., among neutral events) and between-category (i.e., between neutral and emotional events) correlations, thus the significant subsequent memory effect likely reflects the engagement of distinct processes engaged during successful neutral relative to emotional encoding.

    • Spectral pattern similarity analysis: Tutorial and application in developmental cognitive neuroscience

      2022, Developmental Cognitive Neuroscience
      Citation Excerpt :

      The specificity with which experiences are encoded into neural representations is thought to be directly related to cognition, in particular how accurately and precisely aspects of the experience can later be remembered (Rissman and Wagner, 2012). Specifically, the stability of neural representations across time as well as their distinctiveness from other representations are considered to be crucial for memory success (e.g., Kobelt et al., 2021; Kuhl et al., 2012; Lu et al., 2015; Xue, 2018). Understanding how information is represented in the brain and how the quality of neural representations influences cognition are major goals in cognitive neuroscience (for related reviews, see Rissman and Wagner, 2012; Xue, 2018).

    • Integration and differentiation of hippocampal memory traces

      2020, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      Most of the studies examining pattern similarity across stimuli have reported increased pattern similarity among stimuli with shared content, such as spatial, temporal, encoding context, relative to stimuli with no shared content. Many of these studies have also reported that higher across-stimulus pattern similarity was associated with better memory performance (Aly and Turk-Browne, 2016a; Kuhl et al., 2012; Schlichting et al., 2014; Zeithamova et al., 2018), suggesting that higher similarity was advantageous in the specific tasks used. Another group of studies, however, has reported lower across-item pattern similarity for stimuli with overlapping content.

    • Neural pattern similarity across concept exemplars predicts memory after a long delay

      2020, NeuroImage
      Citation Excerpt :

      Trials were correlated across (rather than within) runs to prevent any spurious correlations that might occur by comparing trials in the same run (Mumford et al., 2014). Within and between similarities were the Fisher-z corrected Pearson’s correlation coefficients from these analyses (Kuhl et al., 2012; LaRocque et al., 2013). Pattern robustness was calculated for each trial, as the difference of within and between similarities.

    View all citing articles on Scopus
    View full text