Regular articleNeural correlates of incongruous visual information: An event-related fMRI study
Introduction
Incongruity is the quality generated by the violation of norms and expectations built up over a lifetime. Multiple behavioral studies have shown a memory advantage for incongruous versus commonplace material (e.g., Riefer and LaMay, 1998). This result is commonly known as the “bizarreness effect.” To provide a better understanding of the neural correlates of this effect, we conducted a functional magnetic resonance imaging (fMRI) experiment directly contrasting responses evoked by incongruous versus ordinary visual stimuli. A corresponding memory advantage was demonstrated for incongruous over ordinary stimuli in a behavioral recognition experiment.
Previous studies have established that semantically incongruous sentences (e.g., “The soldier licked the kittens”) are better recalled than ordinary sentences (e.g., “The man read a book”) Cornoldi et al 1988, Hirshman et al 1989, McDaniel and Einstein 1986, Nicolas and Marchal 1996, Riefer and LaMay 1998, Worthen and Marshall 1996. Incongruity in these studies typically is induced by violating semantic consistency, for example, by attributing human actions to animals and artifacts or vice versa. Subjects typically are asked to read incongruous and ordinary sentences with attention to semantic content. A memory advantage on subsequent testing for incongruous material has so far been reliably demonstrated mostly with recall (as opposed to recognition) tasks and with mixed (including incongruous and ordinary items) rather than pure lists.
Several mutually nonexclusive interpretations of the bizarreness effect are currently under debate (see, e.g., Worthen et al., 2000). One possibility is that incongruous stimuli elicit extra (more elaborated) processing because they are more difficult to make sense of in the context of expected properties and semantic norms (the elaboration hypothesis) Merry 1980, Wollen and Cox 1981. According to this view, the extra elaboration given to incongruous items strengthens their memory trace much as deep, semantically elaborated tasks encourage memorization in standard memory paradigms (e.g., Craik and Lockhart 1972, Craik and Tulving 1975. Related to this is the finding that the memory benefits of incongruity are due in part to distinctiveness in the encoding context Hunt and Elliot 1980, McDaniel and Einstein 1986. Another possibility is that incongruous stimuli elicit a surprise response due to the violation of expectations, which enhances contextual cues available for later recall (the surprise hypothesis; Hirshman et al., 1989).
The physiological basis of the bizarreness effect is incompletely understood. Relevant data derive from studies of electrophysiological responses to incongruous stimuli. The N400 potential is a central-parietal scalp negativity that peaks 400 ms after the presentation of (1) semantically incongruous words in the context of sentences (see Kutas and Hillyard, 1983), or (2) semantically unrelated words or pictures (see Holcomb, 1988, and McPherson and Holcomb, 1999). The N400 phenomenon is thought to be a correlate of the semantic conflict arising when perceived information is different from information expected based on prior context. Three recent fMRI studies Kuperberg et al 2000, Newman et al 2001, Ni et al 2001 have compared responses to semantic (e.g., “The man sailed the hotel to China”) versus syntactic (e.g., “The woman read the with letter attention”) anomalies. All three studies report functional anatomic differences in responses to semantic versus syntactic anomalies, but, in aggregate, do not provide a consistent picture of the specifically activated regions.
The physiological correlates of the surprise response proposed by Hirshman et al. (1989) have not been defined. The “surprise” may originate in an orienting response, which is usually evoked by unexpected stimuli and corresponds to a complex of processes including arousal, capture of attention, and memory formation (Sokolov, 1963). The electrophysiological phenomenon most associated with orienting is the P3 potential, i.e., scalp positivity peaking about 300 ms after infrequent (oddball) stimuli are presented Johnson 1986, Johnson 1993, Matt et al 1992. Two P3 subtypes may be distinguished: Whereas the P3b is strongly modulated by stimulus task relevance and does not habituate, the P3a occurs in response to unexpected or context-violating events, exhibits a somewhat earlier latency and more anterior scalp distribution, and habituates rapidly Knight and Nakada 1998, Knight and Scabini 1998. Several fMRI correlates of the P3a have recently been reported suggesting that the hippocampus as well as the temporal, prefrontal, and parietal cortices contribute to the generation of this potential Kirino et al 2000, Knight 1996, McCarthy et al 1997, Strange et al 2000.
In a series of two studies, we explored the neural and behavioral correlates of incongruous visual material to test whether such stimuli elicit extra elaboration and/or a surprise response. Incongruous pictures incorporating semantic violations concerning artifacts and living things were generated by combining incongruous parts (e.g., head of cat on body of lobster, kangaroo with the head replaced by the hose/nozzle assembly of a gasoline pump). One condition was designed to examine the response to incongruous materials under conditions usually used in previous behavioral memory experiments. In this condition (frequent incongruous, or Freq-Inc condition), incongruous and ordinary pictures were presented with equal frequency. Such a condition was expected to allow examination of additional encoding activity (if any) related to incongruity. Another condition (infrequent incongruous, or Inf-Inc condition) was designed to examine the surprise response (if any) associated with the perception of incongruous pictures. We tried to maximize the possibility of a surprise response by (1) presenting only few incongruous pictures among several ordinary pictures, and (2) always running this condition first so that these incongruous pictures were the first seen in the experiment. A small number of oddball stimuli (color inverted but otherwise ordinary pictures) were also included as a control for frequency effects.
Although firm predictions are not possible, the elaboration hypothesis tentatively predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying the processing of these stimuli. This extra processing should occur independently of the frequency of the incongruous pictures. Accordingly, data supporting this hypothesis should show greater activity for incongruous pictures across our conditions in visual extrastriate cortex and prefrontal cortex, reflecting visual and semantic encoding (see Buckner et al 1999, Fletcher et al 1998, Tulving et al 1994. The surprise hypothesis predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying emotional responses such as the thalamus, the amygdala, and/or the frontal operculum (see Strange et al., 2000). This emotional response should be enhanced (or only appear) when incongruous stimuli are novel and infrequent (Inf-Inc condition).
In addition to the fMRI study, we conducted a separate experiment to examine the effect of stimulus incongruity on recognition memory. The study phase of this behavioral experiment was closely matched to the Freq-Inc condition of the fMRI experiment. A novel sequential recognition design was implemented that provided sensitive measures of recognition accuracy as well as a measure of recognition confidence. We hypothesized that incongruous stimuli would be recognized more accurately and more quickly than ordinary stimuli.
Section snippets
Participants
Twenty-four participants (13 females, mean age = 21.8 years) were recruited from the Washington University community. All participants had normal or corrected-to-normal vision, were native English speakers, showed a strong right-handed preference as measured by the Edinburg Handedness Inventory (Raczkowski et al., 1974), and reported no history of significant neurological problems. Participants were paid and provided informed consent in accordance with guidelines set by the Washington
Experiment 2: recognition memory for incongruous pictures
Experiment 2 was designed to assess recognition memory for the stimuli used in Experiment 1. To exclude frequency effects it was necessary to study a group of participants separate from those in the fMRI experiment. In accordance with standard practice, both the study and test phases of Experiment 2 included equal numbers of incongruous and ordinary pictures (as in the fMRI Freq-Inc condition). A recognition memory design with a novel response method was used.
General discussion
Neural and behavioral correlates of visual incongruity were explored to better understand how such information is processed and why it is remembered better than ordinary information. First, an extensive network of regions was differentially activated by incongruous pictures both during frequent and infrequent presentation contexts. Regions modulated by incongruous pictures spanned the ventral and dorsal visual pathways bilaterally, the frontal cortex along the inferior frontal gyrus, parietal
Acknowledgements
Support was provided by grants from the James S. McDonnell Foundation. We thank Laura E. Williams for her help during the scanning sessions.
References (72)
- et al.
Dissociating working memory from task difficulty in human prefrontal cortex
Neuropsychologia
(1997) - et al.
Levels of processinga framework for memory research
J. Verb. Learn. Behav.
(1972) - et al.
Functional MRI studies of spatial and nonspatial working memory
Cogn. Brain Res.
(1998) - et al.
The novelty P3an event-related brain potential (ERP) sign of the brain’s evaluation of novelty
Neurosci. Behav. Rev.
(2001) - et al.
An investigation of paradoxical memory effects
J. Memory Lang.
(1989) Automatic and attentional processingan event-related brain potential analysis of semantic priming
Brain Lang.
(1988)- et al.
Recognizing words, pictures, and conceptsa comparison of lexical, object, and reality decisions
J. Verb. Learn. Verb. Behav.
(1984) - et al.
Characterizing the hemodynamic responseeffects of presentation rate, sampling procedure, and the possibility of ordering brain activity based on relative timing
Neuroimage
(2000) - et al.
Event-related brain potentials during initial encoding and recognition memory of congruous and incongruous words
J. Memory Lang.
(1986) - et al.
Anatomic localization and quantitative analysis of gradient refocused echo-planar fMRI susceptibility artifacts
Neuroimage
(1997)
Reliability and validity of some handedness questionnaire items
Neurospsychologia
Event-related potentials and the recollection of associative information
Cogn. Brain Res.
Brain mechanisms for detecting perceptual, semantic and emotional deviance
Neuroimage
Event-related potentials during discourse-level semantic integration of complex pictures
Cogn. Brain Res.
Bizarreness versus integration of mental images as determinants of learning
Cogn. Psychol.
Processing strategies for time-course data sets in functional MRI of the human brain
Magn. Reson. Med.
Making memoriesbrain activity that predicts how well visual experience will be remembered
Science
Frontal cortex contributes to human memory formation
Nat. Neurosci.
Randomized event-related experimental designs allow for extremely rapid presentation rates using functional MRI
Neuroreport
Psyscopean interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers
Behav. Res. Methods Instr. Computers
The influence of the nature of material and of mental operations on the occurrence of the bizarreness effect
Q. J. Exp. Psychol.
Transient and sustained activity in a distributed neural system for human working memory
Nature
Depth of processing and the retention of words in episodic memory
J. Exp. Psychol. Gen.
Bizarreness effects in verbal tasks and subject-performed tasks
Eur. J. Cogn. Psych.
P300 and recall in an incidental memory paradigm
Psychophysiology
The neural consequences of conflict between intention and the senses
Brain
The functional roles of prefrontal cortex in episodic memory. I. Encoding
Brain
The search for “common sense:”an electrophysiological study of the comprehension of words and pictures in reading
J. Cogn. Neurosci.
Perceptual differentiation and category effects in normal object recognition. A PET study
Brain
Searching for a baselinefunctional imaging and the resting human brain
Nat. Rev. Neurosci.
The role of nonsemantic information in memoryorthographic distinctiveness effects upon retention
J. Exp. Psychol. Gen.
A triarchic model of P300 amplitude
Psychophysiology
On the neural generators of the P300 component of the event-related potential
Psychophysiology
Neural sources involved in auditory target detection and novelty processingan event-related fMRI study
Psychophysiology
Prefrontal activation evoked by infrequent target and novel stimuli in a visual target detection taskan event-related functional magnetic resonance imaging study
J. Neurosci.
Cited by (84)
Effect of the congruity of emotional contexts at encoding on source memory: Evidence from ERPs
2022, International Journal of PsychophysiologyMemory influences visual cognition across multiple functional states of interactive cortical dynamics
2019, Psychology of Learning and Motivation - Advances in Research and TheoryAuditory conflict and congruence in frontotemporal dementia
2017, NeuropsychologiaCitation Excerpt :These models form predictions about the environment based on current and previous sensory experience, detect unexpected or ‘surprising’ events as prediction errors and adjust behavioural output to minimise those errors (Friston, 2009; Moran et al., 2004). The underlying neural computations engage large-scale brain networks: these networks encompass posterior cortical areas that parse sensory traffic into component objects; medial fronto-parietal cortices that direct and control attention and the detection of salient sensory events according to behavioural context; antero-medial temporal areas that store previously learned knowledge and schemas about sensory objects and regularities; insular and prefrontal cortices that implement and assess violations in rule-based algorithms; and striatal and other subcortical structures that code emotional and physiological value (Christensen et al., 2011; Cohen, 2014; Dieguez-Risco et al., 2015; Dzafic et al., 2016; Gauvin et al., 2016; Groussard et al., 2010; Henderson et al., 2016; Jakuszeit et al., 2013; Klasen et al., 2011; Merkel et al., 2015; Michelon et al., 2003; Nazimek et al., 2013; Remy et al., 2014; Ridderinkhof et al., 2004; Rosenbloom et al., 2012; Silvetti et al., 2014; Watanabe et al., 2014), Within this distributed circuitry, separable mechanisms have been identified for the processing of semantic and affective congruence (Dieguez-Risco et al., 2015) and for elementary versus more abstract levels of incongruity decoding (Paavilainen, 2013). On clinical as well as neuroanatomical grounds, abnormal processing of conflict and congruence is a candidate generic mechanism of disease phenotypes in the frontotemporal dementias (Warren et al., 2013).
The parietal memory network activates similarly for true and associative false recognition elicited via the DRM procedure
2017, CortexCitation Excerpt :Removal of the linear slope on a voxel-by-voxel basis corrected for frequency drift (Bandettini, Jesmanowicz, Wong, & Hyde, 1993), and whole brain normalization to a common mode of 1000 facilitated comparisons across subjects. Functional data were transformed to stereotaxic atlas space (Talairach & Tournoux, 1988) by aligning each subjects' T1-weighted scan to a target T1-weighted template (711-2B) using a series of affine transforms (Fox et al., 2005; Michelon, Snyder, Buckner, McAvoy, & Zacks, 2003). Although analyses were conducted in 711-2B space, all coordinates reported have been transformed to MNI152 space using software written by Avi Snyder.