Elsevier

NeuroImage

Volume 19, Issue 4, August 2003, Pages 1612-1626
NeuroImage

Regular article
Neural correlates of incongruous visual information: An event-related fMRI study

https://doi.org/10.1016/S1053-8119(03)00111-3Get rights and content

Abstract

Incongruous information is better remembered than ordinary information. This result has been attributed both to semantic incongruity and surprise. To determine the contribution of each factor, we performed a functional magnetic resonance imaging study in which participants viewed pictures depicting ordinary and incongruous objects (e.g., head of a wrench fused onto a sheep body). To maximize surprise we administered novel incongruent pictures infrequently in an initial scan. (This scan also included infrequent color-inverted pictures as a control for frequency.) To obtain a pure measure of the effect of incongruity we conducted a second scan in which participants viewed equal numbers of ordinary and incongruous pictures. Signal increases were greater for incongruous versus ordinary and oddball stimuli throughout the ventral and dorsal visual pathways, and in prefrontal cortex bilaterally. Signal decreases were larger for incongruous than for ordinary stimuli bilaterally in lateral parietal regions. A subset of regions near the right frontal operculum and extending laterally responded only to, or more strongly to, infrequent incongruous pictures. A second, purely behavioral, experiment involving a separate group of participants demonstrated that incongruous pictures were better recognized than ordinary pictures. We interpret our results as suggesting that, although correlates of a surprise response can be observed, better memory for incongruous visual information is attributable mainly to more processing and, consequently, better encoding.

Introduction

Incongruity is the quality generated by the violation of norms and expectations built up over a lifetime. Multiple behavioral studies have shown a memory advantage for incongruous versus commonplace material (e.g., Riefer and LaMay, 1998). This result is commonly known as the “bizarreness effect.” To provide a better understanding of the neural correlates of this effect, we conducted a functional magnetic resonance imaging (fMRI) experiment directly contrasting responses evoked by incongruous versus ordinary visual stimuli. A corresponding memory advantage was demonstrated for incongruous over ordinary stimuli in a behavioral recognition experiment.

Previous studies have established that semantically incongruous sentences (e.g., “The soldier licked the kittens”) are better recalled than ordinary sentences (e.g., “The man read a book”) Cornoldi et al 1988, Hirshman et al 1989, McDaniel and Einstein 1986, Nicolas and Marchal 1996, Riefer and LaMay 1998, Worthen and Marshall 1996. Incongruity in these studies typically is induced by violating semantic consistency, for example, by attributing human actions to animals and artifacts or vice versa. Subjects typically are asked to read incongruous and ordinary sentences with attention to semantic content. A memory advantage on subsequent testing for incongruous material has so far been reliably demonstrated mostly with recall (as opposed to recognition) tasks and with mixed (including incongruous and ordinary items) rather than pure lists.

Several mutually nonexclusive interpretations of the bizarreness effect are currently under debate (see, e.g., Worthen et al., 2000). One possibility is that incongruous stimuli elicit extra (more elaborated) processing because they are more difficult to make sense of in the context of expected properties and semantic norms (the elaboration hypothesis) Merry 1980, Wollen and Cox 1981. According to this view, the extra elaboration given to incongruous items strengthens their memory trace much as deep, semantically elaborated tasks encourage memorization in standard memory paradigms (e.g., Craik and Lockhart 1972, Craik and Tulving 1975. Related to this is the finding that the memory benefits of incongruity are due in part to distinctiveness in the encoding context Hunt and Elliot 1980, McDaniel and Einstein 1986. Another possibility is that incongruous stimuli elicit a surprise response due to the violation of expectations, which enhances contextual cues available for later recall (the surprise hypothesis; Hirshman et al., 1989).

The physiological basis of the bizarreness effect is incompletely understood. Relevant data derive from studies of electrophysiological responses to incongruous stimuli. The N400 potential is a central-parietal scalp negativity that peaks 400 ms after the presentation of (1) semantically incongruous words in the context of sentences (see Kutas and Hillyard, 1983), or (2) semantically unrelated words or pictures (see Holcomb, 1988, and McPherson and Holcomb, 1999). The N400 phenomenon is thought to be a correlate of the semantic conflict arising when perceived information is different from information expected based on prior context. Three recent fMRI studies Kuperberg et al 2000, Newman et al 2001, Ni et al 2001 have compared responses to semantic (e.g., “The man sailed the hotel to China”) versus syntactic (e.g., “The woman read the with letter attention”) anomalies. All three studies report functional anatomic differences in responses to semantic versus syntactic anomalies, but, in aggregate, do not provide a consistent picture of the specifically activated regions.

The physiological correlates of the surprise response proposed by Hirshman et al. (1989) have not been defined. The “surprise” may originate in an orienting response, which is usually evoked by unexpected stimuli and corresponds to a complex of processes including arousal, capture of attention, and memory formation (Sokolov, 1963). The electrophysiological phenomenon most associated with orienting is the P3 potential, i.e., scalp positivity peaking about 300 ms after infrequent (oddball) stimuli are presented Johnson 1986, Johnson 1993, Matt et al 1992. Two P3 subtypes may be distinguished: Whereas the P3b is strongly modulated by stimulus task relevance and does not habituate, the P3a occurs in response to unexpected or context-violating events, exhibits a somewhat earlier latency and more anterior scalp distribution, and habituates rapidly Knight and Nakada 1998, Knight and Scabini 1998. Several fMRI correlates of the P3a have recently been reported suggesting that the hippocampus as well as the temporal, prefrontal, and parietal cortices contribute to the generation of this potential Kirino et al 2000, Knight 1996, McCarthy et al 1997, Strange et al 2000.

In a series of two studies, we explored the neural and behavioral correlates of incongruous visual material to test whether such stimuli elicit extra elaboration and/or a surprise response. Incongruous pictures incorporating semantic violations concerning artifacts and living things were generated by combining incongruous parts (e.g., head of cat on body of lobster, kangaroo with the head replaced by the hose/nozzle assembly of a gasoline pump). One condition was designed to examine the response to incongruous materials under conditions usually used in previous behavioral memory experiments. In this condition (frequent incongruous, or Freq-Inc condition), incongruous and ordinary pictures were presented with equal frequency. Such a condition was expected to allow examination of additional encoding activity (if any) related to incongruity. Another condition (infrequent incongruous, or Inf-Inc condition) was designed to examine the surprise response (if any) associated with the perception of incongruous pictures. We tried to maximize the possibility of a surprise response by (1) presenting only few incongruous pictures among several ordinary pictures, and (2) always running this condition first so that these incongruous pictures were the first seen in the experiment. A small number of oddball stimuli (color inverted but otherwise ordinary pictures) were also included as a control for frequency effects.

Although firm predictions are not possible, the elaboration hypothesis tentatively predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying the processing of these stimuli. This extra processing should occur independently of the frequency of the incongruous pictures. Accordingly, data supporting this hypothesis should show greater activity for incongruous pictures across our conditions in visual extrastriate cortex and prefrontal cortex, reflecting visual and semantic encoding (see Buckner et al 1999, Fletcher et al 1998, Tulving et al 1994. The surprise hypothesis predicts greater activity for incongruous pictures than for ordinary pictures in areas underlying emotional responses such as the thalamus, the amygdala, and/or the frontal operculum (see Strange et al., 2000). This emotional response should be enhanced (or only appear) when incongruous stimuli are novel and infrequent (Inf-Inc condition).

In addition to the fMRI study, we conducted a separate experiment to examine the effect of stimulus incongruity on recognition memory. The study phase of this behavioral experiment was closely matched to the Freq-Inc condition of the fMRI experiment. A novel sequential recognition design was implemented that provided sensitive measures of recognition accuracy as well as a measure of recognition confidence. We hypothesized that incongruous stimuli would be recognized more accurately and more quickly than ordinary stimuli.

Section snippets

Participants

Twenty-four participants (13 females, mean age = 21.8 years) were recruited from the Washington University community. All participants had normal or corrected-to-normal vision, were native English speakers, showed a strong right-handed preference as measured by the Edinburg Handedness Inventory (Raczkowski et al., 1974), and reported no history of significant neurological problems. Participants were paid and provided informed consent in accordance with guidelines set by the Washington

Experiment 2: recognition memory for incongruous pictures

Experiment 2 was designed to assess recognition memory for the stimuli used in Experiment 1. To exclude frequency effects it was necessary to study a group of participants separate from those in the fMRI experiment. In accordance with standard practice, both the study and test phases of Experiment 2 included equal numbers of incongruous and ordinary pictures (as in the fMRI Freq-Inc condition). A recognition memory design with a novel response method was used.

General discussion

Neural and behavioral correlates of visual incongruity were explored to better understand how such information is processed and why it is remembered better than ordinary information. First, an extensive network of regions was differentially activated by incongruous pictures both during frequent and infrequent presentation contexts. Regions modulated by incongruous pictures spanned the ventral and dorsal visual pathways bilaterally, the frontal cortex along the inferior frontal gyrus, parietal

Acknowledgements

Support was provided by grants from the James S. McDonnell Foundation. We thank Laura E. Williams for her help during the scanning sessions.

References (72)

  • D. Raczkowski et al.

    Reliability and validity of some handedness questionnaire items

    Neurospsychologia

    (1974)
  • M.D. Rugg et al.

    Event-related potentials and the recollection of associative information

    Cogn. Brain Res.

    (1996)
  • B.A. Strange et al.

    Brain mechanisms for detecting perceptual, semantic and emotional deviance

    Neuroimage

    (2000)
  • W.C. West et al.

    Event-related potentials during discourse-level semantic integration of complex pictures

    Cogn. Brain Res.

    (2002)
  • K.A. Wollen et al.

    Bizarreness versus integration of mental images as determinants of learning

    Cogn. Psychol.

    (1972)
  • P.A. Bandettini et al.

    Processing strategies for time-course data sets in functional MRI of the human brain

    Magn. Reson. Med.

    (1993)
  • J.B. Brewer et al.

    Making memoriesbrain activity that predicts how well visual experience will be remembered

    Science

    (1998)
  • R.L. Buckner et al.

    Frontal cortex contributes to human memory formation

    Nat. Neurosci.

    (1999)
  • M.A. Burock et al.

    Randomized event-related experimental designs allow for extremely rapid presentation rates using functional MRI

    Neuroreport

    (1998)
  • Clark, V.P., Fannon, -S., Lai, S., Benson, R., Bauer, L., 2000. Responses to rare visual target and distractor stimuli...
  • J.D. Cohen et al.

    Psyscopean interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers

    Behav. Res. Methods Instr. Computers

    (1993)
  • C. Cornoldi et al.

    The influence of the nature of material and of mental operations on the occurrence of the bizarreness effect

    Q. J. Exp. Psychol.

    (1988)
  • S.M. Courtney et al.

    Transient and sustained activity in a distributed neural system for human working memory

    Nature

    (1997)
  • F.I. Craik et al.

    Depth of processing and the retention of words in episodic memory

    J. Exp. Psychol. Gen.

    (1975)
  • J. Engelkamp et al.

    Bizarreness effects in verbal tasks and subject-performed tasks

    Eur. J. Cogn. Psych.

    (1993)
  • M. Fabiani et al.

    P300 and recall in an incidental memory paradigm

    Psychophysiology

    (1986)
  • G.R. Fink et al.

    The neural consequences of conflict between intention and the senses

    Brain

    (1999)
  • P.C. Fletcher et al.

    The functional roles of prefrontal cortex in episodic memory. I. Encoding

    Brain

    (1998)
  • G. Ganis et al.

    The search for “common sense:”an electrophysiological study of the comprehension of words and pictures in reading

    J. Cogn. Neurosci.

    (1996)
  • C. Gerlach et al.

    Perceptual differentiation and category effects in normal object recognition. A PET study

    Brain

    (1999)
  • D.A. Gusnard et al.

    Searching for a baselinefunctional imaging and the resting human brain

    Nat. Rev. Neurosci.

    (2001)
  • R.R. Hunt et al.

    The role of nonsemantic information in memoryorthographic distinctiveness effects upon retention

    J. Exp. Psychol. Gen.

    (1980)
  • R. Johnson

    A triarchic model of P300 amplitude

    Psychophysiology

    (1986)
  • R. Johnson

    On the neural generators of the P300 component of the event-related potential

    Psychophysiology

    (1993)
  • K.A. Kiehl et al.

    Neural sources involved in auditory target detection and novelty processingan event-related fMRI study

    Psychophysiology

    (2001)
  • E. Kirino et al.

    Prefrontal activation evoked by infrequent target and novel stimuli in a visual target detection taskan event-related functional magnetic resonance imaging study

    J. Neurosci.

    (2000)
  • Cited by (84)

    • Memory influences visual cognition across multiple functional states of interactive cortical dynamics

      2019, Psychology of Learning and Motivation - Advances in Research and Theory
    • Auditory conflict and congruence in frontotemporal dementia

      2017, Neuropsychologia
      Citation Excerpt :

      These models form predictions about the environment based on current and previous sensory experience, detect unexpected or ‘surprising’ events as prediction errors and adjust behavioural output to minimise those errors (Friston, 2009; Moran et al., 2004). The underlying neural computations engage large-scale brain networks: these networks encompass posterior cortical areas that parse sensory traffic into component objects; medial fronto-parietal cortices that direct and control attention and the detection of salient sensory events according to behavioural context; antero-medial temporal areas that store previously learned knowledge and schemas about sensory objects and regularities; insular and prefrontal cortices that implement and assess violations in rule-based algorithms; and striatal and other subcortical structures that code emotional and physiological value (Christensen et al., 2011; Cohen, 2014; Dieguez-Risco et al., 2015; Dzafic et al., 2016; Gauvin et al., 2016; Groussard et al., 2010; Henderson et al., 2016; Jakuszeit et al., 2013; Klasen et al., 2011; Merkel et al., 2015; Michelon et al., 2003; Nazimek et al., 2013; Remy et al., 2014; Ridderinkhof et al., 2004; Rosenbloom et al., 2012; Silvetti et al., 2014; Watanabe et al., 2014), Within this distributed circuitry, separable mechanisms have been identified for the processing of semantic and affective congruence (Dieguez-Risco et al., 2015) and for elementary versus more abstract levels of incongruity decoding (Paavilainen, 2013). On clinical as well as neuroanatomical grounds, abnormal processing of conflict and congruence is a candidate generic mechanism of disease phenotypes in the frontotemporal dementias (Warren et al., 2013).

    • The parietal memory network activates similarly for true and associative false recognition elicited via the DRM procedure

      2017, Cortex
      Citation Excerpt :

      Removal of the linear slope on a voxel-by-voxel basis corrected for frequency drift (Bandettini, Jesmanowicz, Wong, & Hyde, 1993), and whole brain normalization to a common mode of 1000 facilitated comparisons across subjects. Functional data were transformed to stereotaxic atlas space (Talairach & Tournoux, 1988) by aligning each subjects' T1-weighted scan to a target T1-weighted template (711-2B) using a series of affine transforms (Fox et al., 2005; Michelon, Snyder, Buckner, McAvoy, & Zacks, 2003). Although analyses were conducted in 711-2B space, all coordinates reported have been transformed to MNI152 space using software written by Avi Snyder.

    View all citing articles on Scopus
    View full text