Multisensory contributions to the perception of motion

https://doi.org/10.1016/S0028-3932(03)00185-4Get rights and content

Abstract

The ability to process motion is crucial for coherent perception and action. While the majority of studies have focused on the unimodal factors that influence motion perception (see, for example, the other chapters in this Special Issue), some researchers have also investigated the extent to which information presented in one sensory modality can affect the perception of motion for stimuli presented in another modality. Although early studies often gave rise to mixed results, the development of increasingly sophisticated psychophysical paradigms are now enabling researchers to determine the spatiotemporal constraints on multisensory interactions in the perception of motion. Recent findings indicate that these interactions stand over-and-above the multisensory interactions documented previously for static stimuli, such as the oft-cited ‘ventriloquism’ effect. Neuroimaging and neuropsychological studies are also beginning to elucidate the network of neural structures responsible for the processing of motion information in the different sensory modalities, an important first step that will ultimately lead to the determination of the neural substrates underlying these multisensory contributions to motion perception.

Introduction

Our ability to extract information regarding movement is fundamental both for deriving and maintaining useful representations of the environment, and also for the planning and execution of action. As for many other perceptual domains, information regarding the movement of stimuli (i.e. their direction, speed, etc.) is often available via several sensory modalities simultaneously (think, for example, of tracking a partially occluded animal moving through the undergrowth). While vision may provide the most salient information with regard to stimulus motion, audition and somatosensation can also provide important cues, particularly when stimuli are occluded, or else move outside the current field of view (such as when objects move behind the head).

Despite its obvious adaptive importance, and after more than a century of intermittent research on this topic (see Ryan, 1940, Urbantschitsch, 1902; Zietz & Werner, 1927, for early work), our knowledge concerning how motion signals from different sensory modalities are integrated is still fairly limited. Nevertheless, there has been a rapid growth of interest in this area over the last few years, with researchers moving from the phenomenological approaches that characterized many early studies (e.g. Zapparoli & Reatto, 1969) to the development of increasingly sophisticated psychophysical paradigms that are currently enabling researchers to investigate the spatio-temporal constraints on multisensory contributions to the perception of dynamic stimuli.1 Our understanding of these constraints on human perception has also been complemented by recent cognitive neuroscience studies that have begun to reveal the networks of neural structures involved in the integration of motion information across the senses.

In the present review, we start by evaluating the behavioural evidence regarding multisensory contributions to the perception of motion, as well as outlining some of the key constraints affecting such integration. We then proceed to highlight recent neuroimaging and neurophysiological data regarding the neural substrates of motion processing that may underlie some of these multisensory perceptual interactions.2

Section snippets

Multisensory contributions to the perception of motion: behavioral evidence

For many years, researchers have investigated the question of whether the presentation of stimuli in one modality (either moving or stationary) can affect the perception of motion of stimuli presented in another modality (see Soto-Faraco & Kingstone, in press, for a detailed review). Many of these behavioral studies have examined whether the presentation of a static stimulus in one modality affects various aspects of motion processing in another modality, such as its trajectory (e.g. Hall &

Neural correlates of multisensory integration of motion

The brain is rich in both subcortical and cortical areas that demonstrate a sensitivity to various properties of moving stimuli such as directionality or velocity. We now know a great deal about the neural networks involved in visual motion processing, but our knowledge of the networks of brain areas involved in auditory and somatosensory motion processing is currently somewhat more limited. Moreover, the study of how motion information from different modalities interacts in the human brain is

Conclusions

Behavioral research into the nature of crossmodal influences on the perception of stimulus motion has a long, albeit intermittent, history. However, it is only in the last few years, thanks in part to the development of a range of new psychophysical paradigms, that a consistent picture has started to emerge. Recent studies have demonstrated that spatial and temporal factors play an important role in motion integration, just as for many other crossmodal effects. There is now convincing

Acknowledgements

The authors would like to thank E. Macaluso, D. Lloyd, and A. Rodriguez-Fornells for helpful comments on an earlier draft of this manuscript, and also the University of Oxford McDonnell-Pew Centre for Cognitive Neuroscience for funding a Network Grant to S.S.-F. and C.S. Funding to AK was provided by Human Frontier Science Program, the Natural Sciences and Engineering Research Council of Canada, and the Michael Smith Foundation for Health Research. Correspondence regarding this article should

References (139)

  • J.E. Raymond

    Attentional modulation of visual motion perception

    Trends in Cognitive Sciences

    (2000)
  • G. Rees et al.

    What can functional imaging reveal about the role of attention in visual awareness?

    Neuropsychologia

    (2001)
  • S. Shimojo et al.

    Visual motion sensation yielded by non-visually driven attention

    Vision Research

    (1997)
  • M. Ahissar et al.

    Encoding of sound-source location and movement: Activity of single neurons and interactions between adjacent neurons in the monkey auditory cortex

    Journal of Neurophysiology

    (1992)
  • D. Alais et al.

    Neural strength of visual attention gauged by motion adaptation

    Nature Neuroscience

    (1999)
  • P.G. Allen et al.

    Sensory specificity of apparent motion

    Journal of Experimental Psychology: Human Perception and Performance

    (1981)
  • S.M. Anstis

    Hearing with the hands

    Perception

    (1973)
  • M.D. Arnoult

    Localization of sound during rotation of the visual environment

    American Journal of Psychology

    (1952)
  • H.B. Barlow et al.

    Mechanism of directionally selective units in rabbit’s retina

    Journal of Physiology

    (1965)
  • F. Baumgart et al.

    A movement-sensitive area in auditory cortex

    Nature

    (1999)
  • Bertelson, P. (1998). Starting from the ventriloquist: The perception of multimodal events (vol. 2). In M. Sabourin, C....
  • P. Bertelson et al.

    Automatic visual bias of perceived auditory location

    Psychonomic Bulletin & Review

    (1998)
  • Bertelson, P., & de Gelder, B. (in press). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.),...
  • P. Bertelson et al.

    The ventriloquist effect does not depend on the direction of deliberate visual attention

    Perception & Psychophysics

    (2000)
  • G.J. Blatt et al.

    Visual receptive field organization and cortico-cortical connections of the lateral intraparietal area (area LIP) in the macaque

    Journal of Computational Neurology

    (1990)
  • C. Bruce et al.

    Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque

    Journal of Neurophysiology

    (1981)
  • A. Caclin et al.

    Tactile ‘capture’ of audition

    Perception & Psychophysics

    (2002)
  • G.A. Calvert et al.

    Activation of auditory cortex during silent lipreading

    Science

    (1997)
  • L.K. Canon

    Intermodality inconsistency of input and directed attention as determinants of the nature of adaptation

    Journal of Experimental Psychology

    (1970)
  • L.K. Canon

    Directed attention and maladaptive ‘adaptation’ to displacement of the visual field

    Journal of Experimental Psychology

    (1971)
  • P. Cavanagh

    Attention-based motion perception

    Science

    (1992)
  • A. Chaudhuri

    Modulation of the motion aftereffect by selective attention

    Nature

    (1990)
  • C.S. Choe et al.

    The ‘ventriloquist effect’: Visual dominance or response bias?

    Perception & Psychophysics

    (1975)
  • Churchland, P. S., Ramachandran, V. S., Sejnowski, T. J. (1994). A critique of pure vision. In C. Koch & J. L. Davis...
  • H.R. Clemo et al.

    Responses to direction of stimulus movement are different for somatosensory and visual cells in cat superior colliculus

    Brain Research

    (1987)
  • M.S. Cohen et al.

    Changes in cortical activity during mental rotation. A mapping study using functional MRI

    Brain

    (1996)
  • C.L. Colby et al.

    Ventral intra-parietal area of the macaque: Anatomical location and visual response properties

    Journal of Neurophysiology

    (1993)
  • R.M. Costanzo et al.

    A quantitative analysis of responses of direction-sensitive neurons in somatosensory cortex of awake monkeys

    Journal of Neurophysiology

    (1980)
  • Dichgans, J., & Brandt, T. (1978). Visual-vestibular interaction (vol. VIII). In R. Held, H. W. Leibowitz, & H. L....
  • P.E. Downing et al.

    The line-motion illusion: Attention or impletion?

    Journal of Experimental Psychology: Human Perception and Performance

    (1997)
  • J. Driver

    Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

    Nature

    (1996)
  • Duhamel, J. R., Colby, C. L., & Goldberg, M. E. (1991). Congruent representations of visual and somatosensory space in...
  • J.R. Duhamel et al.

    Ventral intra-parietal area of the macaque: Congruent visual and somatic response properties

    Journal of Neurophysiology

    (1998)
  • W.H. Ehrenstein et al.

    A cross-modal aftereffect: Auditory displacement following adaptation to visual motion

    Perceptual and Motor Skills

    (1996)
  • Gemelli, A. (1951). The effect of illusory perception of movement on sound localization. In G. Ekman, G. Johansson, &...
  • G.M. Gilbert

    Dynamic psychophysics and the phi phenomenon

    Archives of Psychology

    (1939)
  • M.S.A. Graziano et al.

    Location of the polysensory zone in the precentral gyrus of anesthetized monkeys

    Experimental Brain Research

    (2000)
  • Graziano, M. S. A., Gross, C. G., Taylor, C. S. R., & Moore, T. (in press). A system of multimodal areas in the primate...
  • M.S.A. Graziano et al.

    Visuo-spatial properties of ventral premotor cortex

    Journal of Neurophysiology

    (1997)
  • M.S.A. Graziano et al.

    Coding of visual space by premotor neurons

    Science

    (1994)
  • Cited by (0)

    View full text