No direction-specific bimodal facilitation for audiovisual motion detection

Abstract

After several decades of unimodal perceptual research, interest is turning increasingly to cross-modal interactions. At a physiological level, the existence of bimodal cells is well documented and it is known that correlated audiovisual input enhances localisation and orienting behaviours. Audiovisual perceptual interactions have also been demonstrated (e.g., the well-known McGurk effect). The present study explores motion perception and asks whether correlated audiovisual motion signals would be better detected than unimodal motions or bimodal motions in opposing directions. Using a dynamic random-dot field with variable motion coherence as a visual stimulus, together with an auditory motion defined by a stereo noise source smoothly translating along a horizontal trajectory, we find that correlated bimodal motion yields only a slight improvement (approximately a square root of two advantage) in detection threshold relative to unimodal detection. The size of this benefit is consistent with a statistical advantage rather than a bimodal facilitation account. Moreover, anticorrelated bimodal motion showed the same modest improvement, again speaking against linear summation but consistent with statistical combination of visual and auditory signals. These findings were replicated in peripheral as well as in central vision, and with translating visual objects as well as with spatially distributed visual motion. The superadditivity observed neurally (especially in deep-layer superior collicular cells), when weak unimodal signals are combined in bimodal cells does not apply to the detection of linear translational motion.

Introduction

Many objects in the external environment are represented in two or more sensory modalities. Touch and vision are commonly co-activated, when objects are taken in hand and inspected. Audition and vision are also frequently activated by the same stimulus event (the sight and sound of speeding cars is a common example). Despite the modularity of our sensory systems, we perceive a unified and coherent world. Indeed, by synthesising complementary information, we enhance the likelihood that our internal perceptions will accurately reflect external realities [7], enabling us to respond more rapidly and appropriately. Two of the interesting questions which arise from this synthesis are: how is information combined across modalities; and does the perceptual system capitalise on complementary information about the same stimulus to improve its performance. The experiments we present deal primarily with the latter question. Specifically, we ask whether the ability to detect movement is improved when that movement is represented in both auditory and visual modalities.

The combination of information across senses has been heavily researched at the neurophysiological level [30], with particular focus on the superior colliculus. Its deep layers contain many ‘multisensory’ neurons–neurons that receive unimodal sensory input from more than one source. Multisensory cells may be bimodal, or even trimodal, with audiovisual bimodal cells a common variety. These are arranged in a topographical representation of external space and have separate but overlapping auditory and visual receptive fields so that they respond to audiovisual input from a single location [35]. Although they can be driven unimodally, they exhibit a strong non-linear response known as “superadditivity” [12], [15] when driven bimodally by spatiotemporally correlated audiovisual input.

Behavioural and attentional studies have demonstrated that cross-modal interactions do indeed occur. Behaviourally, it known that bimodal superadditivity improves orienting behaviours such as eye movements [8], [30], [32], [37] and aids stimulus localisation [1], [31]. This likely reflects response enhancement to correlated bimodal stimuli creating a salient peak on collicular topography. Several studies have also confirmed the physiological observation that response enhancement (superadditivity) is maximal when auditory and visual inputs arrive synchronously [12], [16], although for perceptual tasks the temporal window within which auditory and visual stimuli can be phenomenally integrated is rather broad [13], [21]. In contrast, discordant stimuli lead to “response depression” [11], [15] and a corresponding decrease in efficiency of orienting behaviours [32], [37]. It has also been shown that attending to a particular spatial location for visual stimuli improves task performance for auditory stimuli (and vice versa) in that location [27], [28], suggesting a linked audiovisual topography. This is consistent with the fact that the superior colliculus (containing multi-modal cells) is strongly implicated in orienting to salient stimuli, whether overtly with eye movements or covertly with attention [6], [26].

Apart from its role in orienting, the superior colliculus also has strong reciprocal links, via the pulvinar, with middle-temporal (MT) cortical area [29]. MT is an area specialised for processing visual movement and activity in this area is strongly correlated with visual motion perception [2], [3]. Outputs from MT project directly to area VIP where they combine with input from auditory areas to create bimodal cells with strong motion selectivity [5], [9], [14]. Based on the evidence for strong audiovisual interactions in sensory processing, both at an early, subcortical level as well as at higher, motion-specialised cortical areas, we conducted experiments to examine whether sensitivity to bimodally represented movement might be improved relative to unimodal baselines when that movement is spatiotemporally concordant in both audition and vision. In particular, we focused on thresholds for motion detection, as the neurophysiological evidence suggests that response enhancement should be stronger when the unimodal stimuli are weak [15]. On this principle, unimodal stimuli too weak to be detected alone could conceivably become detectable when part of a correlated bimodal stimulus. We therefore measured motion detection thresholds unimodally for vision and for audition, and then again when the stimuli were presented together as a bimodal motion stimulus. We compared conditions in which the auditory and visual components were either matched in direction (correlated) or were opposed (anticorrelated). This was repeated for visual motion in central and in peripheral vision, and for visual stimuli that were spatially distributed or were a spatially localised object. The results show no evidence of a facilitative audiovisual interaction for detection of linear translations, whether in the central or peripheral field.