Abstract
The best format in which information that has to be recalled is presented has been investigated in several studies, which focused on the impact of bimodal stimulation on working memory performance. An enhancement of participant’s performance in terms of correct recall has been repeatedly found, when bimodal formats of presentation (i.e., audiovisual) were compared to unimodal formats (i.e, either visual or auditory), in providing implications for multimedial learning. Several theoretical frameworks have been suggested in order to account for the bimodal advantage, ranging from those emphasizing early stages of processing (such as automatic alerting effects or multisensory integration processes) to those centred on late stages of processing (as postulated by the dual coding theory). The aim of this paper is to review previous contributions to this topic, providing a comprehensive theoretical framework, which is updated by the latest empirical studies.
Similar content being viewed by others
References
Baddeley A (1990) Human memory: theory and practice. Lawrence Erlbaum Associates Inc, Hillsdale
Baddeley A (1992) Working memory. Science 255:556–559
Baddeley AD (2000) The episodic buffer: a new component of working memory? Trends Cogn Sci 4:417–423
Baddeley AD, Hitch GJ (1974) Working memory. In: Bower G (ed) The psychology of learning and motivation. Academic, New York, pp 47–89
Bauer MI, Johnson-Laird PN (1993) How diagrams can improve reasoning. Psychol Sci 6:372–378
Buckingham HW (1981) Lexical and semantic aspects of aphasia. In: Sarno MT (ed) Acquired aphasia. Academic, New York, pp 183–214
Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657
Campbell R, Dodd B (1980) Hearing by eye. Q J Exp Psychol 32:85–99
Crowder RG, Morton J (1969) Precategorical acoustic storage (PAS). Percept Psychophys 5:363–373
De Gelder B, Vroomen J (1992) Abstract versus modality-specific memory representations. Mem Cognit 20:533–538
De Gelder B, Vroomen J (1994) Memory for consonants versus vowels in heard and lipread speech. J Mem Lang 31:737–756
De Gelder B, Vroomen J (1997) Modality effects in immediate recall of verbal and non-verbal information. Eur J Cogn Psychol 9:97–110
Engelkamp J, Zimmer HD (1994) The human memory: a multi-modal approach. Hogrefe, Toronto
Foos PW, Goolkasian P (2005) Presentation format effects in working memory: the role of attention. Mem Cognit 33:499–513
Fort A, Delpuech C, Pernier J, Giard MH (2002) Early auditory–visual interactions in human cortex during nonredundant target identification. Cogn Brain Res 14:20–30
Giard MH, Peronnet F (1999) Auditory–visual integration during multimodal object recognition in humans: a behavioural and electrophysiological study. J Cogn Neurosci 11:473–490
Goodglass H (1980) Disorders of naming following brain injury. Am Sci 68:647–655
Goolkasian H (1996) Picture-word difference in a sentence verification task. Mem Cognit 24:584–594
Goolkasian P (2000) Pictures, words, and sounds: from which format are we best able to reason? J Gen Psychol 127:439–459
Goolkasian P, Foos PW (2002) Presentation format and its effect on working memory. Mem Cognit 30:1096–1105
Goolkasian P, Foos PW (2005) Bimodal format effects in working memory. Am J Psychol 118:61–77
Greene RL, Samuel AG (1986) Recency and suffix effects in serial recall of musical stimuli. J Exp Psychol Learn Mem Cogn 12:517–524
Hasher L, Zacks RT (1979) Automatic and effortful processes in memory. J Exp Psychol Gen 108:356–388
Johnson CJ, Paivio A, Clark JM (1996) Cognitive components of picture naming. Psychol Bull 120:113–139
Jones DM (1994) Disruption of memory for lip read lists by irrelevant speech: further support for the changing state hypothesis. Q J Exp Psychol 47A:143–160
Jones DM, Macken WJ (1993) Irrelevant tones produce an irrelevant speech effect: implications for phonological coding in working memory. J Exp Psychol Learn Mem Cogn 19:369–381
Jones DM, Macken WJ (1995) Phonological similarity in the irrelevant speech effect: within- or between-stream similarity? J Exp Psychol Learn Mem Cogn 21:103–133
Larkin JH, Simon HA (1987) Why an illustration is (sometimes) worth ten thousand words. Cogn Sci 11:65–99
Mayer RE, Sims VK (1994) Why a diagram is (sometimes) worth ten thousand words. Cogn Sci 11:65–69
Morton J (1969) Interaction of information in word recognition. Psychol Rev 76:165–178
Morton J (1979) Facilitation in word recognition: experiments causing change in the logogen model. In: Kolers PA, Wrolstead M, Bouma H (eds) Processing of visible language, vol 1. Plenum Press, New York, pp 259–268
Oberauer K (2002) Access to information in working memory: exploring the focus of attention. J Exp Psychol Learn Mem Cogn 28:411–421
Oberauer K, Demmruch A, Mayr U, Kilegl R (2001) Dissociating retention and access in working memory: an age-comparative study of mental arithmetic. Mem Cognit 29:18–33
Olivetti Belardinelli M, Sestieri C, Di Matteo R, Delogu F, Del Gratta C, Ferretti A, Caulo M, Tartaro M, Romani GL (2004) Audio–visual crossmodal interactions in environmental perception: An fMRI investigation. Cogn Process 5:167–174
Paivio A (1971) Imagery and verbal processes. Holt, Rinehart and Winston, New York
Paivio A (1986) Mental representations: a dual coding approach. Oxford University Press, Oxford
Penney CG (1975) Modality effects in short-term verbal memory. Psychol Bull 82:68–84
Penney CG (1989) Modality effects and the structure of short-term verbal memory. Mem Cognit 17:398–422
Santangelo V, Spence C (2007) Multisensory cues capture spatial attention regardless of perceptual load. J Exp Psychol Hum Percept Perform (in press)
Santangelo V, Mastroberardino S, Botta F, Marucci FS, Olivetti Belardinelli M (2006) On the influence of audio–visual interactions on working memory performance: a study with non-semantic stimuli. Cogn Process 7:187
Santangelo V, Olivetti Belardinelli M, Spence C (2007) The suppression of reflexive visual and auditory orienting when attention is otherwise engaged. J Exp Psychol Hum Percept Perform 33:137–148
Spoehr KT, Corin WJ (1978) The stimulus suffix effect as a memory coding phenomenon. Mem Cognit 6:583–589
Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge
Surprenant AM, LeCompte DC, Neath I (2000) Manipulations of irrelevant information: suffix effects with articulatory suppression and irrelevant speech. Q J Exp Psychol A 53:325–348
Sweller J, Chandler P, Tierney P, Cooper M (1990) Cognitive load as a factor in the structuring of technical material. J Exp Psychol Gen 119:176–192
Teder-Sälejärvi WA, McDonald JJ, Di Russo F, Hillyard SA (2002) An analysis of audio–visual crossmodal integration by means of event-related potential (ERP) recordings. Cogn Brain Res 14:106–114
Thompson V, Paivio A (1994) Memory for pictures and sounds: independence of auditory and visual codes. Can J Exp Psychol 48:380–396
Tindall-Ford S, Chandler P, Sweller J (1997) When two sensory modes are better than one. J Exp Psychol Appl 3:257–287
Acknowledgment
We wish to thank Angela Tagini for the language revision.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mastroberardino, S., Santangelo, V., Botta, F. et al. How the bimodal format of presentation affects working memory: an overview. Cogn Process 9, 69–76 (2008). https://doi.org/10.1007/s10339-007-0195-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10339-007-0195-6