Rights statement: This is the author’s version of a work that was accepted for publication Journal of Memory and Language. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Journal of Memory and Language, 93, 2016 DOI: 10.1016/j.jml.2016.08.005
Accepted author manuscript, 8.39 MB, PDF document
Available under license: CC BY-NC-ND: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
Final published version
Research output: Contribution to Journal/Magazine › Journal article › peer-review
Research output: Contribution to Journal/Magazine › Journal article › peer-review
}
TY - JOUR
T1 - The multimodal nature of spoken word processing in the visual world
T2 - testing the predictions of alternative models of multimodal integration
AU - Smith, Alastair C.
AU - Monaghan, Padraic
AU - Huettig, Falk
N1 - This is the author’s version of a work that was accepted for publication Journal of Memory and Language. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Journal of Memory and Language, 93, 2016 DOI: 10.1016/j.jml.2016.08.005
PY - 2017/4
Y1 - 2017/4
N2 - Ambiguity in natural language is ubiquitous, yet spoken communication is effective due to integration of information carried in the speech signal with information available in the surrounding multimodal landscape. Language mediated visual attention requires visual and linguistic information integration and has thus been used to examine properties of the architecture supporting multimodal processing during spoken language comprehension. In this paper we test predictions generated by alternative models of this multimodal system. A model (TRACE) in which multimodal information is combined at the point of the lexical representations of words generated predictions of a stronger effect of phonological rhyme relative to semantic and visual information on gaze behaviour, whereas a model in which sub-lexical information can interact across modalities (MIM) predicted a greater influence of visual and semantic information, compared to phonological rhyme. Two visual world experiments designed to test these predictions offer support for sub-lexical multimodal interaction during online language processing.
AB - Ambiguity in natural language is ubiquitous, yet spoken communication is effective due to integration of information carried in the speech signal with information available in the surrounding multimodal landscape. Language mediated visual attention requires visual and linguistic information integration and has thus been used to examine properties of the architecture supporting multimodal processing during spoken language comprehension. In this paper we test predictions generated by alternative models of this multimodal system. A model (TRACE) in which multimodal information is combined at the point of the lexical representations of words generated predictions of a stronger effect of phonological rhyme relative to semantic and visual information on gaze behaviour, whereas a model in which sub-lexical information can interact across modalities (MIM) predicted a greater influence of visual and semantic information, compared to phonological rhyme. Two visual world experiments designed to test these predictions offer support for sub-lexical multimodal interaction during online language processing.
KW - Visual world paradigm
KW - Visual attention
KW - Spoken word recognition
KW - Connectionist modelling
KW - Multimodal processing
U2 - 10.1016/j.jml.2016.08.005
DO - 10.1016/j.jml.2016.08.005
M3 - Journal article
VL - 93
SP - 276
EP - 303
JO - Journal of Memory and Language
JF - Journal of Memory and Language
SN - 0749-596X
ER -