Home > Research > Publications & Outputs > Eye Gaze and Perceptual Adaptation to Audiovisu...

Electronic data

  • Banks_et_al_2021_JSLHR_accepted_manuscript

    Accepted author manuscript, 611 KB, PDF document

    Embargo ends: 1/03/22

    Available under license: CC BY-NC: Creative Commons Attribution-NonCommercial 4.0 International License


Text available via DOI:

View graph of relations

Eye Gaze and Perceptual Adaptation to Audiovisual Degraded Speech

Research output: Contribution to journalJournal articlepeer-review

<mark>Journal publication date</mark>14/09/2021
<mark>Journal</mark>Journal of Speech, Language, and Hearing Research
Issue number9
Number of pages14
Pages (from-to)3432-3445
Publication StatusPublished
Early online date31/08/21
<mark>Original language</mark>English


Visual cues from a speaker's face may benefit perceptual adaptation to degraded speech, but current evidence is limited. We aimed to replicate results from previous studies to establish the extent to which visual speech cues can lead to greater adaptation over time, extending existing results to a real-time adaptation paradigm (i.e., without a separate training period). A second aim was to investigate whether eye gaze patterns toward the speaker's mouth were related to better perception, hypothesizing that listeners who looked more at the speaker's mouth would show greater adaptation.

A group of listeners (n = 30) was presented with 90 noise-vocoded sentences in audiovisual format, whereas a control group (n = 29) was presented with the audio signal only. Recognition accuracy was measured throughout and eye tracking was used to measure fixations toward the speaker's eyes and mouth in the audiovisual group.

Previous studies were partially replicated: The audiovisual group had better recognition throughout and adapted slightly more rapidly, but both groups showed an equal amount of improvement overall. Longer fixations on the speaker's mouth in the audiovisual group were related to better overall accuracy. An exploratory analysis further demonstrated that the duration of fixations to the speaker's mouth decreased over time.

The results suggest that visual cues may not benefit adaptation to degraded speech as much as previously thought. Longer fixations on a speaker's mouth may play a role in successfully decoding visual speech cues; however, this will need to be confirmed in future research to fully understand how patterns of eye gaze are related to audiovisual speech recognition. All materials, data, and code are available at https://osf.io/2wqkf/.