Voluntary attention modulates how audiovisual contexts affect subsequent fusion

Evidence

Experiment B shows that attentional mechanisms may intervene at the level of single audiovisual sources in an audiovisual speech scene, selectively increasing or decreasing the amount of fusion depending on the coherence of the attended source. Interestingly, attention intervened only for “Video sentences”. Our interpretation is that binding could be pre-attentive for syllables, because of their salient audiovisual comodulations making them pop out as strong bottom-up audiovisual primitives. In contrast, since the coherence of AV sentences is low, the attentional focus could enhance audiovisual top-down schemas increasing binding. — Dijk, Pim van, Physiology, Psychoacoustics and…, 5137

The present experiments extend the AVSSA paradigm by creating contexts made of competing audiovisual sources and measuring their effect on McGurk targets. The competing audiovisual sources have respectively a high and a low audiovisual coherence (that is, large vs. small audiovisual comodulations in time). The first experiment involves contexts made of two auditory sources and one video source associated to either the first or the second audio source. It appears that the McGurk effect is smaller after the context made of the visual source associated to the auditory source with less audiovisual coherence. In the second experiment with the same stimuli, the participants are asked to attend to either one or the other source. The data show that the modulation of fusion depends on the attentional focus. — Dijk, Pim van, Physiology, Psychoacoustics and…, 5134

In the second experiment with the same stimuli, the participants are asked to attend to either one or the other source. The data show that the modulation of fusion depends on the attentional focus. — Dijk, Pim van, Physiology, Psychoacoustics and…, 5135

Categorization

• XREF: The claim that salient bottom-up comodulations bind pre-attentively while weak coherence needs top-down attention parallels scene-analysis dual-process accounts in vision and hearing. — connects to: Exposure to low-coherence audiovisual contexts reduces subsequent multisensory fusion; Audiovisual fusion is context-modulated, not automatic, and scales with audio-video coherence — Dijk, Pim van, Physiology, Psychoacoustics and…, 5177; Attention modulates audiovisual binding only for low-coherence stimuli, not salient ones (+4 more)
• SEED: Potential angle: the coherence-before-binding architecture as a general account of how attention selects among competing multisensory streams — relevant to cocktail-party problems in both vision and hearing.
• QUESTION: Does the coherence index operate via comodulation in time as a low-level cue, or through higher-order semantic matching? Worth exploring what ‘audiovisual comodulations’ are measured as.
• corroborated: 5 independent passages support this claim — Dijk, Pim van, Physiology, Psychoacoustics and…, 5137; Dijk, Pim van, Physiology, Psychoacoustics and…, 5134; Dijk, Pim van, Physiology, Psychoacoustics and…, 5135

Broader Connections

• Audiovisual fusion is context-modulated, not automatic, and scales with audio-video coherence #expands (0.85) — Dijk, Pim van, Physiology, Psychoacoustics and…, 5177
• Selective attention modulates fusion only for low-coherence video sentence contexts. #expands (0.85)
• Audiovisual speech fusion is not an automatic process but context-modulated. #corroborates (0.82) — Dijk, Pim van, Physiology, Psychoacoustics and…, 5137
• Attention modulates audiovisual binding only for low-coherence stimuli, not salient ones #qualifies (0.80)
• Top-down instructional attention modulates audiovisual speech perception in unimpaired listeners #corroborates (0.78)