Temporal coherence of audiovisual signals predicts how strongly vision captures auditory perception.

Evidence

However, these studies were based on audiovisual scenes that never implied competing sources. The objective of the present study was to test the “binding and fusion” AVSSA process in scenes including competition between audiovisual sources. For this aim, we generated two audiovisual sources, one made of a sequence of isolated syllables, and the other one made of a sequence of sentences. We prepared two kinds of combinations, with the same auditory content (mixing the two audio sources, syllables and sentences) and two different video contents, either the syllables or the sentences. These two combinations (“Video syllables” and “Video sentences”) were used as the context in a McGurk experiment. We hypothesized that since syllables correspond to stronger audiovisual modulations in time and hence stronger audiovisual coherence than sentences, the association between the visual input and the corresponding auditory input would be stronger for syllables than for sentences. Hence the coherence of the audiovisual context would be stronger for syllables, and it would lead to a larger visual weight and more McGurk effect than with visual sentences. Furthermore, the introduction of a competition between sources made it possible to introduce attention factors in the paradigm, and we tested whether the attentional focus put by the participants on either syllables or sentences would play a role in the AVSSA process. — Dijk, Pim van, Physiology, Psychoacoustics and…, 5144

Categorization

• DEFINE: Introduces AVSSA “binding and fusion” as a framework for how audiovisual sources get associated.