Multi-stimulus quality tests fail when samples vary in several perceptual dimensions at once

Evidence

The method has also been criticized. For example, if the samples differ from each other in multiple dimensions, the listeners may be confused as to how to rate them. Let us imagine that the sample set consists of audio content with a reference case of 5.1 audio, stereophonic and monophonic down-mixes, and low-pass-filtered versions of the 5.1 audio content. The listeners then have to judge the degree of degradation of sound quality in two dimensions, since both spatial and timbral aspects vary. This may result in data that are difficult to interpret. — Ville Pulkki, Matti Karjalainen, Communication Acoustics- An…, 2357

Categorization

• XREF: Parallels a known limitation in multidimensional scaling and survey design where confounded dimensions produce uninterpretable data.