Uncontrolled loudness differences confound audio quality listening-test results

Evidence

The non-linear processing in the ear causes sound with different sound pressure levels to be perceived differently. The loudness differences between the samples may cause undesired effects if the loudness itself is not studied. Thus, when conducting a listening test, the effect of the level of presentation of the signals should be taken into account. For example, if the audio quality provided by different loudspeakers is tested, and if one loudspeaker delivers slightly but noticeably higher SPL to the listener, it quite probably will be rated to provide the best quality of the tested items. — Ville Pulkki, Matti Karjalainen, Communication Acoustics- An…, 866

Categorization

• ADDENDUM: Directly relevant to the soundscape and listening articles: a concrete experimental pitfall where perceived quality is confounded by loudness, a psychological phenomenon rather than true quality.
• QUESTION: What calibration methods and loudness metrics are actually used to equalize levels across test items?