Averaged listener ratings can hide hearing-loss-driven individual disagreement

Evidence

In Figure 10.8, it is seen that fidelity ratings for loudspeakers D and U are closely grouped by both sets of listeners, but the ratings in the shaded areas are simply lower. Things change for loudspeakers V and X, where the close groupings of the low-variability listeners change to widely dispersed ratings by the high-variability listeners. This is a case where an averaged rating does not reveal what is happening. Listener ratings simply dispersed to cover the available range of values; some listeners thought it was not very good (fidelity ratings below 6), whereas others thought it was among the best loudspeakers they had ever heard (fidelity ratings above 8). Listeners simply exhibited strongly individualistic opinions. Hearing loss is very likely involved. Be careful whose opinions you trust. — Toole, Floyd E, Sound reproduction- the acoustics and…, 1075

Categorization

• DEEPTHINKING: This is a genuinely paradigm-shifting point for anyone doing subjective quality evaluation: the standard practice of averaging ratings actively destroys signal when hearing variation is present — an averaging paradox worth sitting with.
• QUESTION: How does one detect and correct for high-variability listeners before averaging, and does this generalize beyond loudspeakers to any subjective rating data?