Consumer Reports loudspeaker ratings lacked rigorous subjective validation

Evidence

In 1985 I hosted a loudspeaker measurement seminar at the NRCC, where we showed and demonstrated our measurement capabilities and had lectures on things that were believed to be measurable and important to what we heard and preferred. The ultimate test was a double-blind subjective evaluation—ears and brains determined what was good, not the appearance of a curve. We also demonstrated those. The audience was a who’s who of loudspeaker designers and researchers from the US and Canada, and there were animated discussions. Among them were two engineers from Consumer Reports. Knowing they were coming, I had an overhead transparency prepared. For four loudspeakers that both of us had evaluated, it showed the subjective “fidelity” rating from our double-blind listening tests, and the “accuracy score” from their publication. The correlation was −0.7 (negative 0.7!). I still have the transparency. There was discussion, some puzzlement on their part, but nothing changed at the magazine. Based on this comparison, readers would have been well advised to invert the page of ratings. — , Toole, Floyd E - Sound reproduction-…, 583 (#exemplifies, via linked claim)

The development of such models is also a tedious and demanding task, and objective models can never fully replace subjective evaluation. The advantages of objective models lie in efficiency, rapid evaluation, and repeatability of results. The development of such models may also yield a deeper understanding of the phenomena than subjective results. The disadvantage of objective models is that they never take into account all factors in full detail, and thus their domain of validity (meaning that they correlate well with subjective results) is limited. — Ville Pulkki, Matti Karjalainen, Communication Acoustics- An…, 2337 (#corroborates, via linked claim)

Categorization

• XREF: connects to: Consumer Reports speaker ratings were negatively correlated with blind-test fidelity ratings; Consumer Reports loudspeaker ratings inversely predicted perceived sound quality. — , Toole, Floyd E - Sound reproduction-…, 583; Sighted loudspeaker evaluations are demonstrably unreliable (+4 more)

Related

• Consumer Reports speaker ratings were negatively correlated with blind-test fidelity ratings (similarity 0.95)
• Consumer Reports loudspeaker ratings inversely predicted perceived sound quality. (similarity 0.94)
• Consumer loudspeakers rated higher than professional monitors in blind tests (similarity 0.90)
• Sighted loudspeaker evaluations are demonstrably unreliable (similarity 0.88)
• Consumer Reports’ accuracy scores inversely correlated with actual listener preferences. (similarity 0.88)
• No universal sound-quality model exists; listener data is the only shared starting point (similarity 0.86)
• Objective sound quality models have limited domains of validity and cannot replace subjective testing (similarity 0.85)
• Objective measurements routinely expose flaws that subjective listening reviews miss (similarity 0.84)
• Sound power alone cannot predict listener preference for loudspeakers (similarity 0.84)
• Audible resonances drove low sound-quality ratings in early blind tests (similarity 0.84)
• Speaker cables and devices show near-zero audible differences in blind tests (similarity 0.84)
• Consumer Reports rated loudspeakers using sone-converted sound power against an ideal target (similarity 0.83)
• Blind listener tests reliably favor the most neutral loudspeakers (similarity 0.83)
• Consumer Reports killed its loudspeaker reviews after research disproved their methods (similarity 0.83)
• Sighted loudspeaker evaluations are contaminated by non-auditory visual and social cues (similarity 0.83)

Broader Connections

• Consumer Reports loudspeaker ratings inversely predicted perceived sound quality. #exemplifies (0.90) — , Toole, Floyd E - Sound reproduction-…, 583
• Sighted loudspeaker evaluations are demonstrably unreliable #corroborates (0.80)
• Objective sound quality models have limited domains of validity and cannot replace subjective testing #corroborates (0.80) — Ville Pulkki, Matti Karjalainen, Communication Acoustics- An…, 2337
• Microphone choice in recordings biases subjective loudspeaker ratings #expands (0.75) — Toole, Floyd E, Sound reproduction- the acoustics and…, 134
• Louder playback is unconsciously rated as better quality in audio listening tests #corroborates (0.75)