Consumer Reports loudspeaker ratings inversely predicted perceived sound quality.

Evidence

In 1985 I hosted a loudspeaker measurement seminar at the NRCC, where we showed and demonstrated our measurement capabilities and had lectures on things that were believed to be measurable and important to what we heard and preferred. The ultimate test was a double-blind subjective evaluation—ears and brains determined what was good, not the appearance of a curve. We also demonstrated those. The audience was a who’s who of loudspeaker designers and researchers from the US and Canada, and there were animated discussions. Among them were two engineers from Consumer Reports. Knowing they were coming, I had an overhead transparency prepared. For four loudspeakers that both of us had evaluated, it showed the subjective “fidelity” rating from our double-blind listening tests, and the “accuracy score” from their publication. The correlation was −0.7 (negative 0.7!). I still have the transparency. There was discussion, some puzzlement on their part, but nothing changed at the magazine. Based on this comparison, readers would have been well advised to invert the page of ratings. — , Toole, Floyd E - Sound reproduction-…, 583

Categorization

• CANDY: A measurement methodology that scored -0.7 against reality, advice to ‘invert the page’ — a delicious party fact about how wrong an authoritative consumer publication can be.
• ADDENDUM: Complements the sacrifice/grafting pieces’ theme that institutional judgments can be disconnected from ground truth — here a prestigious review methodology actively misled consumers.
• XREF: connects to: Brand, cost, and prior exposure bias subjective audio judgments; Consumer Reports speaker ratings were negatively correlated with blind-test fidelity ratings; Consumer Reports loudspeaker ratings lacked rigorous subjective validation (+4 more)

Related

• Consumer Reports speaker ratings were negatively correlated with blind-test fidelity ratings (similarity 0.95)
• Consumer Reports loudspeaker ratings lacked rigorous subjective validation (similarity 0.94)
• Consumer Reports’ accuracy scores inversely correlated with actual listener preferences. (similarity 0.91)
• Consumer loudspeakers rated higher than professional monitors in blind tests (similarity 0.89)
• Consumer Reports rated loudspeakers using sone-converted sound power against an ideal target (similarity 0.84)
• Consumer Reports killed its loudspeaker reviews after research disproved their methods (similarity 0.83)
• Sound power alone cannot predict listener preference for loudspeakers (similarity 0.83)
• Sighted loudspeaker evaluations are demonstrably unreliable (similarity 0.82)
• Louder playback is unconsciously rated as better quality in audio listening tests (similarity 0.81)
• Objective measurements routinely expose flaws that subjective listening reviews miss (similarity 0.80)
• Historical loudspeaker measurements were non-standardized and largely kept from the public (similarity 0.80)
• Historically, published loudspeaker specifications were often artistic rather than factual. (similarity 0.79)
• Audible resonances drove low sound-quality ratings in early blind tests (similarity 0.79)
• Consumer loudspeakers at moderate prices rival professional monitors in sound quality. (similarity 0.79)
• Uncontrolled loudness differences confound audio quality listening-test results (similarity 0.79)

Broader Connections

• Louder playback is unconsciously rated as better quality in audio listening tests #contradicts (0.90)