Naive raters in sufficient numbers yield more reliable subjective quality judgments than experts.

Evidence

When speech signals are rated subjectively by listeners, perhaps for the evaluation of the signal provided by a mobile phone or the output from a speech-synthesis system, it is usually the quality of the signal that is of interest. The number of listeners is important, since each will have a personal opinion, and it is generally suggested that at least 16 are used to ensure that statistical analysis of the results is sufficiently powerful. However, the greater the number of listeners, the more reliable the results are. It is also most appropriate to use listeners who are potential users of whatever system might result from the work and listeners who are definitely not experts in the area. — David M. Howard, Jamie Angus, Acoustics and Psychoacoustics, 2386

Categorization

• SHARE: The ‘at least 16 listeners, and definitely not experts’ rule is exactly the sort of methodological gossip a speech-coding or UX-research friend would relish.