Aggregate reliability percentages can mask complete disagreement on individual coding decisions

Evidence

For the main interview study nine protocols were coded by two raters. There were some ninety cat-egories to be coded and these were put together as pairs such that one member was identified as a High Prejudice category and the other as a Low Prejudice category. We should like to know how well the authors agreed in their decisions for each pair even though there could only be nine items per pair. We are not given this information but instead The Authoritarian Personality reports for both raters the percentage of High categories scored in each total interview. For the most part these are closely similar but closely similar overall percentages do not guarantee closely similar decision patterns on particular categories. Both raters, for example, could have scored half of the pairs as High and half as Low but they might have exactly reversed one another in terms of the particular categories scored each way. — John T. Jost and Jim Sidanius, Political Psychology, 610

Categorization

• QUESTION: This methodological critique of The Authoritarian Personality’s reliability reporting raises the question of how common this aggregate-vs-item reliability gap is across famous studies.