High inter-coder reliability can conceal systematically shared bias.

Evidence

For the interview protocols it was suggested that the subject who is tolerant of ambiguity will make much use of limiting and qualifying language forms. The subject intolerant of ambiguity would take a more absolute tone. Of course the coders were free to consult the total protocol and so may simply have coded intolerance of ambiguity where there seemed to be intolerance of ambivalence or any of the other stigmata of authoritarianism. Consequently the interview results are not good evidence that prejudiced subjects were intolerant of ambiguity. — John T. Jost and Jim Sidanius, Political Psychology, 571 (#complicates, via linked claim)

Categorization

• XREF: connects to: Aggregate reliability percentages can mask complete disagreement on individual coding decisions — John T. Jost and Jim Sidanius, Political Psychology, 610; Blind coders reliably scored integrative complexity with high interrater agreement — John T. Jost and Jim Sidanius, Political Psychology, 2314; Sensitivity correlations detect accuracy but cannot reveal direction of bias (+6 more)

Broader Connections

• Coder contamination undermined the interview evidence for intolerance of ambiguity #complicates (0.75) — John T. Jost and Jim Sidanius, Political Psychology, 571