Demonstrating overall protocol agreement does not establish reliable category-level coding

Evidence

The reliability data reported suggest that coders can agree as to whether a total protocol is more likely to be the product of a prejudiced or an unprejudiced subject. But that reliability is not to the point since the discussion of the interviews chiefly concerns the particular content categories characteristic of the two kinds of subject. The study does not report the data that would tell us whether the individual categorical judgments can be made in a reliable fashion. — John T. Jost and Jim Sidanius, Political Psychology, 611