Sensitivity correlations detect accuracy but cannot reveal direction of bias

Evidence

While these sorts of sensitivity correlations have the potential to overcome judgmental scale usage problems, they of course make directional comparisons between the criteria and judgments impossible. … For instance, if there are inaccuracies, one cannot determine whether judgments overestimate or underestimate the stereotypic differences between groups. — , Rudman, Laurie A - On the Nature of…, 872 (#expands, via linked claim)

A second alternative to direct difference comparisons between judgments and the criterion is to use correlational methods. We have called such correlational assessments sensitivity correlations. Across multiple attribute dimensions, one can assess for each perceiver the degree to which judgments covary with the criterion. In terms of stereotypic accuracy, substantial correlations would indicate that perceivers are sensitive to the varying prevalence of particular attributes within a group: perceivers judge there to be a greater prevalence of attribute dimensions that are in fact more prevalent. While these sorts of sensitivity correlations have the potential to overcome judgmental scale usage problems, they of course make directional comparisons between the criteria and judgments impossible. For instance, if there are inaccuracies, one cannot determine whether judgments overestimate or underestimate the stereotypic differences between groups. — , Rudman, Laurie A - On the Nature of…, 872 (#expands, via linked claim)

All of our data up to this point have used correlations calculated on aggregates as evidence of greater or lesser constraint among elements in belief systems. While we believe these correlations to be informative indicators, they do depend for their form upon cumulations among individuals and therefore can never be seen as commenting incisively upon the belief structures of individuals. — John T. Jost and Jim Sidanius, Political Psychology, 1933 (#complicates, via linked claim)

Categorization

• XREF: connects to: Polarization similarity hides distinct causes requiring opposite interventions — Cailin O’Connor and James Owen Weatherall, The Misinformation Age: How False…, 311; High inter-coder reliability can conceal systematically shared bias.; No proper critical apparatus for diagnosing bias has yet been developed (+3 more)

Broader Connections

• Correlational accuracy measures cannot reveal over- or underestimation direction. #expands (0.95) — , Rudman, Laurie A - On the Nature of…, 872
• Correlational accuracy measures sacrifice directional information about stereotype errors. #expands (0.92) — , Rudman, Laurie A - On the Nature of…, 872
• Aggregate correlations cannot reveal the belief structures of individuals #complicates (0.72) — John T. Jost and Jim Sidanius, Political Psychology, 1933