Base-rate-trained models yield false causal conclusions without social-context experts

Evidence

In our previous discussion of the COMPAS system, we considered how training a model on data that reflects base rates can lead to incorrect conclusions about causality. An example of this is the statistical fact that most nurses are female. To correct for bias of this type, it is necessary to involve experts who are sensitive to the social context in which the AI will be operating (West, Whittaker, & Crawford, 2019). If the domain is law, and the machine is evaluating drug offenses, then lawyers who understand historical racial bias and the fact that minorities—and especially African Americans— have been over-policed and over-prosecuted for marijuana offenses, must be involved in critiquing the models’ outputs. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 1181

Categorization

• XREF: Connects to the reader’s core interest in whether rational correction of threat-hardened statistical reasoning backfires — here the correction is expert contextualization rather than data alone.
• DEFINE: Illustrates how base rates mislead causal inference, a mechanism central to measuring whether predictive tools encode rather than reflect bias.
• corroborated: 2 independent passages support this claim — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law: A Definitive Look at…, 1181