Base-rate training data can mislead models about causality

Evidence

In our previous discussion of the COMPAS system, we considered how training a model on data that reflects base rates can lead to incorrect conclusions about causality. An example of this is the statistical fact that most nurses are female. To correct for bias of this type, it is necessary to involve experts who are sensitive to the social context in which the AI will be operating (West, Whittaker, & Crawford, 2019). If the domain is law, and the machine is evaluating drug offenses, then lawyers who understand historical racial bias and the fact that minorities—and especially African Americans— have been over-policed and over-prosecuted for marijuana offenses, must be involved in critiquing the models’ outputs. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 1181

Categorization

Category: Causal Relationships

• XREF: Connects to the earlier kept claim about COMPAS assigning higher risk scores to Black defendants who did not reoffend, from the same book.

Related

• Base-rate-trained models yield false causal conclusions without social-context experts (similarity 0.94)
• Machine learning models encode the historical base rates present in their training data (similarity 0.81)
• COMPAS errors fall unequally on Black and White defendants (similarity 0.79)