Machine learning models encode the historical base rates present in their training data

Evidence

Second, there is bias that emerges from base rates, which machines may interpret as reflective of causal relationships. For instance, doctors may statistically be more likely to be male, but a machine that always concludes that doctors are male (or that a female could not be a doctor) would be making a significant error. Something along these lines occurred when Amazon attempted to automate its resume-screening procedures (Dastin, 2018). Because many of Amazon’s early, successful employees had been male (i.e., the base rate was heavily male), the machine screened out nearly all female candidates. In short, the machine’s bias was the result of it picking up on the base rate that humans had created. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 1154

Categorization

• XREF: Directly relevant to the reader’s bias-in-law interest: algorithmic bias is a form of ‘measured bias’ encoded in data rather than professed intent, contrasting with the sincere-conviction-vs-measured-bias gap focus.
• corroborated: 2 independent passages support this claim — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law: A Definitive Look at…, 1154