Correlated omitted variables like income make racial bias estimates unreliable.

Evidence

The error term εdjc captures all other factors that impact the outcome Y. Any determinant of Y that is left out and not included in Equation (1) is embedded in the error term ε. The key assumption in the analysis of judicial decisions with a regression model, as in Equation (1), is that any left-out, unobserved factor, captured by the error term, is not correlated with the variable Minority Defendant. For example, if defendant income is not included in regression Equation (1) (if income is not part of the vector X), this implies that income is embedded in the error term (ε). To the extent that minority status and income are correlated, the error term ε (which now captures income) and the variable Minority Defendant will be correlated. Thus, any variation in incomes between defendants will translate into variation in the error terms and therefore in the case outcomes between the defendants, and such income variation will also impact the key explanatory variable Minority Defendant. In this situation it is impossible to obtain an unbiased estimate of the parameter β1 to determine the true impact of the defendant’s minority status on the outcome of interest. That is, it would not be possible to disentangle the true “causal” impact of minority status from the impact of income on the outcome. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 736 (#expands, via linked claim)

This is a troublesome result, and it is understandable that objections can be raised. For instance, some might argue that it is impossible for researchers to control for every single factor that can matter in a judge’s decision; there could be additional features of the case that can be observed by the judge but unobserved by researchers, and these omitted variables could be driving the findings. This objection, while reasonable, does not have the power it used to have thirty years ago when empirical analyses were much less sophisticated and big and detailed data sets were not available. This “omitted variable bias” argument has less credibility now because researchers apply increasingly sophisticated statistical techniques to large data sets which include rich information on the case: most, if not all, of the information that is available to the judge. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 778 (#complicates, via linked claim)

Categorization

• XREF: connects to: Measurement error in the affluent proxy biases elite influence estimates downward, not upward. — Martin Gilens and Benjamin I. Page, Testing Theories of American Politics:…, 58; Omitted variables correlated with race make regression estimates of judicial bias unreliable; Omitted variables correlated with race bias the discrimination estimate — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law: A Definitive Look at…, 736 (+2 more)

Broader Connections

• Omitted variables correlated with race make regression estimates of judicial bias unreliable #expands (0.95)
• Omitted variables correlated with race bias the discrimination estimate #expands (0.92) — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 736
• Omitted-variable critiques of bias studies have weakened with richer data #complicates (0.75) — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 778