Missing counterfactuals make outcome-based evaluation of street-level programs impossible

Evidence

Still another reason that street-level bureaucrats’ performance often eludes effective evaluation is that there are too many variables to take into account to make evaluation realistic. It is not only that human beings are complex and that a metric of correct responses is inappropriate. Equally important, there is rarely any way to determine on a regular basis what would have happened to clients in the absence of intervention. A job training program may have a 40 percent placement rate, but to assess its effectiveness one would want to know the employment potential in the trainee group, and what the employment rate of participants would have been without the program. If the program drew its participants from the most work-resistant part of the population, then a 40 percent rate might be extremely good. However, if participants were drawn from a work-oriented part of the population-that is, if the population were “creamed”-then its placement rate might be judged inadequate. Placement rates can only be evaluated if one knows the quality of the material with which programs started. — Michael Lipsky, Street-Level Bureaucracy, 390

Categorization

• DEEPTHINKING: The ‘quality of the material’ problem — where programs can look good or bad depending on intake risk — is a genuinely dense point: it disables even outcome metrics, not just process oversight, so nothing external can discipline frontline judgment.
• XREF: connects to: Ambiguous societal goals make street-level performance measurement nearly arbitrary — Michael Lipsky, Street-Level Bureaucracy: Dilemmas of…, 388; Predictive models can suggest case outcomes under counterfactual race-neutral processing. — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law: A Definitive Look at…, 1210; Procedural norms substitute for outcome-control when agent behavior is hard to specify.

Broader Connections

• Ambiguous societal goals make street-level performance measurement nearly arbitrary #corroborates (0.82) — Michael Lipsky, Street-Level Bureaucracy, 388
• Predictive models can suggest case outcomes under counterfactual race-neutral processing. #responds-to (0.78) — Joseph Avery, Jim Sidanius, Joel Cooper, Gregory Davis, Amanda N., Bias in the Law, 1210
• Procedural norms substitute for outcome-control when agent behavior is hard to specify. #qualifies (0.75)