Adding irrelevant cases inflates significance and can flip causal signs

Evidence

The use of concepts to select cases has a variety of very important impacts on causal inference. As the Raven Paradox illustrates, one can easily inflate the number of observations and hence significance levels. Causal strength and direction depend on the distribution of observations in the negative set (and the positive, of course). Because observations are not randomly distributed in the irrelevant set, significance levels and signs can easily vary when irrelevant observations are included. Finally, the use of variables simultaneously to define populations and in basic-level statistical analyses poses major issues of causal interpretation. One must constantly keep in mind how the population is defined when making causal interpretations of statistical results. — Gary Goertz, Social Science Concepts, 1561

Categorization

• DEEPTHINKING: The Raven Paradox framing — that confirming a hypothesis with irrelevant observations is a feature of case-selection, not just a curiosity — reframes routine dataset construction as a validity threat in every observational study.
• SEED: Angle: ‘The Raven Paradox ate your regression’ — how inclusion criteria silently manufacture or destroy findings across published political science.