Defining the relevant population changes causal findings before any sampling occurs

Evidence

We have shown how the Possibility Principle addresses fundamental but rarely discussed issues entailed in defining a relevant population of cases. The definition of the relevant population can affect findings about the significance, strength, and even direction of hypothesized causal relationships. The relevant population therefore should be carefully defined before one turns to specific techniques of case sampling. Regardless of the sampling procedure, when irrelevant cases are included in one’s understanding of the population, one will overrepresent observations with a zero value on the dependent variable. Thus, our discussion has called attention to ways of defining the population that generate too many negative cases, whereas most of the literature on case selection has focused on procedures that generate error through an insufficient number of negative cases. — Gary Goertz, Social Science Concepts, 1405

Categorization

• SEED: A piece on the neglected error of too many negative cases: most methods scholarship worries about too few, but denominator padding biases findings just as badly in the opposite direction.