When one measure is a subset of another, their disagreement is patterned, not random.

Evidence

Suppose that the relationship between the two measures is like that of table 7.1 (see chapter 3 in the previous edition of this book for an extended analysis with real data). There is excellent agreement on the extremes but some substantial disagreement in the middle. Yet a high correlation of 0.87 masks large differences between the two. Notably, measure X1 is always less than measure X2. This means that one measure is a subset of the other, which can be interpreted as meaning that one is a stricter measure. This then can have a significant impact on causal inference. But because a large percentage of observations do lie on the diagonal, one will get substantial correlations. This example suggests that there may not only be disagreement in the middle zone but there is a pattern to that disagreement. — Gary Goertz, Social Science Concepts and Measurement, 1101

Categorization

• SEED: Article angle: ‘The 0.87 Trap’ — how subset/stricter measures systematically bias causal inference while passing every reliability check.