Statistically significant results can mask large, decisive differences between measures.

Evidence

In both cases, most of the measures did show a significant relationship with dispute initiation. We suspect that in many situations this will be the case. So scholars will almost inevitably be confronted with choosing among various (dyadic) measures, most of which will produce significant results. At the same time our analyses show that the strength of these positive results can vary significantly across measures; hence an attitude that “they are all about the same” is not confirmed at all. — Gary Goertz, Social Science Concepts, 990

Categorization

• DEEPTHINKING: Cuts against the reflexive ‘they’re all basically the same’ empiricism of quantitative work; the routine absence of measurement debate conceals the fact that effect sizes swing by a third across equally defensible measures.
• SEED: Angle: ‘The Comfort of Significance’ — a piece challenging the practice of treating any p<0.05 finding as equivalent to any other, using dyadic trade measures as the worked case.