Hunting through multidimensional data produces spurious significant findings.

Evidence

If one closely examines two sets of complex multidimensional data it will usually be possible to find some differences between the two that are consistent enough to be statistically significant. Suppose all of the conceivable dimensions of contrast were, in the full population of subjects, unrelated to the dimension that governs the division of our data into two sets. Suppose that across the whole population of middle class Americans none of the coding categories that might be used in an analysis is significantly related to being prejudiced or unprejudiced. It could nevertheless happen that in any small sample from this population some categories would be related to prejudice at such levels of significance as a p of . or .01. Consider what the . level of significance means: differences as great or greater than the one obtained would not occur more than five times out of a hundred in samples of this size if there were no difference in the population in question. If we had predicted our differences in advance and they were significant at this level we could be reasonably confident that these were not chance outcomes. However, if we permit ourselves to pick over the data until we find something significant then we may simply be seizing upon those few of the hundreds of conceivable contrasts which will in any particular sample fall by chance into a five-times-in-a-hundred pattern of contrast. — John T. Jost and Jim Sidanius, Political Psychology, 602

Categorization

• XREF: This is a classic statement of the multiple-comparisons / data-dredging problem, connecting to the replication crisis in psychology and to p-hacking literature. — connects to: Small studies more often yield spurious results a propagandist can exploit — Cailin O’Connor and James Owen Weatherall, The Misinformation Age: How False…, 365
• DEEPTHINKING: The p-value-as-audition point — that a 5-in-100 chance event becomes near-certain to appear when scanning hundreds of contrasts — is a fundamental epistemological warning about exploratory analysis that rewards re-reading.
• QUESTION: Worth chasing: how did the pre-registration movement evolve out of exactly these concerns, and is the lesson fully understood by modern practitioners?