Statistically significant p-values sometimes reflect nothing real

Evidence

We have to remember that every now and then when p is < .05, nothing is actually there. But if a “gossamer finding” materializes the first time you study something, it may take you years to discover you are chasing “the stuff that dreams are made of.” — Bob Altemeyer, The Authoritarian Specter, 4096 (#expands, via linked claim)

The larger samples used in Study 2 have the advantage that they are more likely to be representative of the population, but they are also more likely to produce significant correlations that explain little variance. — Roderick M. Kramer, Geoffrey J. Leonardelli, Robert W. Livingston, Social Cognition, Social Identity, and…, 1036 (#complicates, via linked claim)

Categorization

• XREF: connects to: Initial significant results often reflect chance and vanish on replication — Bob Altemeyer, The Authoritarian Specter, 4096; Large samples produce significant correlations that explain little variance — Roderick M. Kramer, Geoffrey J. Leonardelli, Robert W. Livingston, Social Cognition, Social Identity, and…, 1036; Thresholds are statistical detection probabilities, not absolute cutoffs — Jennifer Lentz, Psychoacoustics- Perception of Normal…, 284 (+3 more)

Broader Connections

• Initial significant results often reflect chance and vanish on replication #expands (0.92) — Bob Altemeyer, The Authoritarian Specter, 4096
• Large samples produce significant correlations that explain little variance #complicates (0.75) — Roderick M. Kramer, Geoffrey J. Leonardelli, Robert W. Livingston, Social Cognition, Social Identity, and…, 1036