Researchers can inflate apparent validity by cherry-picking a test’s best items.

Evidence

Actually, almost all such studies produced confirmatory results. However, such confirmatory results are one of the reasons I am wary of studies that use only part of a test. You can, if you choose the items carefully, get an appreciably stronger relationship between a miniversion of a scale and some target variable of interest than you could with the scale as a whole. It will then appear to the uncritical reviewer that the construct validity of the test as a whole has been splendidly verified. (This is an old trick, but it apparently still works.) Another potential “sleight of hand” maneuver involves using just the items with the best intercorrelations on the test. This also ignores the probability that those items no longer faithfully represent the construct for which the test was conceived and named. — Bob Altemeyer, The Authoritarian Specter, 3949

Categorization

• SEED: An essay on how item-subset cherry-picking lets researchers manufacture strong-looking scale validations.
• XREF: Echoes methodological critique common across psychology literatures in the vault (survey validity problems).