Scale-shortening item selections are unstable across different samples

Evidence

Finally, shortening a scale on the basis of two studies (a doctoral dissertation and a master’s thesis) involving altogether 311 persons seems rather preliminary to me. Would the same items be discarded in studies involving other, larger samples? The inherent instability of such choices is driven home by a comparison with Schulze’s ten items (chosen on the basis of two studies involving altogether 272 university students). You would expect the ten items Schulze picked to make the Top Twenty on Troldahl and Powell’s list, would you not, if the procedure were valid? But only five of them did, exactly chance on a twenty item version of a forty-item scale. — Bob Altemeyer, The Authoritarian Specter, 4262

Categorization

• XREF: Connects to the general unreliability-of-measurement theme in psychometrics and the Chapter 1 warning about using this technique to shorten scales.
• QUESTION: The dramatic chance-level overlap between two abbreviated scales (only 5 of 10 items in common) is a vivid demonstration — worth remembering as a cautionary example for methodology sections.