Shortened scale items are unstable and fail to replicate.

Evidence

Finally, shortening a scale on the basis of two studies (a doctoral dissertation and a master’s thesis) involving altogether 311 persons seems rather preliminary to me. Would the same items be discarded in studies involving other, larger samples? The inherent instability of such choices is driven home by a comparison with Schulze’s ten items (chosen on the basis of two studies involving altogether 272 university students). You would expect the ten items Schulze picked to make the Top Twenty on Troldahl and Powell’s list, would you not, if the procedure were valid? But only five of them did, exactly chance on a twenty item version of a forty-item scale. — Bob Altemeyer, The Authoritarian Specter, 4262 (#expands, via linked claim)

Alvarez et al. (1996), who most clearly take a necessary and sufficient condition view of democracy, use no weighting, which is directly related to their use of dichotomous variables. Almost by definition dichotomous variables cannot be weighted (an argument against them); hence the use of dichotomous variables directly leads to unweighted dimensions. — Gary Goertz, Social Science Concepts, 314 (#complicates, via linked claim)

Categorization

• XREF: connects to: Scale-shortening item selections are unstable across different samples — Bob Altemeyer, The Authoritarian Specter, 4262; Shortened scales inherit and add measurement problems from originals — Bob Altemeyer, The Authoritarian Specter, 4249; Dichotomous variables make weighting dimensions impossible, forcing equal treatment — Gary Goertz, Social Science Concepts: A User’s Guide, 314

Broader Connections

• Scale-shortening item selections are unstable across different samples #expands (0.95) — Bob Altemeyer, The Authoritarian Specter, 4262
• Dichotomous variables make weighting dimensions impossible, forcing equal treatment #complicates (0.72) — Gary Goertz, Social Science Concepts, 314