Repeated identical surveys track longitudinal behavior change

Evidence

However my third study, run in 1991-92, could produce only p < . evidence for the value-manipulation effect. And a fourth experiment conducted in 1992-93 found the value confrontation had virtually no effect on Highs; in fact, the Control Highs were almost significantly more willing to set up the aboriginal scholarships than the Experimental Highs were. Put all four studies together, with all the work and money involved (my money!), and you get nothing. — Bob Altemeyer, The Authoritarian Specter, 4094 (#complicates, via linked claim)

Furthermore, researchers noted long ago that unbalanced scales usually correlate best with other unbalanced scales (Hyman and Sheatsley, 1954; Campbell, Siegman, and Rees, 1967)—or indeed with anything potentially affected by yea-saying, including single-item questions. Shared acquiescence could thus create a connection between any two things, no matter what they (supposedly) measure. — Bob Altemeyer, The Authoritarian Specter, 775 (#complicates, via linked claim)

So unbalanced scales can easily “work” (produce statistically significant connections with something else), whether they have much construct validity or not. In addition, direction-of-wording effects would provide gravy for this meat loaf they serve up. Like most response biases, acquiescence would raise the correlations among the items on a unidirectionally worded test, making it look more internally consistent than the content really is. — Bob Altemeyer, The Authoritarian Specter, 776 (#complicates, via linked claim)

Categorization

• XREF: connects to: Repeated identical methods across twelve years reveal genuine behavioral trends — Bob Altemeyer, The Authoritarian Specter, 2212; Consistent longitudinal methods reveal genuine generational change in sexual behavior; Response acquiescence can fabricate correlations between unrelated survey scales — Bob Altemeyer, The Authoritarian Specter, 775 (+6 more)

Broader Connections

• A social-psychology intervention effect faded across replications to nothing #complicates (0.80) — Bob Altemeyer, The Authoritarian Specter, 4094
• Response acquiescence can fabricate correlations between unrelated survey scales #complicates (0.78) — Bob Altemeyer, The Authoritarian Specter, 775
• One-directional survey items make scales falsely look valid and consistent #complicates (0.75) — Bob Altemeyer, The Authoritarian Specter, 776
• Identical survey responses can stem from opposite interpretations #complicates (0.74) — Bob Altemeyer, The Authoritarian Specter, 3977
• Participant observation yields deeper portraits of mentality than verbal survey data do #complicates (0.72) — John T. Jost and Jim Sidanius, Political Psychology, 369