Unbalanced survey scales allow response bias to inflate apparent reliability
Evidence
Having contraits on a test will not make people more motivated, although it probably slows down sloppy, automatic responding that continual protraits might spark. But it does let you use the invalid “agrees” from one half of the items to cancel out the invalid “agrees” from the other half, when the answers get summed. The same applies to invalid “disagrees.” You do have to balance scales, for uncontrolled yea-saying and naysaying can produce buckets of error variance. And all the while they will lead you to believe, through inflated alphas and puffed up empirical correlations with other unbalanced measures, that things are great. The road to Measurement Hell is paved with protrait items. — Bob Altemeyer, The Authoritarian Specter, 820
Categorization
Category: Causal Relationships
- DEFINE: Defines contraits and protraits as opposite- vs. same-direction survey items, and explains the cancellation logic.
Related
- Unbalanced scales inflate reliability alphas and correlations through acquiescence (similarity 0.93)
- Balanced scales understate a test’s real internal consistency (similarity 0.85)
- One-directional survey items make scales falsely look valid and consistent (similarity 0.85)
- Response bias makes balanced surveys appear spurious multifactor structures (similarity 0.84)
- Strongly held beliefs reduce response-set bias in survey measures (similarity 0.81)
- Unbalanced scales can show spurious significance without true construct validity (similarity 0.81)
- Unbalanced survey scales produce inflated internal correlations but deflated cross-measure correlations (similarity 0.80)
- Response acquiescence can forge fake statistical links between unrelated measures (similarity 0.79)
- Response acquiescence can fabricate correlations between unrelated survey scales (similarity 0.78)