Vague evaluative wording makes survey items unreliable across respondents

Evidence

Written feedback on each item, along with suggestions for additional items to include in the instruments, were requested of reviewers. Feedback on index items included suggestions to increase clarity in terms of the phrasing and wording of items, reducing the number of duplicate items, and adding other locations where Muslims may face discrimination. A reviewer stated that an item regarding media portrayals of Muslims was unclear (“The news media’s portrayal of Muslims is inaccurate.”) as the word “inaccurate” relies on a participant’s understanding of what “inaccurate” implies, the meaning of which may vary among potential participants. This item was dropped and another item on news media depictions (“News media reports on Muslims are primarily about terrorism.”) retained based on this feedback. — Naheed Ahmed, Measurement of Perceived Interpersonal…, 50

Categorization

• XREF: Connects to survey design and psychometrics literature on item clarity and respondent interpretation — e.g., work on how abstract descriptors like ‘inaccurate’ import uncontrolled variance.
• SEED: Angle: ‘The word that ruined the question’ — how one adjective (‘inaccurate’ vs. a concrete factual claim) determines whether a survey item measures perception or measures vocabulary.