Easy cases inflate reliability while hybrid regimes expose measures’ true error.

Evidence

We would suggest two explanations. First, high agreement among existing scales is a product of important numbers of stable autocracies and democracies. There is little disagreement that most advanced capitalist countries have been democratic and many African and Asian countries have been consistently authoritarian. Agreement about a large percentage of the cases fuels the high correlations, suggesting higher levels of scale reliability than actually exist. Second, existing indices get the facts wrong for an important set of cases that contain quite a few countries whose regimes are often in “purgatory”-regimes with often shifting authoritarian and democratic characteristics. (Bowman et al. 2005, 945-46) We would suggest two explanations. — Gary Goertz, Social Science Concepts and Measurement, 1093

Categorization

• SEED: Angle: ‘The Reliability Illusion’ — how benchmarks are validated on unambiguous cases and quietly fail exactly where they matter.
• DEEPTHINKING: Explains why confident measurement consensus on well-studied populations coexists with near-random disagreement on contested ones.