Information content increases as a message becomes less probable

Evidence

On the other hand, if, during a web chat, you got a message that said “hhhhhhhhhhhhhhhhhhhhhhhh …” it would be carrying no information, other than the possibility that the other per­ son has fallen asleep on the keyboard! So the more surprising a thing is the more information it carries. That is, the less probable something is the more information it carries. — David Howard, Jamie Angus, coustics and Psychoacoustics, 2587

Categorization

• SHARE: A tweetable, self-contained illustration of information theory — send to someone who works on compression or signal processing and would enjoy the ‘asleep on the keyboard’ framing.
• XREF: Shannon-style information theory’s core definition; ties directly to the entropy-coding stage described in the same chapter.