A symbol’s information content rises as its probability of occurring falls

Evidence

For any of the eight symbols shown in Table 7.8, the self-information is 3 bits because they all have the same probability. However, if the probability of a particular symbol was one, that is, it was like our repeating “h” discussed earlier, the self-information would be zero. On the other hand, if the probability of one of the symbols was lower than one eighth, for example, one hundredth or (0.01), then that symbol’s self-information would be log2 (1/0.01), which would equal 6.64 bits, which would mean that symbol was carrying more information than the other symbols. — David M. Howard, Jamie Angus, Acoustics and Psychoacoustics, 2617

Categorization

• DEFINE: Introduces self-information as log2(1/p), the core quantitative definition of information content.