Entropy is minimized when all symbols occur with equal probability.

Evidence

Table 7.9 shows the calculated entropy for four different data sources—the first three correspond to the examples that have been discussed already, and the fourth represents the output of a five-level audio signal. Notice that, except for the case of all symbols having equal probability, all the other sources have a total source information, or entropy, of less than 3 bits. This is always true: the most efficient information source is one that uses all its possible symbols with equal probabilities. If we encoded the sources shown in Table 7.9 using a 3-bit binary code, then we would be wasting bits. — David M. Howard, Jamie Angus, Acoustics and Psychoacoustics, 2633

Categorization

• DEFINE: Entropy is defined operationally as the total source information of a data source, giving the minimum bits needed to encode it.