A data source’s entropy gives the minimum bits needed to encode it

Evidence

This is also known as the “entropy” because the equation is analogous to one used to calculate entropy in physics. In general we are interested in the total information content, or entropy of the data source, because this gives us the minimum number of bits required to encode it. — David Howard, Jamie Angus, coustics and Psychoacoustics, 2625

Categorization

• DEFINE: Establishes ‘entropy’ as information theory’s term for total information content, borrowed by analogy from physics.
• XREF: Connects to Shannon information theory and any prior reading on signal bandwidth versus information rate. — connects to: Unequal symbol probabilities always reduce a source’s entropy below its bit capacity; Entropy coding trade-offs limit how close code rates approach information-theoretic limits; Arithmetic coding approaches source entropy better than table-based Huffman methods — David M. Howard, Jamie Angus, Acoustics and Psychoacoustics, 2644 (+1 more)
• corroborated: 3 independent passages support this claim — David Howard, Jamie Angus, coustics and Psychoacoustics, 2625

Broader Connections

• Unequal symbol probabilities always reduce a source’s entropy below its bit capacity #complicates (0.85)
• Entropy coding trade-offs limit how close code rates approach information-theoretic limits #qualifies (0.80)
• Arithmetic coding approaches source entropy better than table-based Huffman methods #expands (0.80) — David M. Howard, Jamie Angus, Acoustics and Psychoacoustics, 2644
• Codec effectiveness trades off against bits spent on side information #qualifies (0.75)