What is Information?
We use the word "information" every day, but in 1948, Claude Shannon gave it a precise mathematical definition that changed the world.
Imagine I tell you two things:
- "The sun will rise tomorrow."
- "There is a giant alien spaceship landing in Times Square right now."
Which sentence contains more information?
The first one tells you almost nothing—you already knew the sun would rise. The probability was basically 100%. The second one is shocking. The probability was near zero.
Measuring Surprise (The Bit)
How do we quantify this "surprise"? We want a function \( h(x) \) that:
- Is 0 if probability \( p(x) = 1 \) (no surprise).
- Increases as \( p(x) \) gets smaller (more surprise).
- Adds up nicely for independent events (surprise of two coin flips = surprise of one + surprise of another).
The only function that fits is the negative logarithm:
Why base 2? Because it gives us units in bits.
| Event | Probability \( p(x) \) | Information \( -\log_2(p) \) | Intuition |
|---|---|---|---|
| Sun rising | 1.0 (100%) | 0 bits | Zero information. Waste of breath. |
| Coin Flip | 0.5 (50%) | 1 bit | Resolves one binary choice. |
| Rolling a 4 (Die) | 0.166 (1/6) | 2.58 bits | More surprising than a coin flip. |
| Winning Lottery | 0.0000001 | ~23 bits | Huge amount of information! |
Entropy: The Average Surprise
So \( -\log(p) \) is the information for a single event. But usually, we care about the entire probability distribution.
Entropy (H) is simply the expected value (average) of the information. It tells us how "unpredictable" a source is on average.
Visualizing Entropy
Let's compare two coins: a Fair Coin and a Rigged Coin (always Heads).
Fair Coin (50/50)
P(Heads) = 0.5
Info = -log(0.5) = 1 bit
P(Tails) = 0.5
Info = -log(0.5) = 1 bit
Average (Entropy):
0.5(1) + 0.5(1) = 1 bit
Maximum uncertainty.
Rigged Coin (99/1)
P(Heads) = 0.99
Info = -log(0.99) ≈ 0.01 bits
P(Tails) = 0.01
Info = -log(0.01) ≈ 6.6 bits
Average (Entropy):
0.99(0.01) + 0.01(6.6) ≈ 0.08 bits
Very predictable. Low entropy.
High Entropy = Very unpredictable (Uniform distribution).
Low Entropy = Very predictable (Spiky distribution).
Why does this matter for AI?
In Machine Learning, we are constantly dealing with probability distributions.
- Classification: We want our model to be confident (Low Entropy) about the correct class.
- Exploration (RL): We want an agent to explore states where it is uncertain (High Entropy).
- Generative Models: We want to generate diverse samples (High Entropy) rather than the same image over and over (Low Entropy).
Entropy gives us the language to talk about how "spread out" or "focused" a distribution is.
Next Up: The Cost of Being Wrong
Now that we know how to measure information for a single distribution, what happens when we have two?
What if the true weather is 50/50, but our model thinks it's 99/1? We will be very surprised when it rains. In Part 2, we'll introduce Cross-Entropy, the most popular loss function in Deep Learning.