Information Theory Part 2: Cross-Entropy

The Setup: Reality vs. Prediction

In Part 1, we learned that Entropy tells us the average number of bits needed to encode events from a distribution \( P \), assuming we know \( P \) perfectly.

But in machine learning, we never know the true distribution \( P \) (reality). We only have our model's estimate, \( Q \) (prediction).

Cross-Entropy answers a practical question:

                    If events actually happen according to \( P \), but we encode them using our estimated distribution
                    \( Q \), what is the average number of bits we use?
                

The Formula

The formula for Cross-Entropy \( H(P, Q) \) looks very similar to Entropy:

H(P, Q) = -\sum_{x} P(x) \log Q(x)

Let's dissect it:

\( P(x) \): The true probability. This tells us how often the event actually happens.
\( -\log Q(x) \): The estimated information. This is the length of the message we assigned based on our model's belief.

We are taking the average message length (based on \( Q \)) weighted by the true frequency (based on \( P \)).

Example: The Weather Forecast

Let's say the weather can be Sunny or Rainy.

Reality (P)

It's actually a rainy season.

P(Sunny) = 0.1

P(Rainy) = 0.9

Model (Q)

Our model is optimistic.

Q(Sunny) = 0.8

Q(Rainy) = 0.2

Calculating Cross-Entropy

How well does our model perform?

\begin{align} H(P, Q) &= - [ P(\text{Sunny})\log Q(\text{Sunny}) + P(\text{Rainy})\log Q(\text{Rainy}) ] \\ &= - [ 0.1 \cdot \log(0.8) + 0.9 \cdot \log(0.2) ] \\ &= - [ 0.1(-0.32) + 0.9(-2.32) ] \\ &= - [ -0.032 - 2.088 ] \\ &\approx 2.12 \text{ bits} \end{align}

Compare this to the optimal Entropy \( H(P) \):
If we knew the truth, the entropy would be: \( -(0.1\log 0.1 + 0.9\log 0.9) \approx 0.47 \text{ bits} \).

Because our model was wrong, we used 2.12 bits instead of 0.47 bits. That's a huge penalty!

Reality vs Model: Weather Predictions

☀️ Sunny

Reality: 10%

Model: 80%

❌ Large Error!

🌧️ Rainy

Reality: 90%

Model: 20%

❌ Large Error!

Reality (P)

Model (Q)

Optimal Cost (Entropy): 0.47 bits

                            Actual Cost (Cross-Entropy):
                            2.12 bits
                        

Penalty for being wrong: +1.65 bits

Why is this a Loss Function?

In Deep Learning classification tasks, we use Categorical Cross-Entropy Loss. Why?

Look at the formula again. Since \( P(x) \) is fixed (it comes from our dataset labels), the only way to minimize \( H(P, Q) \) is to make \( Q(x) \) as close to \( P(x) \) as possible.

Specifically, if the true label is class \( k \) (so \( P(k)=1 \) and others are 0), the formula simplifies to:

\text{Loss} = - \log Q(k)

This means: "Maximize the log-probability of the correct class."

                    Connection to MLE: Minimizing Cross-Entropy is mathematically identical to
                    Maximizing Likelihood (MLE)!
                

Next Up: Measuring the Difference

We saw that our wrong weather model cost us 2.12 bits, while the optimal cost was 0.47 bits.

The difference \( 2.12 - 0.47 = 1.65 \) bits represents the pure inefficiency of our model. This difference has a special name: KL Divergence.

In Part 3, we'll finally put it all together.

← Part 1: Entropy Next: Part 3 - KL Divergence →