GloVe Explained: Global Vectors for Word Representation

The Gap Between Two Worlds

In 2013, Word2Vec revolutionized NLP with its neural, predictive approach to learning word embeddings. It was fast, scalable, and produced remarkable results on analogy tasks like "king - man + woman = queen".

But there was a catch: Word2Vec only looked at local context windows. It would slide a window over text, learning from immediate neighbors, but never seeing the full picture of how words co-occur across the entire corpus.

Meanwhile, traditional methods like Latent Semantic Analysis (LSA) used global co-occurrence statistics—counting how often words appeared together across the entire dataset. These methods captured global structure but performed poorly on tasks like analogies.

                    The Question: What if we could combine the best of both approaches?
                    Use global statistics like LSA, but learn vectors that perform well on semantic tasks like Word2Vec?
                

This is exactly what GloVe (Global Vectors for Word Representation) achieves. Published by Pennington, Socher, and Manning at Stanford in 2014, GloVe bridges the gap between count-based and predictive models.

1. Building the Co-occurrence Matrix

GloVe starts by constructing a co-occurrence matrix \(X\) from the entire corpus. Let \(X_{ij}\) represent the number of times word \(j\) appears in the context of word \(i\).

Unlike Word2Vec, which processes text sequentially and discards information after each update, GloVe makes a single pass through the corpus to build this matrix, capturing all global statistics.

Example Co-occurrence Matrix

	ice	steam	solid	gas	water	fashion
ice	8.9	0.02	5.4	0.04	3.6	0.01
steam	0.02	8.1	0.05	5.2	3.4	0.01

Notice: "ice" co-occurs frequently with "solid", while "steam" co-occurs frequently with "gas"

We also define \(X_i = \sum_k X_{ik}\) as the total number of times any word appears in the context of word \(i\), and \(P_{ij} = P(j|i) = X_{ij} / X_i\) as the probability that word \(j\) appears in the context of word \(i\).

2. The Key Insight: Ratios Reveal Meaning

Here's the brilliant observation from the GloVe paper: the ratio of co-occurrence probabilities encodes semantic relationships better than raw probabilities or counts.

Consider two words \(i\) and \(j\) (like "ice" and "steam"), and a probe word \(k\). The ratio \(P_{ik} / P_{jk}\) reveals how \(k\) relates to \(i\) versus \(j\):

Probe word: solid

P(solid|ice) / P(solid|steam) = 8.9

Related to ice, not steam

Probe word: gas

P(gas|ice) / P(gas|steam) = 0.008

Related to steam, not ice

Probe word: water

P(water|ice) / P(water|steam) = 1.06

Related to both equally

Probe word: fashion

P(fashion|ice) / P(fashion|steam) = 1.0

Unrelated to both

Why ratios?

Large ratio (>> 1): probe word is related to \(i\) but not \(j\)
Small ratio (<< 1): probe word is related to \(j\) but not \(i\)
Ratio ≈ 1: probe word is either related to both or neither

This ratio structure naturally discriminates between relevant and irrelevant words, making it perfect for learning semantic relationships.

3. Deriving the Objective Function

Now comes the mathematical elegance. We want to design a function that captures these ratio relationships using word vectors. Let's derive it step by step, with full justification for each choice.

Step 1: Start with a General Function

We want some function \(F\) that relates word vectors to the probability ratio:

F(w_i, w_j, \tilde{w}_k) = \frac{P_{ik}}{P_{jk}}

Where \(w_i, w_j\) are word vectors and \(\tilde{w}_k\) is a context vector

This is our starting point: we want a function that takes three vectors and outputs the ratio of probabilities. But what form should \(F\) take?

Step 2: Enforce Linear Structure

Since we're working in vector space, we want \(F\) to respect the linear structure of vectors. The key insight: the ratio depends on the difference between \(w_i\) and \(w_j\), not their individual values.

Why the difference? Think about it: if we swap \(i\) and \(j\), the ratio inverts: \(\frac{P_{ik}}{P_{jk}} \rightarrow \frac{P_{jk}}{P_{ik}}\). This suggests the function should depend on \(w_i - w_j\), because swapping gives us \(w_j - w_i = -(w_i - w_j)\).

So we refine our function:

F(w_i - w_j, \tilde{w}_k) = \frac{P_{ik}}{P_{jk}}

Step 3: From Vectors to Scalars

We have a vector difference \(w_i - w_j\) and a context vector \(\tilde{w}_k\), but we need a scalar output (the ratio). The most natural way to combine two vectors into a scalar is the dot product:

F\left((w_i - w_j)^T \tilde{w}_k\right) = \frac{P_{ik}}{P_{jk}}

Now \(F\) is a function from scalars to scalars: \(F: \mathbb{R} \rightarrow \mathbb{R}\).

Step 4: The Homomorphism Constraint

Here's the crucial mathematical insight. Notice that:

The left side involves vector addition: \((w_i - w_j)^T \tilde{w}_k = w_i^T \tilde{w}_k - w_j^T \tilde{w}_k\)
The right side involves scalar multiplication (division is multiplication by inverse): \(\frac{P_{ik}}{P_{jk}} = P_{ik} \cdot P_{jk}^{-1}\)

We need \(F\) to convert addition into multiplication. In mathematical terms, \(F\) must be a homomorphism from the additive group \((\mathbb{R}, +)\) to the multiplicative group \((\mathbb{R}^+, \times)\).

                    Homomorphism Property: We need \(F(a + b) = F(a) \cdot F(b)\) for all \(a, b \in
                    \mathbb{R}\).
                    
                    The only continuous function with this property is the exponential: \(F(x) =
                    \exp(x)\)
                    
                    Proof: \(\exp(a + b) = \exp(a) \cdot \exp(b)\) ✓

Step 5: Apply the Exponential

Using \(F(x) = \exp(x)\), we get:

\exp\left((w_i - w_j)^T \tilde{w}_k\right) = \frac{P_{ik}}{P_{jk}}

Expanding the left side using the exponential property:

\exp(w_i^T \tilde{w}_k - w_j^T \tilde{w}_k) = \frac{\exp(w_i^T \tilde{w}_k)}{\exp(w_j^T \tilde{w}_k)} = \frac{P_{ik}}{P_{jk}}

Step 6: Solve for Individual Probabilities

From the equation above, we can deduce:

\exp(w_i^T \tilde{w}_k) = \lambda_i \cdot P_{ik}

where \(\lambda_i\) is some constant that depends only on \(i\) (not on \(k\)). Why? Because when we form the ratio, \(\lambda_i\) cancels out: \(\frac{\lambda_i P_{ik}}{\lambda_j P_{jk}} = \frac{\lambda_i}{\lambda_j} \cdot \frac{P_{ik}}{P_{jk}}\). For this to equal \(\frac{P_{ik}}{P_{jk}}\), we need \(\lambda_i = \lambda_j\) for all \(i, j\), so we can call it just \(\lambda\).

Taking the logarithm of both sides:

w_i^T \tilde{w}_k = \log(\lambda) + \log(P_{ik})

Recall that \(P_{ik} = \frac{X_{ik}}{X_i}\), so:

w_i^T \tilde{w}_k = \log(\lambda) + \log(X_{ik}) - \log(X_i)

Step 7: Restore Symmetry with Bias Terms

The co-occurrence matrix is symmetric: \(X_{ik} = X_{ki}\) (if word \(i\) appears near word \(k\), then word \(k\) appears near word \(i\)). But our current formulation isn't symmetric in \(i\) and \(k\).

To fix this, we absorb the constants into bias terms:

Let \(b_i = \log(\lambda) - \log(X_i)\) be a bias for word \(i\)
Let \(\tilde{b}_k\) be a bias for context word \(k\)

This gives us the final, symmetric relationship:

w_i^T \tilde{w}_k + b_i + \tilde{b}_k = \log(X_{ik})

                    The Final Relationship: This beautiful equation says that the dot product of word
                    vectors,
                    plus bias terms, should equal the logarithm of their co-occurrence count.
                    
                    This is essentially matrix factorization of the log co-occurrence matrix!
                    We're trying to factorize \(\log(X)\) into the product of two matrices: \(W\) and \(\tilde{W}^T\).

Why can't we enforce this exactly?

Sparsity: Many word pairs never co-occur, so \(X_{ik} = 0\) and \(\log(0)\) is undefined
Noise: Rare co-occurrences are noisy and shouldn't be trusted equally with frequent ones
Dominance: Very frequent words would dominate if we weighted all pairs equally

This is why we need a weighted least squares objective instead of exact equality.

4. The Weighted Least Squares Objective

We now have a target relationship: \(w_i^T \tilde{w}_j + b_i + \tilde{b}_j = \log(X_{ij})\). But we can't enforce this exactly for every word pair because:

\(\log(0)\) is undefined for word pairs that never co-occur
Rare co-occurrences are noisy and shouldn't be trusted as much
Very frequent words (like "the") would dominate the objective

The solution is a weighted least squares objective:

J = \sum_{i,j=1}^{V} f(X_{ij}) \left( w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2

Where \(f(X_{ij})\) is a weighting function that balances rare and frequent co-occurrences

The Weighting Function

The weighting function \(f(x)\) should have these properties:

\(f(0) = 0\) so that \(\log(0)\) is never computed
\(f(x)\) should be non-decreasing (trust frequent co-occurrences more)
\(f(x)\) should be relatively small for large \(x\) (prevent very frequent words from dominating)

The paper proposes:

f(x) = \begin{cases} (x/x_{\text{max}})^\alpha & \text{if } x < x_{\text{max}} \\ 1 & \text{otherwise} \end{cases}

Standard values: \(x_{\text{max}} = 100\), \(\alpha = 3/4\)

Weighting Function f(x)

The function increases for rare words, then caps at 1 to prevent frequent words from dominating

Why this weighting? Without it, very common words like "the" and "and" would dominate the loss function. The cap at \(x_{\text{max}}\) ensures that all frequent co-occurrences contribute equally, while the power \(\alpha < 1\) gives some weight to rarer co-occurrences.

5. Training GloVe

The training process is straightforward:

Build the co-occurrence matrix \(X\) from the corpus (single pass)
Initialize word vectors \(w\) and context vectors \(\tilde{w}\) randomly
Optimize the weighted least squares objective using AdaGrad
Final vectors = \((w + \tilde{w}) / 2\) (average word and context vectors)

                    
// Pseudocode for GloVe training

// 1. Build co-occurrence matrix

X = build_cooccurrence_matrix(corpus, window_size)

// 2. Initialize vectors

W = random_init(vocab_size, vector_dim)

W_tilde = random_init(vocab_size, vector_dim)

b = zeros(vocab_size)

b_tilde = zeros(vocab_size)

// 3. Training loop

for epoch in range(num_epochs):

    for (i, j, X_ij) in non_zero_entries(X):

        // Compute loss

        diff = dot(W[i], W_tilde[j]) + b[i] + b_tilde[j] - log(X_ij)

        loss = f(X_ij) * diff^2

        // Compute gradients and update (AdaGrad)

        grad_W[i] = f(X_ij) * diff * W_tilde[j]

        grad_W_tilde[j] = f(X_ij) * diff * W[i]

        // ... update with AdaGrad

// 4. Final embeddings

embeddings = (W + W_tilde) / 2

Why average word and context vectors? Due to the symmetry in the objective, \(w\) and \(\tilde{w}\) are equivalent. Averaging them reduces noise and often improves performance.

6. GloVe vs. Word2Vec

Both methods learn high-quality word embeddings, but they take fundamentally different approaches:

Word2Vec (Skip-gram)

Approach: Predictive, neural network
Scope: Local context windows
Training: Online, streaming over corpus
Objective: Maximize probability of context given word
Pros: Fast, scalable, works well on huge corpora
Cons: Doesn't leverage global statistics

GloVe

Approach: Count-based, matrix factorization
Scope: Global co-occurrence statistics
Training: Batch, on co-occurrence matrix
Objective: Factorize log co-occurrence matrix
Pros: Uses all corpus statistics, interpretable
Cons: Requires building full co-occurrence matrix

Word2Vec Philosophy

"Learn by predicting context from words, one window at a time"

GloVe Philosophy

"Learn by factorizing global co-occurrence statistics into vector space"

Common Ground

Both learn dense vectors where similar words have similar representations

                    Empirical Results: The GloVe paper shows that GloVe often outperforms Word2Vec
                    on word analogy tasks, especially when trained on the same corpus. However, both methods produce
                    high-quality embeddings, and the choice often depends on practical considerations like corpus size
                    and computational resources.
                

Conclusion

GloVe elegantly bridges two paradigms in word embedding research:

The global statistical information of count-based methods (LSA)
The meaningful vector space of predictive methods (Word2Vec)

By starting with the insight that ratios of co-occurrence probabilities encode semantic relationships, the paper derives a principled objective function that factorizes the log co-occurrence matrix into word vectors.

The result is a model that:

Leverages all corpus statistics (unlike Word2Vec's local windows)
Produces vectors that excel at semantic tasks (unlike traditional count-based methods)
Has a clear, interpretable objective (weighted matrix factorization)

While modern NLP has moved toward contextual embeddings (BERT, GPT), GloVe remains an important milestone in understanding how global statistics and vector semantics can work together to capture the meaning of words.