January 15, 2026 AI & NLP

Tokenization & Subword Embeddings

Understanding how LLMs convert text into numbers: from Byte Pair Encoding to embedding vectors. The complete pipeline from raw text to neural network input.

Introduction: Why Can't Neural Networks Read Text?

Neural networks are mathematical functions that operate on numbers, not text. Before an LLM like GPT-4 or Claude can process the sentence "Hello, world!", it must first convert it into a sequence of numbers that the model can understand.

This conversion happens in three critical steps:

  1. Tokenization - Breaking text into smaller units (tokens)
  2. ID Assignment - Mapping each token to a unique integer ID
  3. Embedding - Converting IDs into dense vector representations
Key Insight: The way we tokenize text fundamentally affects what the model can learn. Poor tokenization leads to poor performance, no matter how large the model is.

The Problem with Word-Level Tokenization

The naive approach is to split text by spaces and treat each word as a token:

"Hello, world!" → ["Hello,", "world!"]

But this creates massive problems:

  • Huge vocabulary - English has 170,000+ words. Add other languages, misspellings, and proper nouns, and you need millions of tokens.
  • Out-of-vocabulary (OOV) words - What happens when the model sees "ChatGPT" but it wasn't in the training data?
  • No morphological understanding - "run", "running", "runner" are treated as completely unrelated words.
  • Memory explosion - A vocabulary of 1 million tokens with 768-dimensional embeddings requires 768 million parameters just for the embedding layer!
The Vocabulary Size Problem: GPT-2 uses ~50,000 tokens. If we used word-level tokenization, we'd need 20x more tokens, making the model much larger and slower.

The Solution: Subword Tokenization

Instead of splitting by words, modern LLMs use subword tokenization. The idea is brilliant: break words into smaller, reusable pieces.

Example: How "unhappiness" Gets Tokenized

Subword Tokenization in Action
"unhappiness"
↓
un
happ
iness

Notice how the model learned that "un-" is a common prefix, "happ" is a root, and "-iness" is a suffix. Now the model can understand:

  • "happiness" = ["happ", "iness"]
  • "unhappy" = ["un", "happ", "y"]
  • "unlikely" = ["un", "like", "ly"]

The model reuses these subword pieces across thousands of words, dramatically reducing vocabulary size while maintaining expressiveness.

Byte Pair Encoding (BPE): The Algorithm Behind GPT

Byte Pair Encoding (BPE) is the most popular subword tokenization algorithm, used by GPT-2, GPT-3, GPT-4, and many other models. It's a simple, elegant algorithm based on data compression.

How BPE Works

BPE starts with individual characters and iteratively merges the most frequent pairs:

BPE Algorithm: Step-by-Step
Step 1: Start with Characters

Split the word "lowest" into individual characters:

l
o
w
e
s
t
Step 2: Find Most Frequent Pair

Across the entire corpus, "es" appears very frequently. Merge it:

l
o
w
es
t
Step 3: Find Next Most Frequent Pair

Now "est" is very common. Merge it:

l
o
w
est
Step 4: Continue Until Target Vocabulary Size

Final result after all merges:

low
est

⚠️ Important: This visualization shows how the word "lowest" is tokenized after BPE training. The vocabulary still contains ALL tokens: ["l", "o", "w", "e", "s", "t", "es", "est", "low", "lowest", ...]. When tokenizing text, we use the longest matching tokens available, so "lowest" becomes ["low", "est"].

Why BPE is Brilliant:
  • Learns from data - Common subwords get their own tokens
  • No OOV problem - Any word can be broken into characters if needed
  • Language-agnostic - Works for any language, even code!
  • Efficient - Vocabulary of ~50K tokens covers most text

BPE Training Process

To build a BPE tokenizer, we:

  1. Start with a corpus of text (e.g., Wikipedia, web pages)
  2. Initialize vocabulary with all individual characters/bytes
  3. Count all adjacent pairs of tokens in the corpus
  4. Merge the most frequent pair into a NEW token (keeping the originals!)
  5. Repeat step 3-4 until we reach the target vocabulary size (e.g., 50,000 tokens)
Important: Vocabulary Growth, Not Replacement!
When we merge "e" + "s" → "es", we ADD "es" as a new token to the vocabulary. We DO NOT remove "e" and "s"! The vocabulary grows with each merge.

This is crucial because:
  • ✅ Fallback mechanism - Rare character combinations can be broken into individual characters
  • ✅ No OOV problem - Any text can be tokenized, even completely new words
  • ✅ Flexibility - The tokenizer chooses the best granularity for each word
Example:
Initial vocab: ["a", "b", "c", ..., "e", "s", "t", ...]
After merge: ["a", "b", "c", ..., "e", "s", "t", ..., "es"]
After another: ["a", "b", "c", ..., "e", "s", "t", ..., "es", "est"]
# Simplified BPE Training Example corpus = ["low", "lower", "lowest", "newer", "newest"] # Iteration 1: Most frequent pair is "es" # Merge: "es" → new token # Result: ["low", "lower", "lowest", "newer", "newest"] # Iteration 2: Most frequent pair is "est" # Merge: "est" → new token # Result: ["low", "lower", "lowest", "newer", "newest"] # Continue until vocabulary size = 50,000...

Step 2: Token ID Assignment

Once we have our vocabulary of tokens, each token is assigned a unique integer ID. This creates a lookup table (vocabulary) that maps tokens to IDs.

Token → ID Mapping
"Hello" 15496
"," 11
" world" 995
"!" 0
"un" 403
"happ" 71
"iness" 1272
"est" 395

For example, the sentence "Hello, world!" becomes:

Text: "Hello, world!" Tokens: ["Hello", ",", " world", "!"] IDs: [15496, 11, 995, 0]
Important: The token IDs are arbitrary integers. ID 15496 doesn't mean anything special - it's just a unique identifier for the token "Hello". The meaning comes from the embedding vectors we'll learn next.

Step 3: From IDs to Embedding Vectors

Token IDs are still just integers. To make them useful for a neural network, we need to convert them into dense vector representations called embeddings.

What is an Embedding?

An embedding is a learned vector of real numbers (typically 768 or 1024 dimensions for modern LLMs) that captures the semantic meaning of a token.

Token ID → Embedding Vector
1. Token ID (Integer)
15496

Token: "Hello"

↓ Lookup in Embedding Table
2. Embedding Table (Learned Parameters)
Matrix of size [vocab_size × embedding_dim]
Example: [50,000 × 768]
0.23
-0.45
0.67
0.12
-0.89
0.34
...
0.56
↓
3. Embedding Vector (768 dimensions)
[0.23, -0.45, 0.67, 0.12, -0.89, 0.34, ..., 0.56]

This vector encodes the semantic meaning of "Hello"

How Embeddings are Learned

The embedding table is a learned parameter of the neural network. During training:

  1. The model starts with random embeddings
  2. As it learns to predict the next token, it adjusts the embeddings
  3. Similar tokens end up with similar vectors (e.g., "happy" and "joyful")
  4. Related tokens cluster together in the embedding space
Embedding Properties:
  • Semantic similarity - Similar words have similar vectors
  • Arithmetic operations - vector("king") - vector("man") + vector("woman") ≈ vector("queen")
  • Dimensionality - Typical sizes: 768 (GPT-2), 1024 (GPT-3), 12,288 (GPT-4)
  • Learned, not hand-crafted - The model discovers patterns automatically

The Complete Pipeline

# Input text text = "Hello, world!" # Step 1: Tokenization (BPE) tokens = tokenizer.encode(text) # Result: ["Hello", ",", " world", "!"] # Step 2: Convert to IDs token_ids = [15496, 11, 995, 0] # Step 3: Lookup embeddings embeddings = embedding_table[token_ids] # Result: Tensor of shape [4, 768] # Each row is a 768-dimensional vector # Now the model can process these vectors!

Conclusion

Tokenization is the critical first step in how LLMs understand text. The journey from raw text to neural network input involves:

  1. Subword tokenization (BPE) - Breaking text into reusable pieces
  2. ID assignment - Mapping tokens to unique integers
  3. Embedding lookup - Converting IDs to dense vectors that capture meaning

This elegant pipeline allows models to handle any text, in any language, with a fixed vocabulary size. The embeddings learned during training capture rich semantic relationships, enabling the model to understand context, meaning, and even perform reasoning.

Key Takeaways:
  • BPE balances vocabulary size with expressiveness
  • Token IDs are just lookup keys - the meaning is in the embeddings
  • Embeddings are learned, not hand-crafted
  • The same tokenization approach works for text, code, and even images (with modifications)
← Back to Blog