Deep Dive into Large Language Models: From Raw Data to ChatGPT

Introduction: The Two-Stage Training Pipeline

Large Language Models (LLMs) like ChatGPT, Claude, and Gemini have revolutionized how we interact with AI. But how are these models actually built? What transforms billions of web pages into an intelligent assistant that can write code, answer questions, and engage in nuanced conversations?

This article is based on Andrej Karpathy's excellent deep dive into LLMs, where he breaks down the entire training pipeline into digestible concepts. Watch the full video here:

                    The Big Picture: Training an LLM involves two major stages:
                    Pretraining - Teaching the model to understand language by predicting the
                            next word in billions of documents
Post-Training (Fine-Tuning) - Teaching the model to be a helpful assistant
                            through supervised learning and human feedback

                

Stage 1: Pretraining - Building the Foundation

Pretraining is where the model learns the fundamental structure of language. This is the most computationally expensive phase, often costing millions of dollars and requiring thousands of GPUs running for weeks or months.

Step 1: Data Collection and Preprocessing

The journey begins with collecting massive amounts of text data from the internet. The most common source is Common Crawl, a nonprofit that continuously archives the web, capturing petabytes of data.

URL Filtering

Not all websites are created equal. The first step is filtering URLs to remove:

Spam and adult content
Low-quality or auto-generated sites
Duplicate content
Sites with privacy concerns or personal data

This is often done using curated lists of high-quality domains (like Wikipedia, GitHub, StackOverflow) and machine learning classifiers trained to identify quality content.

Text Extraction

Web pages contain HTML, CSS, JavaScript, and ads—we only want the actual text content. Tools like Trafilatura or custom parsers extract clean text while preserving structure (paragraphs, headings, lists).

Language Filtering

Most LLMs are primarily English-focused (though multilingual models exist). Language detection models identify and filter documents by language. Additional quality filters remove:

Documents with excessive profanity or toxicity
Extremely short or repetitive content
Content with poor grammar or readability scores

Deduplication

The internet is full of duplicates. Deduplication algorithms (like MinHash or exact matching) remove near-identical documents to prevent the model from memorizing repeated content.

FineWeb Dataset: The result of this preprocessing pipeline is a high-quality dataset like FineWeb (15 trillion tokens) or The Pile (800GB). These datasets represent the cleaned, filtered essence of human knowledge on the internet.

Data Preprocessing Pipeline

📄

Raw Data

Common Crawl

→

🔍

URL Filter

Remove spam

→

📝

Text Extract

Clean HTML

→

🌍

Language Filter

Quality check

→

✨

FineWeb

15T tokens

Step 2: Tokenization

Neural networks don't understand text—they work with numbers. Tokenization is the process of converting text into numerical tokens that the model can process.

What is a Token? A token is a chunk of text—it could be a word, part of a word, or even a single character. Modern LLMs use subword tokenization (like Byte-Pair Encoding or SentencePiece) that balances vocabulary size with flexibility.

Tokenization Process

"The cat sat on the mat"

↓

The

cat

sat

on

the

mat

↓

15

2834

3332

319

262

2603

The model sees the sequence of integers [15, 2834, 3332, ...], not the words themselves.

Common vocabulary sizes range from 32,000 to 100,000 tokens. GPT-4 uses a vocabulary of approximately 100,000 tokens, allowing it to efficiently represent text in many languages.

Step 3: The Training Objective - Next Token Prediction

The core of pretraining is deceptively simple: predict the next token.

Autoregressive Training: Given a sequence of tokens, the model learns to predict the next token. This is repeated billions of times across the entire dataset.

Autoregressive Next-Token Prediction

Input: "The cat"

→

sat

(Prob: 12%)

Input: "The cat sat"

→

on

(Prob: 45%)

Input: "The cat sat on"

→

the

(Prob: 88%)

This simple objective forces the model to learn:

Grammar and syntax - How words combine to form valid sentences
Semantics - The meaning and relationships between concepts
World knowledge - Facts, dates, names, and common sense
Patterns - Code syntax, mathematical notation, structured data

Step 4: Context Window

The model can't see the entire internet at once. It processes text in chunks called the context window.

Context Window Limitations: Early models like GPT-2 had a context window of 1,024 tokens (~750 words). Modern models have much larger windows:

GPT-3.5: 4,096 tokens
GPT-4: 8,192 tokens (with 32K and 128K variants)
Claude 3: 200,000 tokens (~150,000 words)

The model can only "see" and remember information within this window.

Interactive Visualization: Sliding Context Window

Watch how the fixed-size context window slides over the token sequence

The

cat

sat

on

the

mat

and

looked

at

the

bird

in

the

tree

Key Insight: The window size is fixed (e.g., 8 tokens shown here). As new tokens are processed, older tokens slide out of view. The model has no memory of tokens outside the current window.

Step 5: The Result - A Base Model

After weeks of training on thousands of GPUs, we get a base model. This model is incredibly knowledgeable but not very useful as an assistant.

What Base Models Are Good At

Completing documents naturally
Understanding context and patterns
Generating coherent text
Code completion

What Base Models Struggle With

Following instructions
Answering questions directly
Refusing harmful requests
Conversational dialogue

                    
// Base Model Behavior

User: "What is the capital of France?"

Base Model: "What is the capital of Germany? What is the capital of Italy?..."

// It continues the pattern, not answering the question!

Stage 2: Post-Training - Creating an Assistant

Post-training transforms the base model into a helpful, harmless, and honest assistant. This stage is much cheaper than pretraining (thousands vs. millions of dollars) but critically important.

Phase 1: Supervised Fine-Tuning (SFT)

In SFT, we teach the model how to behave like an assistant by showing it examples of ideal conversations.

Creating Training Data

Human labelers (often contractors) write thousands of high-quality conversation examples:

                        
User: "Explain quantum computing in simple terms"

Assistant: "Quantum computing uses quantum mechanics principles like

superposition and entanglement to process information. Unlike classical

computers that use bits (0 or 1), quantum computers use qubits that can

be both 0 and 1 simultaneously..."

User: "How do I hack into someone's email?"

Assistant: "I can't help with that. Hacking into someone's email is

illegal and violates privacy. If you're locked out of your own account,

I can help you with legitimate account recovery methods..."

The model is fine-tuned on these examples using the same next-token prediction objective, but now it learns to:

Respond to questions directly
Follow instructions accurately
Maintain a helpful and polite tone
Refuse inappropriate requests
Admit when it doesn't know something

Phase 2: Reinforcement Learning from Human Feedback (RLHF)

RLHF is the secret sauce that makes models like ChatGPT truly impressive. Instead of just imitating examples, the model learns to optimize for human preferences.

Step 1: Collect Comparisons

For a given prompt, the model generates multiple responses. Human evaluators rank these responses from best to worst:

                        
Prompt: "Explain why the sky is blue"

Response A: "The sky appears blue due to Rayleigh scattering..."

Ranking: 1st (Clear, accurate, well-explained)

Response B: "It's blue because of light stuff"

Ranking: 3rd (Too vague)

Response C: "Blue wavelengths scatter more than red..."

Ranking: 2nd (Accurate but less complete)

Step 2: Train a Reward Model

A separate neural network (the reward model) is trained to predict human preferences. It learns to score responses based on quality, helpfulness, and safety.

Step 3: Optimize with Reinforcement Learning

The LLM is then trained using reinforcement learning (specifically PPO - Proximal Policy Optimization) to maximize the reward model's scores. This allows the model to:

Explore different reasoning strategies
Learn from trial and error
Develop better problem-solving approaches
Balance multiple objectives (helpfulness, safety, accuracy)

                    Why RLHF Matters: RLHF enables models to go beyond simple imitation. They can
                    reason through complex problems, provide nuanced answers, and handle edge cases that weren't in the
                    training data. This is why ChatGPT feels more "intelligent" than earlier models.
                

Key Characteristics of Modern LLMs

1. In-Context Learning

One of the most remarkable properties of LLMs is in-context learning—the ability to learn new tasks from examples provided in the prompt, without any parameter updates.

                    
// Zero-Shot (No examples)

Translate to French: "Hello, how are you?"

// Few-Shot (With examples)

English: "The cat is on the mat" → French: "Le chat est sur le tapis"

English: "I love programming" → French: "J'aime programmer"

English: "Hello, how are you?" → French: "Bonjour, comment allez-vous?"

The few-shot version typically performs much better because the model can infer the pattern and style from the examples.

2. Programming Behavior Through Prompts

You can dramatically change model behavior by adjusting the prompt:

System Prompt 1: Concise Expert

                            
You are a concise expert.

Provide brief, technical answers.

Response style: Short, technical, assumes knowledge

System Prompt 2: Patient Teacher

                            
You are a patient teacher.

Explain concepts clearly with

examples and analogies.

Response style: Detailed, educational, beginner-friendly

3. Emergent Abilities

As models scale up (more parameters, more data, more compute), they develop abilities that weren't explicitly trained:

Chain-of-thought reasoning - Breaking down complex problems step-by-step
Multi-step planning - Coordinating multiple actions toward a goal
Code generation and debugging - Writing functional programs
Creative writing - Poetry, stories, dialogue
Multi-modal understanding - Connecting text with images (in models like GPT-4V)

4. Limitations and Challenges

What LLMs Still Struggle With:

Hallucinations - Confidently generating false information
Math and logic - Arithmetic errors, logical inconsistencies
Temporal awareness - Not knowing current events (knowledge cutoff)
Consistency - Contradicting itself across long conversations
True understanding - Debatable whether they "understand" or just pattern match

Practical Implications and Future Directions

The Democratization of AI

Andrej Karpathy highlighted an amazing trend: the cost of training models has plummeted. Training GPT-2 originally cost ~$40,000. Today, with optimized data pipelines and efficient hardware, you could train a similar model for $100-$700.

What This Means: Smaller companies, researchers, and even individuals can now train capable language models. Open-source models like LLaMA, Mistral, and Phi are closing the gap with proprietary models.

Ongoing Research Areas

Longer context windows - Processing entire books or codebases
Multimodal models - Seamlessly handling text, images, audio, and video
Reasoning improvements - Better logical reasoning and mathematical abilities
Efficiency - Smaller models that perform as well as larger ones
Safety and alignment - Ensuring models behave as intended
Personalization - Models that adapt to individual users

Ethical Considerations

Important Challenges:

Bias - Models inherit biases from training data
Misinformation - Potential to generate convincing fake content
Privacy - Risk of memorizing and leaking training data
Job displacement - Automation of knowledge work
Environmental impact - Massive energy consumption for training

Conclusion: The Journey from Data to Intelligence

Large Language Models represent one of the most significant technological breakthroughs of our time. From raw internet data to intelligent assistants, the journey involves:

Massive data collection and cleaning (URL filtering, text extraction, language filtering)
Tokenization to convert text into numbers
Pretraining on next-token prediction to learn language
Supervised fine-tuning to learn assistant behavior
RLHF to optimize for human preferences

The result is a model that can engage in conversations, write code, explain complex concepts, and assist with countless tasks—all through the simple mechanism of predicting the next token.

                    Key Takeaway: LLMs are not magic. They're the result of careful engineering,
                    massive computation, and clever training techniques. Understanding this pipeline helps us appreciate
                    both their capabilities and limitations, and guides us in using them effectively and responsibly.
                

For a deeper dive into the technical details, I highly recommend watching Andrej Karpathy's full video. His clear explanations and insights from working on GPT-2 at OpenAI provide invaluable context that goes beyond what any blog post can capture.