The Problem Google Faced
Imagine you're Google in 2003. You need to store and process petabytes of web crawl data, search indexes, and user data. Your options?
- Expensive enterprise storage? Too costly at this scale. You'd go bankrupt.
- Buy thousands of cheap commodity machines? They fail constantly. How do you build reliable storage from unreliable parts?
Design a file system that expects failure. Make it so normal that you don't even notice when a machine dies. Store everything in huge files (multi-GB), split them into chunks, and replicate everything 3 times across different machines.
GFS was built for Google's specific workload:
- Files are huge (multi-GB is normal)
- Reads are mostly sequential (read entire file from start to end)
- Writes are append-only (add data to end of file, rarely modify middle)
- Machines fail constantly (with 1000s of machines, something is always broken)
GFS Architecture: The Big Picture
GFS has three main components. Let's visualize how they work together:
The Three Components Explained
1. Master (The Brain)
Job: Keep track of metadata (which chunks belong to which file, where each chunk is stored).
Does NOT store: Actual file data. Only knows "file.txt is made of chunks A, B, C" and "chunk A is on servers 1, 3, 7".
Why this matters: Master stays lightweight. All heavy data transfer happens directly between clients and chunkservers.
2. Chunkservers (The Workers)
Job: Store the actual file data as 64MB chunks on local disk.
Key detail: Each chunk is just a normal Linux file stored on the local file system (ext3, ext4). No special storage needed.
Why 64MB chunks? Reduces metadata overhead. A 1GB file = only 16 chunks to track.
3. Client (Your App)
Job: Read and write files. Links your application to GFS.
Process:
- Ask Master: "Where is chunk 5 of file.txt?"
- Master replies: "Servers A, B, C"
- Client contacts closest chunkserver directly
How Reading a File Works
Let's walk through reading a file step by step:
Replication: Handling Failures
Every chunk is stored on 3 different chunkservers (default). Why?
How Replication Works
- Master detects the failure (server stops sending heartbeats)
- Master finds all chunks that server was storing
- For each chunk, if replicas drop below 3, master orders re-replication
- Another chunkserver copies the chunk from a surviving replica
How Writing Works (Appends)
GFS is optimized for append operations. Instead of overwriting data in the middle of a file (complex), most applications just add new data to the end.
For each chunk, one chunkserver is designated the primary. The primary decides the order of writes. This prevents conflicts when multiple clients write simultaneously.
Write Flow
// Client wants to append data to a file
1. Client → Master: "I want to append to file.txt"
2. Master → Client: "Write to chunk #5. Primary = server7, replicas = [server3, server11]"
3. Client → All replicas: Push data to ALL chunkservers (primary + replicas)
// Data is cached but not yet written to disk
4. Client → Primary: "Write the data now!"
5. Primary: Assigns sequence number, writes data to its local disk
6. Primary → Replicas: "Write this data at offset X with sequence Y"
7. Replicas: Ack back to primary after writing
8. Primary → Client: "Write successful!" (or "Failed" if any replica failed)
Consistency Model (Simplified)
GFS doesn't guarantee perfect consistency. Here's what you get:
- Record Append: Data is guaranteed to be written atomically at least once. Might appear multiple times if there's a failure and retry.
- Random Writes: If multiple clients write to the same region, the result is undefined. GFS is not designed for this!
Master: The Critical Component
The master stores all metadata in memory for fast access. But what if the master crashes?
Operation Log
Every metadata change is logged to disk before being applied. This log can replay all operations to rebuild state after a crash.
Checkpoints
Periodically, master writes its entire state to disk in a compact format. Recovery starts from latest checkpoint, then replays log.
Shadow Masters
Read-only replicas of the master. They follow the operation log slightly behind. If primary master dies, a shadow can take over (with some data lag).
Why GFS Mattered
GFS proved you could build reliable storage from unreliable parts at massive scale. Key insights:
- Design for failure: Component failures are the norm, not the exception
- Large files, large chunks: Reduces metadata overhead
- Append-optimized: Matches Google's workload (logs, data processing)
- Separate control and data: Master stays light, data flows directly
- Relax consistency: Simpler design, higher performance
Inspired HDFS (Hadoop), which powers much of the big data ecosystem. Principles still used in modern distributed file systems like Ceph and Colossus (GFS's successor at Google).
Further Reading
Original Paper
The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
SOSP 2003