Question 16 · Track 5 · Protect it and charge for it
Design a distributed cache
Adding nodes is easy. Not moving every key when you do is the question.
Protect it and charge for it. 4 clarifying questions to ask first, the answer in 4 moves, the follow-up that catches a memorised answer and 3 supporting topics to watch or read.
What it is really testing
Whether you can place keys without a full reshuffle, and whether you know what a hot key does to a cluster.
Ask these first
Before a single box goes on the board. The answers change the design, which is the point of asking out loud.
- What is the working-set size, and what hit rate do we need?
- Is stale data acceptable, and for how long?
- Read-through, or does the application write the cache?
- What happens on a cold start — can the database survive it?
The answer, in 4 moves
In this order. Each move earns the next one — say them out loud rather than drawing all four and narrating afterwards.
- 1Place keys with consistent hashing, so adding a node moves 1/N keys, not all of them.
- 2Replicate hot keys, and let clients read from any replica.
- 3Choose eviction for the access pattern — LRU is a default, not an answer.
- 4Decide where the routing lives: a smart client, or a proxy in front.
The trap
The follow-up that separates a rehearsed answer from a real one.
One hot key saturates one node while the cluster dashboard looks idle. Consistent hashing spreads keys, not traffic — and traffic is what melts.
Watch or read
The pieces of this answer, each covered on its own. Take them whichever way suits you, then give the whole answer without looking.
Hot keys — one key, one node
Your cache dashboard says 10% average and everything is fine. One machine is at 96% and checkout is timing out. That is a hot key — one popular key, one node, and no amount of hardware. Here is how hash(key) picks the machine, why adding nodes takes the hot one from 96.00% to 95.50%, why consistent hashing with virtual nodes does nothing for it, and the three fixes that actually work — key splitting, an in-process cache, and request coalescing — with their real costs, every number executed.
Cache vs CDN
Cache and CDN both store copies for faster reads, but solve different problems: cache cuts backend work for dynamic data; CDN cuts network distance for static content. Learn when to use each.
Write-through vs write-back
Every caching tutorial teaches the read path. The bugs live on the write. Write-through, write-back and write-around with the real code for each, the measured cost of a durable write, and the data-loss window reproduced step by step.
Part of System Design Interviews, Answered Out Loud. The fundamentals underneath it are the free CS course.