Read Replicas Explained: How to Scale Database Reads…

Prefer to watch? ▶ The 3-minute version ✈ Telegram

Here is a system design question that comes up constantly, and that almost everyone answers half-correctly:

Your database has 300 million rows. It's serving reads and writes, 24 hours a day. It's falling over. How do you scale it?

The reflex answer is "add read replicas." That's not wrong — it's the right answer to the read half of the question. But the half people skip is the half where the bugs live, and it's the half that separates someone who has read about replicas from someone who has run them.

Let's build it from the floor up.

Start below the concept: one box

DevTalk is a developer forum. Threads, comments, upvotes. It runs on one database box.

Every page view hits that machine. Every new comment hits that machine. One CPU, one disk, one queue for all of it. That works fine — until it doesn't.

Before reaching for a solution, measure what the traffic actually is. On a forum, it looks like this:

Thousands of people reading threads.
A handful of people posting.

Roughly 90% reads, 10% writes. That lopsided ratio isn't a detail. It's the entire opening — it's the reason a fix exists at all.

Why one box fails, specifically

The failure mode is more interesting than "it gets slow." As read traffic grows, the flood of reads crowds out the writes. Your database has finite CPU, finite disk IO, finite connection slots, and reads are consuming nearly all of them.

So the user who is contributing — writing a comment, the thing your product actually exists for — is queued behind 10,000 people who are just browsing. The cheapest, least valuable requests are starving the most valuable one.

The mechanism: copy the box

The fix is exactly as literal as it sounds. Copy the box.

The original machine becomes the primary. The copies are read replicas — full, live copies of the same data, on separate machines.

And then one rule does all the work:

Every WRITE goes to the primary, and only the primary.
Every READ fans out across the replicas.

One writer, many readers. That's the whole idea. Everything else is plumbing.

How the copies stay current

The primary already writes every change it makes to a log — Postgres calls it the WAL (write-ahead log), MySQL calls it the binlog. It exists for crash recovery, and it's a complete, ordered record of every change.

Replication reuses it. The primary streams that log to each replica, continuously. Each replica replays it, applying the same changes in the same order. The replica isn't running your queries again — it's replaying the outcome.

Add replicas, add read capacity, roughly linearly. Three replicas, three times the read throughput.

Routing, in practice

Most stacks give you two connection strings and let you pick per query:

# Two engines, two roles.
primary = create_engine("postgresql://primary.db.internal/devtalk")
replica = create_engine("postgresql://replica.db.internal/devtalk")  # usually behind a LB

def get_thread(thread_id: int):
    # A read: safe to be a few ms stale.
    with replica.connect() as conn:
        return conn.execute(SELECT_THREAD, {"id": thread_id}).fetchone()

def post_comment(thread_id: int, user_id: int, body: str):
    # A write: primary, always. There is no "sometimes" here.
    with primary.begin() as conn:
        conn.execute(INSERT_COMMENT, {"thread": thread_id, "user": user_id, "body": body})

The failure mode to watch for is a write that sneaks onto a replica connection — replicas are read-only, so it doesn't corrupt anything, it just throws. That's a good thing: the database enforces the rule you're trying to enforce in code.

The upper bound nobody mentions

Here's where "just add more replicas" stops being a strategy.

Every replica replays every write. All of them. A replica isn't doing less work than the primary on the write path — it's doing exactly the same write work, just without originating it.

Which means:

Replicas buy you zero write capacity. If your bottleneck is writes, replicas do nothing. Worse than nothing — you now have more machines applying the same write load.
Each replica costs money, and adds one more copy that can fall behind.

Replicas are a knob for one specific problem. Know which problem you have before you turn it.

Dimension	Primary	Read replica
Accepts writes	Yes — the only one that does	No, read-only
Serves reads	Yes, but you want to spare it	Yes — this is its whole job
Data freshness	Always current, by definition	Milliseconds behind, sometimes much worse
Write work performed	Originates every change	Replays every change anyway
What adding one buys you	n/a — there is exactly one	More read throughput, ~linearly
What adding one does NOT buy you	n/a	Any write capacity at all
Scaling limit	Vertical only — a bigger box	Cost, and lag pressure per replica

Replication lag: the reason this is a real topic

Now the part that turns this from a diagram into an incident.

That streaming is asynchronous. The primary commits your write, tells the client "done", and moves on. It does not wait for the replicas to catch up. That's deliberate — if it waited, every write would be as slow as your slowest replica, and you'd have traded a read problem for a much worse write problem.

The consequence: a replica is always a little behind. Usually single-digit milliseconds. During a bulk import, a long-running vacuum, or a network hiccup, it can be seconds or minutes.

The bug you will actually hit

Here is the sequence, and it is so ordinary that it ships to production constantly:

A user types a comment and hits Post.
The write lands on the primary. It commits. The API returns 200.
The page reloads and fetches the thread.
That read is a read, so it goes to a replica — one that happens to be 40ms behind.
The comment isn't there.

From the user's side, the app just ate their comment. So they post it again. Now you have duplicate comments, and a support ticket that says "the site is broken" with no error in any log, because nothing errored. Every component did exactly what it was told.

This is a consistency problem wearing a UI bug's clothing, and it is the single most common way read replicas hurt teams that adopted them successfully.

The fix: read-your-own-writes

The fix has a name, and knowing the name is most of the battle: read-your-own-writes consistency.

The rule: for a few seconds after a user writes something, route that one user's reads to the primary. Everyone else keeps hitting replicas.

RYOW_WINDOW = timedelta(seconds=5)

def engine_for_read(user_id: int):
    # After you write, you read from the primary — briefly, and only you.
    last_write = cache.get(f"lastwrite:{user_id}")
    if last_write and (now() - last_write) < RYOW_WINDOW:
        return primary
    return replica

def post_comment(thread_id: int, user_id: int, body: str):
    with primary.begin() as conn:
        conn.execute(INSERT_COMMENT, {"thread": thread_id, "user": user_id, "body": body})
    cache.set(f"lastwrite:{user_id}", now(), ttl=RYOW_WINDOW)

Note how narrow this is. It's not "turn off replicas." It's a per-user, time-boxed exception that costs the primary a trickle of extra reads — the reads of people who just wrote, which by the 90/10 ratio is a tiny slice of traffic.

There are stronger variants when you need them: pin reads to a replica that has confirmed it replayed at least the log position of your write (Postgres exposes this via LSN comparison), or run one synchronous replica for the reads that truly cannot be stale. Both cost latency. Start with the time-boxed version.

Watch the lag, or the lag will find you

Replication lag is a first-class metric, not a debugging afterthought. Alert on it:

-- Postgres, on a replica: how far behind is this copy, in seconds?
SELECT now() - pg_last_xact_replay_timestamp() AS replication_delay;

And have a rule for what happens when it spikes: a replica lagging 30 seconds should be pulled out of the read pool, not left quietly serving stale pages.

Replicas are not sharding

Keep this distinction clean, because interviews probe it and architectures die on it.

Replication copies all the data to more machines. It scales reads. Every machine has everything.
Sharding splits the data across machines by some key. It scales writes (and dataset size). Each machine has a slice.

If your writes are the bottleneck, replicas will not help you, no matter how many you add. That's sharding's problem, and sharding is a substantially harder tool — cross-shard queries, rebalancing, and a shard key you can never comfortably change.

They compose, too: a sharded system usually has replicas per shard. But reach for them for different reasons.

The AI reframe: replicas under a RAG stack

This isn't only a 2010s web-forum concern. It reappears, with the same shape and the same bug, the moment you put a retrieval layer in front of an LLM.

A typical RAG service has exactly the DevTalk asymmetry, only more extreme:

Reads: every user question triggers a vector similarity search, often several (query expansion, multi-hop retrieval, re-ranking a wide candidate set). One question can be five reads.
Writes: ingestion — documents chunked, embedded, upserted. Bursty, and comparatively rare.

If your embeddings live in Postgres with pgvector, you have a database serving 95%+ reads, and read replicas are the obvious lever: fan the similarity searches across replicas, keep ingestion on the primary. Vector search is CPU-hungry per query, so this scales unusually well.

And then you hit the same bug, in a form that's harder to diagnose:

A user uploads a document. The ingestion job embeds it and writes it to the primary. The UI says "indexed". The user immediately asks a question about that document — the retrieval read hits a replica that hasn't replayed the upsert yet — and the model answers "I don't have information about that."

The user doesn't see a stale row. They see a model that is confidently wrong, which is far more corrosive to trust than a missing comment. The fix is identical: read-your-own-writes on the retrieval path, scoped to the user (or tenant) who just ingested.

A second wrinkle worth knowing: an ANN index (HNSW, IVFFlat) has to be built, and replaying an index build on every replica is real, sustained work. Bulk ingestion is exactly the workload that pushes replication lag from milliseconds into minutes — which is to say, the moment you most want the new documents visible is the moment your replicas are furthest behind. Ingest in throttled batches, and watch the lag metric during them.

The same asymmetry logic also drives the serving tier around the model — read-heavy traffic gets fanned out and cached, the small write path stays authoritative — but the database layer is where the correctness bug actually bites.

The verdict

Read replicas scale reads by giving up "now."

That's the trade in one line. Your data on a replica is not wrong — it is correct, just slightly late. And the engineering judgment isn't whether to accept that; it's deciding, read by read, which ones can tolerate lateness and which absolutely cannot.

A working default:

Fan out to replicas: thread lists, search results, feeds, dashboards, analytics, recommendation lookups, anything a cache would already be serving stale.
Keep on the primary: the read immediately after that user's own write, balance and inventory checks, anything that gates a decision (auth, permissions, "can this user do X"), and any read whose result you're about to write back.

Add replicas when reads are your bottleneck. Reach for sharding when writes are. And whichever you add, ship the lag metric with it — because the failure mode isn't an error page, it's a user who quietly stops trusting your app.

References and further reading

The read/write asymmetry and the one-box floor

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — chapter 5, "Replication", is the single best treatment of everything in this article; it opens with exactly this argument for why a read-heavy workload has a replication-shaped answer.
Silvia Botros & Jeremy Tinley, High Performance MySQL, 4th ed. (O'Reilly, 2021) — the operational view: what actually saturates first on one box, and how replication topologies are run in practice.

The mechanism — log streaming and replay

PostgreSQL documentation, High Availability, Load Balancing, and Replication — streaming replication, hot standby, and the knobs (synchronous_commit, hot_standby_feedback) referenced above.
MySQL documentation, Replication — the binlog-based equivalent, including row- vs statement-based replication.
Alex Petrov, Database Internals (O'Reilly, 2019) — part II on distributed systems; useful for understanding why the log is the replication unit rather than the query.

Replication lag and read-your-own-writes

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — "Problems with Replication Lag" in chapter 5 names read-your-own-writes, monotonic reads, and consistent prefix reads. The comment-disappears bug is his "reading your own writes" example.
PostgreSQL documentation, Monitoring — replication views — pg_stat_replication and the LSN positions you need to alert on lag or to pin a read to a caught-up replica.
Amazon, Working with Amazon RDS read replicas — the managed-service version, and a clear statement of the asynchronous guarantee you are actually buying.

Replicas vs. sharding

Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — chapter 6, "Partitioning", is the other tool; reading 5 and 6 back to back makes the distinction permanent.

The retrieval/AI angle

pgvector, project documentation — index types (HNSW, IVFFlat), build cost, and the read characteristics that make vector search a good replica candidate.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: the 3-minute version builds DevTalk from one box, adds the replicas on screen, and walks through the disappearing-comment bug and its fix.