Replication

Synchronous or asynchronous replication — which do you pick?

It is a choice about what you are willing to lose. Synchronous means the primary waits for a replica to confirm before acknowledging the write, so a failover loses nothing — and every write now pays a network round trip, and a slow replica slows the primary. Asynchronous acknowledges immediately and ships the change after, so writes stay fast and a failover can lose the last few seconds. Ask what the data is before answering: payments and session state have very different answers.

The middle ground worth naming: acknowledge when any one of N replicas confirms, which bounds the loss without waiting for the slowest.

You added a second server and were still down for 41 minutes. Why?

Because a standby is not a failover. Active-passive gives you a machine with the data and nothing that promotes it — someone has to notice, decide, promote the replica, repoint the application and drain stale connections, and every one of those minutes is downtime. Active-active takes traffic on both from the start, so the failure mode is capacity loss rather than an outage — at the cost of having to resolve conflicting writes.

The number to give: your recovery time is not the replication lag, it is detection plus decision plus promotion plus DNS or connection draining.

What is replication lag and what breaks because of it?

The delay between a write landing on the primary and appearing on a replica. It breaks read-your-own-writes: a user updates their profile, the next request is served by a lagging replica, and their change is gone. The fixes are routing a user to the primary for a short window after they write, or pinning them to one replica, or carrying the write position in the session and waiting for the replica to catch up.