Why Replicate?

Replication serves three different goals, often at the same time:

1. Availability - if one server crashes, another can take over.
2. Read scalability - many followers can serve read traffic in parallel.
3. Geographic locality - place a replica near each region so user reads are fast.

What replication does not automatically buy you is write scalability. In leader-follower replication, every write still funnels through one leader. To scale writes you need sharding (a separate lesson) or multi-leader replication.

The Three Topologies

1. Leader-Follower (Primary-Replica)

One node is the leader. All writes go to the leader. The leader streams its change log to one or more followers, which apply the changes in the same order. Reads can go to either leader or followers.

---------- Leader-Follower topology ----------
                  writes
   client ---------------> [ leader ]
                              | (replication stream)
                              v
                          [ follower ] [ follower ] [ follower ]
        reads ---------------> any of: leader or followers

This is the default in PostgreSQL, MySQL, MongoDB replica sets, Redis, and most managed cloud databases. It is the simplest topology that gives you failover and read scaling.

Pros: simple, strong consistency on the leader, well-understood failover. Cons: writes do not scale beyond one leader, followers can lag, failover is a non-trivial dance.

2. Multi-Leader (Active-Active)

Two or more nodes accept writes. Each leader replicates its changes to the others. Useful when:

• Multiple data centers must each accept writes locally for low latency.
• You need to keep writing during a partition between regions.
• Offline-capable clients (mobile apps) act as 'leaders' that sync when online.

---------- Multi-leader topology (two regions) ----------
  region us-east             region eu-west
   client                      client
     |                           |
     v                           v
  [ leader ]  <-- replication --> [ leader ]
     |                           |
     v                           v
  [ followers ]               [ followers ]

Engines: Cassandra (every node is a leader within its replication group), CockroachDB, MySQL Group Replication, BDR for PostgreSQL, Aurora Multi-Master.

Pros: writes happen locally in every region, survives whole-region failures. Cons: write conflicts between leaders are unavoidable and need a resolution strategy.

3. Leaderless (Dynamo-style)

No special leader. Every write is sent to several replicas in parallel; the client waits until W of them acknowledge. Reads go to several replicas; the client waits for R to respond and picks the latest version. With N replicas, the inequality R + W > N gives you strong consistency for any single key.

Engines: Cassandra (also leaderless at the partition level), Riak, Amazon DynamoDB internally, Voldemort.

Pros: no special node to fail over, tunable consistency per request, scales horizontally. Cons: every read can hit multiple nodes (latency cost), conflicts are common and need resolution (vector clocks, CRDTs).

Synchronous vs Asynchronous Replication

The second axis: when does the leader confirm the write to the client?

Asynchronous

• Leader writes locally and acknowledges the client immediately.
• Followers receive the change later (milliseconds to seconds).
• Risk: if the leader crashes before any follower receives the write, that write is lost.
• Benefit: fast writes, low latency, the standard mode for most setups.

Synchronous

• Leader waits until at least one follower has durably stored the write before acknowledging.
• Risk: if the follower is slow or down, the leader stalls. One failed follower can cause an outage.
• Benefit: guaranteed durability across machines.

Semi-Synchronous (the practical middle ground)

• One follower is synchronous, the rest are async. If the sync follower lags, it is silently demoted and another follower takes its place.
• Used in MySQL semi-sync, PostgreSQL synchronous_commit = remote_apply with multiple potential sync followers, MongoDB write concern w: majority.

A practical configuration: write-concern majority in MongoDB or synchronous_standby_names with quorum in PostgreSQL gives durable, low-latency writes across an odd number of replicas.

Replication Lag and Consistency Anomalies

The single hardest concept in replication is replication lag: the time between a write landing on the leader and the same write becoming visible on a follower. Lag is usually milliseconds, but under load it can stretch to seconds or even minutes.

Lag creates three classic anomalies. Memorize them - they appear in interviews and bug reports.

Anomaly 1: Stale Read

---------- Stale read after write ----------
T0  client writes 'name=Alice' to leader        OK
T1  client immediately reads from a follower    returns the old name

The write committed, but the follower has not received it yet.

Mitigation - read-your-writes consistency:

• Route reads of a user's own data to the leader for a short window after they write.
• Track the latest replication position (LSN, oplog timestamp) per session and only read from a follower that has caught up to it.
• For account-critical data, simply read from the leader.

Anomaly 2: Monotonic Reads Violated

---------- Monotonic read violation ----------
T0  client reads from follower A    sees version v5
T1  client reads from follower B    sees version v4 (older!)

The user appears to time-travel because consecutive reads went to different followers.

Mitigation: pin a session to the same follower, or pick a follower whose log position is at least as new as the previous read.

Anomaly 3: Causal Inconsistency

A reply appears before the original message because the original arrived to a slower follower second. Mitigation: vector clocks, causal tokens, or sequencing all related operations through the same partition.

Failover: Promoting a Follower

When the leader fails, the system must promote a follower to take its place. This sounds simple and is famously not.

The dance:

1. Detect failure - heartbeat / health check timeout. False positives waste your time; long timeouts increase outage duration.
2. Choose a new leader - typically the follower with the most up-to-date replication log. Built-in tools: Patroni for Postgres, MySQL Orchestrator, MongoDB's automatic election within a replica set.
3. Reconfigure clients - update connection strings or DNS so traffic flows to the new leader.
4. Recover the old leader - when it comes back, it must rejoin as a follower and reconcile any writes that did not make it to the new leader.

Hazards:

• Lost writes - asynchronous replication means recent writes on the failed leader may never have reached the new one. Some writes are silently lost. Customers see 'I'm sure I clicked save'.
• Split brain - the old leader and the new leader both think they are leader, both accept writes. Fencing tokens, leases, and STONITH ('Shoot The Other Node In The Head') are mitigations.
• Cascading overload - the new leader inherits all traffic and may be slower than the old, triggering further failures.

This is why automated failover should be tested in production-like environments quarterly. It is not a tool you set and forget.

Pseudocode for read-your-writes routing

async function readUserProfile(userId, ctx) {
    // If this session wrote in the last 30s, read from leader
    if (ctx.lastWriteAt && Date.now() - ctx.lastWriteAt < 30_000) {
        return await leaderDb.query('SELECT * FROM users WHERE id = $1', [userId]);
    }
    return await followerDb.query('SELECT * FROM users WHERE id = $1', [userId]);
}

async function updateUserProfile(userId, patch, ctx) {
    const result = await leaderDb.query('UPDATE users SET ... WHERE id = $1', [userId]);
    ctx.lastWriteAt = Date.now();
    return result;
}

Conflict Resolution in Multi-Leader Setups

With two leaders, the same row can be updated in two places at the same time. The replication stream brings the conflicting versions together and the system must decide which wins.

The key insight: multi-leader without a conflict strategy will silently corrupt your data. Always choose one before deploying.

How Replication Streams Work Under the Hood

• PostgreSQL: streams the WAL (Write-Ahead Log) byte-for-byte. Followers replay WAL records to rebuild state. Logical replication (Postgres 10+) decodes the WAL into row-level changes for cross-version or cross-cluster replication.
• MySQL: streams the binlog in row, statement, or mixed format. Async by default. Group Replication adds a Paxos-like protocol for multi-leader.
• MongoDB: every write is appended to the oplog capped collection on the primary; secondaries tail it. Elections use the Raft protocol.
• Cassandra: every coordinator forwards each write to N replicas. No leader; consistency is a per-request choice via R, W, and N.
• Amazon Aurora: storage layer is shared across instances; replicas read the same storage volume rather than replaying a log. Replication lag is microseconds, not milliseconds.

Knowing these specifics is what separates a junior answer from a staff-level answer.