← Back to System Internals

Zero Data Loss During a Network Partition

A system design interview guide to building a distributed system that must guarantee zero data loss even when a multi-region network partition cuts the cluster in half — the consistency and consensus layer beneath a durable service.

Some data is unforgiving. A payment, a ledger entry, an order confirmation: once you tell a client “done,” that fact must survive anything short of the whole planet going dark — including the moment a fiber cut or a switch failure splits your cluster into two halves that can no longer talk to each other. That moment, the network partition, is where casual designs quietly lose writes or, worse, let two halves diverge into histories that can never be reconciled. This guide is about the layer that prevents both outcomes: how the CAP theorem forces a real choice the instant a partition happens, how consensus and quorums decide what “committed” means, how synchronous replication buys a zero recovery-point objective at a latency cost, and how fencing and epochs stop a split brain. Its companion, Multi-Datacenter Routing & Failover, covers the routing and RPO/RTO operational side — getting users to a healthy region and evacuating a dead one; this page is the consistency and consensus layer underneath it.

1. Requirements & Scope

The problem statement is deceptively short — “zero data loss during a partition” — so the first job is to pin down exactly what must hold, and what is explicitly out of scope.

• Durability (the hard requirement). Any write the system has acknowledged to a client must never be lost, even if a whole region disappears mid-partition. This is a recovery-point objective (RPO) of zero.
• Correctness under partition. The system must never let two sides both believe they are the writable authority and diverge — no split brain, no conflicting committed histories.
• Availability (the negotiable requirement). We would like to keep serving through a partition, but for the critical data we will trade availability for correctness when forced. We state this explicitly rather than pretending we can have both.
• Bounded latency. The zero-loss guarantee has a latency cost on the write path; the design must make that cost visible and place it deliberately, not everywhere.
• Non-goals. This is not about how users reach a region or how fast we fail over — that is the routing & failover guide. It assumes each region is independently healthy and focuses on the replication and consensus layer.

2. What “Zero Data Loss” Means

The phrase sounds absolute, but it hides a precise distinction that separates a senior answer from a hand-wave: the difference between a write that is acknowledged and a write that is committed. Zero data loss is a promise about committed writes only, and the whole design is an effort to make those two words mean the same thing.

• Received. A node has the write in memory but has not made it durable or agreed on it. It can vanish on a crash.
• Committed. A quorum of nodes has durably stored the write and agreed on its position in the log. It will survive the loss of a minority of nodes.
• Acknowledged. The client has been told “done.” The cardinal rule is: never acknowledge before committed. If you ack on receipt and the node dies before the write replicates, you have lost data the client believes is safe — the exact failure “zero data loss” forbids.

So RPO = 0 is achievable only if acknowledgement strictly follows durable, replicated commit. The numbers below frame the cost of holding that line across regions.

The load-bearing number is the ~70 ms cross-region RTT. Zero data loss means paying it on the write path; every design choice below is really a decision about where to pay it, or whether the data is critical enough to pay it at all.

3. CP vs AP: The Partition Choice

The CAP theorem is often mangled in interviews. It does not say “pick two of three” as a design-time slider. It says something narrower and sharper: when a partition happens, and only then, you must choose between consistency and availability — you cannot have both while the network is split. The rest of the time you can have both, which is why the honest framing (PACELC) adds: else, when there is no partition, you still trade latency against consistency.

• CP (consistency + partition tolerance). Only the side that can form a quorum keeps serving writes; the minority side rejects them. Nothing diverges, so no committed write is ever lost or contradicted. The price is that the minority is unavailable for writes until the partition heals. This is the correct choice for money, inventory, and any data where a wrong answer is worse than no answer — and it is the only choice compatible with genuine zero data loss.
• AP (availability + partition tolerance). Both sides keep accepting writes and reconcile afterward. The system stays up everywhere, but concurrent writes to the same key on different sides conflict, and reconciliation can drop one. AP suits data where availability beats strict correctness — a like counter, a shopping-cart merge, presence status.

Strong vs eventual consistency is the same fork seen from the read side. Strong consistency (linearizability) means every read sees the latest committed write, as if there were a single copy; it requires coordination and is what CP delivers. Eventual consistency means replicas converge given enough time and no new writes, which is what AP settles for. A zero-data-loss system for critical data lands firmly on the CP, strongly-consistent side; the art is scoping that expensive guarantee to only the data that truly needs it.

4. Consensus & the Replicated Log

How does a set of nodes agree on a single, durable history of writes — and keep agreeing even as members fail? That is the job of a consensus algorithm. Raft and Paxos are the two canonical answers; Raft is the one to explain in an interview because its structure maps cleanly to the guarantees you need. The unifying idea is a replicated log: every write is an entry appended in the same order on every node, and a write is committed once a quorum has durably stored it.

• Leader election. Raft elects a single leader for a bounded term (a monotonically increasing epoch). All writes flow through the leader, which gives a single point to order the log. A leader must win votes from a majority to be elected — and because a majority cannot exist on both sides of a partition, at most one leader can be elected per term. This is the structural root of split-brain prevention.
• Quorum writes. The leader appends an entry and replicates it; it is committed only when a majority (including the leader) has it durably. With N = 5 the write quorum is 3, so the system survives losing 2 nodes without losing a committed write.
• Quorum reads. The invariant is W + R > N. If a write is acknowledged by W replicas and a read consults R replicas, then W + R > N guarantees the two sets overlap, so a read always sees the latest committed value. Choosing W = R = majority gives strong consistency; reading from the leader (or a lease-holder) is the common optimization.
• Log matching. Consensus also guarantees that if two logs agree on an entry, they agree on everything before it — so committed history never forks. A new leader must contain every committed entry, which is why elections restrict votes to sufficiently up-to-date candidates.

function on_write(entry):
  require self.is_leader                      # only the leader appends
  log.append(entry, term=current_term)        # tentative, not yet committed
  replicate_to_followers(entry)               # parallel fan-out
  wait_until acks >= majority                  # W = majority durable
  commit_index = entry.index                  # now committed = safe
  ack_client(entry)                           # only now: acknowledged

5. Synchronous Replication & Durability

Consensus decides who agrees; durability decides that the agreement actually survives power loss. Zero data loss needs both, and the write path has to be synchronous end to end: a client is told “done” only after the write is durable on a quorum of nodes, not merely present in their memory.

• Write-ahead log (WAL). Before any change is applied, it is appended to an on-disk log. On a crash, replay of the WAL reconstructs state. The WAL is the unit of replication too — shipping log entries is what keeps replicas identical.
• fsync. Appending to the WAL is not durable until it is flushed from the OS page cache to the device. Skipping the fsync is the classic silent way to lose “committed” data on power loss — the write was acknowledged but never truly on disk. Zero data loss requires fsync (or an equivalent durable-write guarantee) before counting a node’s ack.
• Synchronous quorum. The leader waits for a durable ack from a majority before committing. This is what makes the guarantee cross-region: if the whole leader region is lost, a surviving region in the quorum still has the write.
• Semi-sync / tunable. A pragmatic middle ground waits for at least one remote replica (not all), balancing latency against durability. The knob is exactly the quorum size — larger quorum, more durable, more latency.

The cost is explicit and unavoidable: every write pays a cross-region round trip (~70 ms) plus an fsync. That is the price of RPO = 0, and stating it out loud — rather than claiming free durability — is what signals you understand the trade.

6. Fencing, Epochs & Split-Brain

Quorum election prevents two leaders from being elected, but there is a subtler failure: an old leader that has been superseded but does not know it — it was merely partitioned away, is still alive, and keeps trying to serve writes with a stale sense of authority. If those writes are accepted anywhere, you have a split brain after all. The defense is fencing backed by monotonic epochs (Raft terms, ZooKeeper zxids, generation numbers).

• Epoch / term numbers. Every leadership reign carries a strictly increasing number. Followers and storage remember the highest epoch they have seen and reject any request carrying a lower one. When a new leader is elected, the epoch bumps, and the old leader’s late writes are stamped with a now-stale epoch and refused. The old leader is fenced out without anyone having to confirm it is dead.
• Fencing tokens. The same idea applied to a downstream resource (a storage volume, a lock). The lock service hands out a monotonically increasing token with each grant; the resource remembers the highest token it has honored and rejects writes bearing an older one. This is what makes a distributed lock safe even when the holder pauses (a long GC) and wakes up thinking it still owns the lock.
• Never trust silence. A node that stops responding is not necessarily dead — it may be partitioned or paused. Correct designs assume the worst (it might come back and try to write) and rely on fencing, not timeouts, to make its stale writes harmless.

function accept_write(req):
  if req.epoch < self.highest_epoch_seen:
    reject(req)                               # stale leader / stale lock holder
    return
  self.highest_epoch_seen = req.epoch         # monotonic, never decreases
  durably_apply(req)                          # fsync then apply

7. Degraded Mode & Disaster Recovery

A partition is not a clean win/lose; it is a state you operate in, sometimes for minutes. The design must define exactly how each side behaves while split, and how the system heals afterward.

• Reject writes (CP, the default for critical data). The minority side returns a clear, retryable error — “no quorum, try later” — rather than accepting a write it cannot safely commit. Reads may still be served from the last known state if slightly stale reads are acceptable, or refused if they must be linearizable. The majority side keeps serving normally.
• Accept and reconcile (AP, only for tolerant data). Both sides accept writes and merge later using conflict-free replicated data types (CRDTs), last-writer-wins with care, or application-level merge. This keeps everyone available but must be reserved for data where a merged or dropped update is acceptable — never for a ledger.
• Healing. When the partition clears, the minority catches up by replaying the log entries it missed from the majority. Because committed history never forked (log matching), catch-up is a replay, not a merge — that is the payoff of having chosen CP.
• Disaster recovery. Beyond partitions, plan for losing a whole region permanently: keep the quorum spread across enough independent failure domains that a majority survives any single region loss, take periodic backups and ship the WAL off-site for the truly catastrophic case, and — critically — rehearse. A recovery path exercised only during a real outage does not work. The operational mechanics of evacuating and repointing traffic live in the routing & failover guide.

8. Bottlenecks & Tradeoffs

The tensions here are fundamental — they come from the network and from CAP/PACELC, not from any particular implementation.

• Durability vs latency. RPO = 0 requires a synchronous quorum commit, taxing every write with a cross-region RTT and an fsync. Async removes the tax but reopens a data-loss window. This is the central trade; it can be placed, not eliminated.
• Consistency vs availability. Under a partition, CP refuses minority writes to stay correct; AP accepts them to stay up and pays in divergence. Choose per data type — money is CP, a counter can be AP — rather than picking one globally.
• Quorum size vs fault tolerance. A larger N tolerates more failures but makes every quorum slower and needs more independent regions; a small N is fast but fragile. Odd N (3, 5) avoids ties and gives a clean majority.
• Write throughput vs single leader. Routing all writes through one leader gives a clean order but makes the leader a bottleneck. Sharding the keyspace into many independent Raft groups (one leader each) restores horizontal write scale while keeping per-shard consistency.
• Safety vs failover speed. Aggressive election lowers downtime but risks flapping on a transient blip; conservative election is safe but slower. Quorum detection plus a brief confirmation window balances the two — and fencing makes even a wrong guess non-catastrophic.

9. Summary

Guaranteeing zero data loss across a partition is one idea applied relentlessly: never call a write safe until a durable quorum agrees, and never let more than one authority exist. Everything else is placing the cost of that promise.