← Back to System Internals

Designing Global Multi-Datacenter Routing & Failover

A system design interview guide to sending every user to the right region, keeping data safe across regions, and surviving the total loss of a datacenter with little or no data loss and a fast recovery.

A service that runs in one datacenter is one power failure, fiber cut, or regional outage away from being down for everyone. Running in several regions fixes that, but it trades one hard problem for two: you must route each user to a healthy nearby region, and you must replicate state between regions so that losing one does not lose data. These two problems pull against each other through the laws of physics — the speed of light makes cross-region coordination slow, and the CAP and PACELC tradeoffs mean you cannot have strong consistency, low latency, and full availability at once. This guide works through both halves: the routing layer (GeoDNS, anycast, and GSLB), the choice between active-active and active-passive, synchronous versus asynchronous replication and what each does to your recovery objectives, and finally how to fail an entire region over without splitting the brain or losing writes.

1. Requirements & Scope

The requirements split cleanly into the two halves — getting users to a region, and keeping data correct across regions — plus the disaster case that ties them together.

• Routing. Send each user to a healthy region, preferably the nearest, and stop sending traffic to a region that is failing or being drained.
• Latency. Serve reads and writes close to the user; keep the extra cost of being multi-region off the common path.
• Durability. Survive the complete loss of one region with a defined, small data-loss bound — ideally zero.
• Availability. Stay up through a regional failure with a short, defined recovery time.
• Correctness. Never let two regions both believe they are the writable primary — no split brain, no divergent histories.
• Non-goals. This is not about scaling a single region (see the scaling guide); it assumes each region is already healthy on its own.

2. Back-of-the-Envelope

Two sets of numbers drive the design: the physics of cross-region latency, and the objectives that flow from the business.

The single most important number is the cross-region RTT: ~70 ms. Every design choice below is really a decision about whether to pay that latency on the write path (synchronous, zero data loss) or to avoid it and accept a small data-loss window (asynchronous). Physics, not preference, forces the trade.

3. RPO, RTO & Key Definitions

The vocabulary of disaster recovery is precise, and interviewers expect it. Two numbers frame every failover design.

The mental model: RPO is a data question answered by replication, and RTO is a time question answered by automation and routing. Sync replication buys RPO = 0 at a latency cost; fast, well-drilled failover buys a low RTO.

4. High-Level Architecture

The shape is a global routing layer on top of two or more self-sufficient regions, with a replication channel between their data stores.

• Global routing layer. Decides which region a user reaches, using health and proximity. It is the control point for failover — changing where traffic lands is how you evacuate a region.
• Regional stacks. Each region is independently able to serve traffic: its own load balancer, its own stateless app tier (so any instance can serve any request), and its own copy of the data.
• Replication channel. The data stores exchange changes across regions. The mode of this channel — synchronous or asynchronous — sets the RPO and the write latency.
• Coordination. A consensus layer (or a managed equivalent) decides who is primary and prevents two regions from writing at once.

5. Routing: GeoDNS, Anycast, GSLB

There are three common ways to get a user to a region, and mature systems layer them. Each makes a different tradeoff between control, speed of failover, and connection stability.

• GeoDNS. The DNS resolver returns a different IP based on the client’s approximate location, sending users to the nearest region. Simple and widely supported, but failover is only as fast as the record’s TTL plus resolver and OS caching — stale caches can keep sending users to a dead region for minutes after you change the record.
• Anycast. The same IP address is announced from many locations via BGP; the network delivers each packet to the topologically nearest announcement. Failover is fast — withdraw the route and traffic reconverges in seconds — and it needs no DNS change. The classic caveat is that a mid-session BGP reconvergence can move a long-lived TCP connection to a different POP and break it, so anycast suits stateless, short-lived requests (DNS itself, CDN edges, UDP/QUIC) best.
• GSLB (Global Server Load Balancing). A smarter DNS/load-balancing tier that answers with a region chosen from live health checks, capacity, and latency — not just geography. It actively steers away from unhealthy or overloaded regions and is the usual home for weighted failover and traffic-shaping policy.

In practice you combine them: anycast to reach the nearest edge/POP fast, GSLB (health-aware DNS) to choose the healthy origin region, and short DNS TTLs so a policy change propagates quickly. Whatever the mechanism, the routing layer must be driven by real health signals, because routing to a region that is failing is worse than routing nowhere.

6. Replication & Consistency

How data moves between regions is the heart of the design, because it fixes both your data-loss bound and your write latency. The choice is synchronous versus asynchronous, and it is a direct expression of the PACELC tradeoff: even when the network is fine, you choose between latency and consistency.

• Synchronous. The primary waits for a remote replica (or a quorum of them) to durably accept a write before acknowledging the client. Nothing is confirmed that is not already safe in another region, so RPO is zero. The cost is a full cross-region round trip on every write, and reduced availability — if the remote side is unreachable, the write cannot complete.
• Asynchronous. The primary commits locally and acknowledges the client immediately, then streams the change to other regions in the background. Writes are fast and stay available during remote hiccups, but any change not yet shipped when the primary is lost is gone — the RPO equals the replication lag.
• Quorum / semi-sync. A middle ground: wait for a majority (or at least one remote) rather than all, balancing latency against durability. Consensus protocols (Raft, Paxos) formalize this — a write is committed once a quorum has it, which also guarantees a single agreed history.

Active-active adds a wrinkle: if more than one region accepts writes, concurrent writes to the same key can conflict. You resolve that either by partitioning ownership (each key’s writes are homed to one region, so there is no conflict), or by using a store with principled conflict resolution (CRDTs, last-writer-wins with care). The simplest correct answer in an interview is usually a single write region per data partition with async read replicas elsewhere — it sidesteps conflict entirely.

7. Failover & Split-Brain

Failover is the moment the design is really tested: a region is gone, and you must promote another to primary without losing data and without ever letting two regions both write. The dangerous failure is not the outage — it is the split brain, where the old primary is not actually dead (just partitioned) and keeps accepting writes while the new one does too, producing two divergent histories that cannot be merged.

The safeguards that make this correct:

• Quorum-based promotion. Only a candidate that a majority of voters agree on can become primary. Because a majority cannot exist on both sides of a partition, only one region can ever win — this is the structural cure for split brain.
• Fencing. The old primary is actively prevented from writing — its lease is revoked, its storage detached, or a monotonic epoch/term number bumped so its late writes are rejected. Never assume a silent node is a dead node.
• Catch-up check. Before promoting, confirm how far behind the replica is (compare log positions). With sync replication it is already current (RPO 0); with async you accept the lag as your data loss and record it.
• Traffic draining. Shift traffic gradually and stop new writes to the old region first, so in-flight work completes and you do not slam the new primary cold.
• DR drills. Failover that is only exercised during a real outage does not work. Regularly rehearsing region evacuation — game days — is what keeps RTO real rather than aspirational.

function failover(region):
  if not quorum.agrees(region.is_dead):
    abort()                              # avoid false failover
  fence(region.primary)                  # revoke lease / bump epoch
  candidate = most_caught_up_replica()   # least data loss
  loss = primary.log_pos - candidate.log_pos
  record_rpo(loss)                       # async: this is the data-loss window
  promote(candidate)                     # new primary, new epoch
  routing.repoint(candidate.region)      # GSLB / DNS / anycast
  drain_and_verify()                     # check RTO met

8. Bottlenecks & Tradeoffs

The tensions here are fundamental — they come from the network and from CAP/PACELC, not from implementation.

• Latency vs durability. Sync replication gives RPO 0 but taxes every write with a cross-region RTT; async gives fast writes but a nonzero data-loss window. This is the central trade and it cannot be optimized away, only placed.
• Consistency vs availability. Under a partition you must choose: refuse writes to stay consistent (CP), or accept writes on both sides and reconcile later (AP). Pick per data type — money is CP, a like counter can be AP.
• Failover speed vs safety. Aggressive automated failover lowers RTO but risks flapping and false failovers on a transient blip; conservative failover is safe but slow. Quorum detection plus a brief confirmation window balances the two.
• Routing freshness vs caching. DNS caching helps performance but slows failover; anycast reroutes fast but can break long connections. Layering (anycast + short-TTL GSLB) mitigates both.
• Cost. Active-active means paying for capacity in every region and cross-region bandwidth; active-passive is cheaper but wastes standby capacity and has a colder, slower failover. Choose by how much downtime and data loss actually cost the business.

9. Summary

Global multi-region design is two problems bound by the speed of light: route users to a healthy region, and replicate data so losing a region does not lose the data. Every choice is a placement of the same latency-versus-durability trade.