← Back to System Internals

Designing a Highly Reliable Service

A system design interview guide to building a service that stays up when its dependencies, machines, and whole datacenters do not — the toolkit an interviewer expects when the prompt is “design a highly reliable service for X.”

Reliability is not a feature you bolt on at the end; it is a set of deliberate choices made at every layer, from how you define “up” to how a single call times out. A highly reliable service assumes that everything fails — processes crash, disks fill, networks partition, dependencies slow to a crawl — and is engineered so that no single failure, and ideally no single class of failure, takes the whole system down. This guide walks the toolkit: first defining reliability precisely with SLIs, SLOs, and an error budget; then the structural defenses of redundancy across failure domains and health-aware load balancing; then the request-path patterns — timeouts, retries, circuit breakers, bulkheads — that keep a slow dependency from becoming an outage; and finally rollout safety and disaster recovery for when a whole region is lost.

1. Requirements & Scope

Start by pinning down what reliability means for this specific service, because the answer drives every later choice. Frame it as functional and non-functional requirements.

• Functional. The service exposes an API — say a request/response endpoint backed by a datastore. It must accept writes, serve reads, and never corrupt or lose acknowledged data.
• Availability. A target like 99.9% (three nines) or 99.99% (four nines). State it explicitly: three nines allows roughly 43 minutes of downtime a month, four nines allows about 4 minutes. The number sets the ambition of the whole design.
• Durability. Acknowledged writes must survive machine and disk loss — usually stated as a number of nines of durability, achieved with replication.
• Latency. A tail-latency target, for example p99 ≤ 100 ms, because a reliable service is one that is fast and up; slow is a kind of down.
• Blast radius & recovery. No single failure domain should take the service down, and after a disaster it must recover within a stated RTO/RPO (covered in section 7).

2. SLIs, SLOs & the Error Budget

You cannot make something reliable that you have not defined and measured. Three linked concepts do this, and getting the vocabulary right is half the battle in an interview.

• SLI (Service Level Indicator). A measured ratio of good events to total events — for example the fraction of requests served successfully under 100 ms. It is a number you actually compute from telemetry.
• SLO (Service Level Objective). The target the SLI must meet over a window — for example “99.9% of requests succeed over a trailing 28 days.” It is an internal goal, stricter than any external promise.
• SLA (Service Level Agreement). A contract with customers, with penalties, set looser than the SLO so you have headroom before you breach it.
• Error budget. The mathematical complement of the SLO: 1 − SLO. A 99.9% SLO grants a 0.1% budget of failed or slow requests — about 43 minutes a month — that you are allowed to spend.

The error budget is the single most useful idea here because it turns reliability from an argument into an arithmetic. When the budget is healthy, the team can ship features and take risks. When an incident burns through it, the policy is automatic: freeze risky changes and spend engineering on stability until the budget recovers. It aligns product velocity and reliability instead of pitting them against each other.

3. Redundancy & Failure Domains

Structural reliability comes from having no single point of failure. Every component that can fail must have a redundant peer, and — crucially — the peers must fail independently. That independence is captured by the idea of a failure domain: a blast radius that can go down together, such as a process, a host, a rack, an availability zone, or an entire region. You get reliability by spreading redundant copies across domains so that losing one domain loses only a fraction of capacity.

The pattern has a few load-bearing choices. Make the service tier stateless so any replica can serve any request and replicas are trivially interchangeable — state lives in the datastore, not in the process. Run N+2 capacity, not N+1, so you survive a failure even while another node is down for maintenance. Spread replicas across availability zones so a single power or network event cannot take them all. And run in more than one region, either active-active (both serving) or active-passive (one hot standby), so that the loss of an entire region degrades rather than destroys the service.

4. Health Checks & Load Balancing

Redundancy only helps if traffic actually avoids the broken copies, and that is the job of health checks feeding a load balancer. A load balancer continuously probes each backend and routes requests only to those reporting healthy, so a crashed or degraded replica is removed from rotation within seconds rather than serving errors.

The subtlety is in what “healthy” means. A shallow (liveness) check confirms the process is running and answering; a deep (readiness) check confirms it can actually do useful work — its dependencies are reachable, its caches are warm, it is not overloaded. Readiness is what should gate load balancing, but it must be designed carefully: if a deep check marks every replica unhealthy the instant a shared dependency blips, the load balancer will drain all traffic and turn a minor dependency wobble into a total outage. The fix is to fail health checks gradually and to distinguish “I am broken” from “a thing I depend on is briefly slow.”

• Balancing algorithm. Round-robin is simple; least-connections or least-load handles uneven request costs better; consistent hashing preserves cache locality and minimizes reshuffling when the backend set changes.
• Outlier detection. Beyond pass/fail health, eject a backend that is returning elevated errors or latency even if it still claims to be healthy.
• Draining. On deploy or scale-down, stop sending new requests to a replica but let in-flight ones finish before it exits, so a rollout is not itself a source of errors.

5. The Request-Path Resilience Toolkit

Structural redundancy protects against a replica dying. A different, equally common threat is a dependency that does not die but turns slow — and a slow dependency, without defenses, will exhaust your threads, queue requests, and cascade into a full outage. The request-path toolkit is the set of patterns that contain that.

Timeouts & deadlines

Never make an unbounded call. Every request to a dependency gets a timeout, and better still a deadline that propagates down the call chain: if the caller has 100 ms left, the callee must not spend 200 ms. Deadline propagation prevents wasted work on requests whose caller has already given up, and it is the single most effective guard against cascading slowness.

Retries with backoff and jitter

Transient failures — a dropped packet, a brief blip — are worth retrying, but naive retries are dangerous. Retrying immediately, or retrying in lockstep across many clients, produces a retry storm that hammers an already-struggling dependency exactly when it is weakest. The fix is exponential backoff (wait longer after each attempt) plus jitter (randomize the wait so clients do not synchronize). Cap the number of attempts, and only retry idempotent operations.

function call_with_retry(op, deadline):
  attempt = 0
  while attempt < MAX_ATTEMPTS and now() < deadline:
    try:
      return op(timeout = remaining(deadline))   # bound each try
    except Transient:
      base  = BASE_DELAY * (2 ** attempt)         # exponential
      sleep(random(0, base))                      # full jitter
      attempt += 1
  raise Unavailable                               # give up, do not storm

Circuit breakers

When a dependency is clearly down, retrying at all is counterproductive. A circuit breaker tracks the recent error rate to a dependency; once it crosses a threshold the breaker opens and calls fail fast without even attempting the network, giving the dependency room to recover and freeing the caller’s threads. After a cooldown it goes half-open, letting a trickle of requests through to test recovery, and closes again if they succeed. This converts a slow, thread-exhausting failure into an instant, cheap one.

Bulkheads

Named after a ship’s watertight compartments, a bulkhead isolates resources so a failure in one area cannot sink the whole vessel. Give each downstream dependency its own connection pool and thread pool, so that if dependency A hangs and saturates its pool, calls to healthy dependency B still have threads to run on. Without bulkheads, one slow dependency consumes the shared thread pool and stalls every request, related or not.

6. Idempotency & Backpressure

Two more properties make the difference between a service that survives load and one that melts under it.

Idempotency. Retries, at-least-once delivery, and client reconnects all mean the same operation can arrive more than once. An operation is idempotent if applying it twice has the same effect as applying it once. Reads and overwrites are naturally idempotent; increments and appends are not. The standard fix is an idempotency key: the client attaches a unique id to each logical operation, the server records which keys it has already applied, and a duplicate returns the original result instead of doing the work again. This is what lets you retry safely without double-charging a customer or double-writing a record.

Backpressure. A reliable service must protect itself from more load than it can handle rather than accepting everything and collapsing. Backpressure is the family of mechanisms that push load back: bounded queues that reject when full, concurrency limits, and load shedding that drops low-priority requests first when the system is saturated. The counterintuitive but essential move is to reject early and cheaply — a fast rejection preserves capacity for the requests you do accept, whereas accepting everything guarantees that all requests get slow and then all fail. Admission control at the front door is more reliable than unbounded queues behind it.

7. Rollout Safety & Disaster Recovery

The leading cause of outages is not hardware — it is change. So a reliable service treats deployment as a first-class reliability concern, and prepares for the rare day it loses an entire region.

Safe rollouts

• Canary. Deploy the new version to a small slice of traffic first, watch its error and latency SLIs against the old version, and only proceed if they hold. A bad change then hurts 1% of traffic, not 100%.
• Progressive / staged rollout. Ramp from 1% → 10% → 50% → 100%, region by region, with automated health gates between stages.
• Automatic rollback. Wire the gates to roll back automatically when SLIs regress, so recovery does not wait on a human noticing.
• Feature flags. Decouple deploy from release: ship code dark, then turn a feature on with a flag you can flip off instantly without a redeploy.

Disaster recovery: RTO & RPO

For catastrophic loss, two numbers define the target. RTO (Recovery Time Objective) is how quickly you must be back up; RPO (Recovery Point Objective) is how much recent data you can afford to lose. They drive the DR strategy directly and trade cost against recovery.

Whichever you pick, the discipline that matters is testing it: run regular failover drills and game days, because a DR plan that has never been exercised is a hypothesis, not a capability.

8. Bottlenecks & Tradeoffs

Every reliability mechanism has a cost, and naming the tradeoffs is what separates a checklist from a design.

• Reliability vs cost. Each extra nine roughly multiplies cost — more replicas, more regions, more on-call. Match the target to the business need; do not buy five nines for a service that needs three.
• Redundancy vs consistency. Multi-region active-active gives the best availability but forces you to confront the CAP tradeoff: under a partition you choose between serving possibly-stale data (AP) or refusing writes to stay consistent (CP). State which you pick and why.
• Retries vs stampedes. Retries improve success rates but, done wrong, cause the very overload they were meant to survive. Backoff, jitter, caps, and circuit breakers are mandatory, not optional.
• Timeouts too tight vs too loose. Too tight and you fail healthy-but-slow requests; too loose and you let slowness cascade. Set them from observed latency percentiles, not guesses.
• Complexity is itself a risk. Every mechanism you add is more code that can fail and more behavior on-call must understand. The most reliable design is the simplest one that meets the SLO.

9. Summary

A highly reliable service is built by assuming failure everywhere and layering defenses: define reliability precisely, remove single points of failure, keep traffic away from broken copies, contain slow dependencies on the request path, and rehearse recovery.