← Back to System Internals

Zero-Downtime Deployment & Rollback

A system design interview guide to shipping a major change to a critical core platform without a single dropped request — and being able to undo it in seconds when the new version misbehaves.

The prompt is deceptively simple: “When rolling out a major update to a critical core platform, how do you design the deployment and rollback strategy to guarantee zero downtime?” The naive answer — stop the old version, start the new one — guarantees the opposite. Real zero-downtime deployment is a discipline built from a few interlocking ideas: never run two incompatible things at once, always change one variable at a time, expose the new code to a sliver of traffic before the whole fleet, watch objective health signals decide whether to proceed, and keep the previous good state one traffic-flip away at all times. This guide walks that toolkit — compatibility as the foundation, the expand/contract pattern for schema and API changes, progressive delivery with canaries and blue-green, feature flags that decouple deploy from release, staged rings with an explicit bug bar, automated guardrails, and the rollback strategies that make all of it safe. It is the sibling of Designing a Highly Reliable Service, viewed through the lens of the single riskiest thing you do to a running system: change it.

1. Requirements & Scope

Start by stating what “zero downtime” actually means for this platform, because it shapes every later choice. Frame the requirements crisply and confirm them with the interviewer.

• Zero downtime. No window where the service is unavailable. In practice that means requests in flight during a deploy complete successfully, and new requests always land on a healthy instance — the availability SLO holds through the rollout, not just around it.
• Fast, safe rollback. When the new version regresses, recovery must be measured in seconds to a few minutes, and must itself be low-risk — a rollback that requires a fresh build or a risky reverse-migration is not a rollback you can trust at 3 a.m.
• No data loss or corruption. Acknowledged writes survive the transition. Migrations never leave data in a shape that either the old or the new code cannot read.
• No broken clients. Callers — other services, mobile apps, SDKs you cannot force to upgrade — keep working across the change. For a core platform this is often the hardest constraint, because you do not control the clients and cannot deploy them atomically with the server.
• Observable and reversible. Every stage emits the signals needed to decide go/no-go, and every stage can be undone.

2. Compatibility Is the Foundation

During any rollout there is a window where old and new code run side by side and talk to the same data and the same clients. Zero downtime is only possible if both versions can operate correctly during that overlap. That gives one non-negotiable rule: every change must be backward and forward compatible with the versions it will coexist with.

• Backward compatible. New code can read data written by, and serve requests from, the old code. This is what lets you roll the new version out gradually.
• Forward compatible. Old code can tolerate data written by, and requests shaped by, the new code — ignoring fields it does not understand rather than crashing. This is what lets you roll back to the old code after the new one has already written new-shaped data.

The cardinal sin, and the thing interviewers probe for, is combining a breaking change with a deploy in a single step. If you rename a column and ship the code that depends on the new name at the same moment, there is an instant where half the fleet expects the old name and half expects the new one, and neither the forward nor the backward direction holds. You have manufactured a window with no safe state and no clean rollback. The rule is absolute: never make a breaking change and a deploy in the same step. Split every breaking change into a sequence of individually compatible steps — which is exactly what the expand/contract pattern does. This is the deployment-time face of the same discipline covered in Reviewing API, Schema & Query Compatibility.

3. The Expand / Contract Pattern

The expand/contract pattern — also called parallel change — is how you make a change that looks breaking without ever having a breaking moment. You never mutate a thing in place; instead you add the new thing alongside the old, migrate everything onto it, and only then remove the old. It applies identically to a database column, an API field, a message schema, or an RPC method.

The three macro-phases, using the example of renaming a column full_name to display_name:

1. Expand. Add the new display_name column (nullable, no default backfill yet). The schema now carries both. Old code ignores the new column entirely; nothing breaks. This migration is additive, so it is safe to run against a live system and safe to roll back.
2. Migrate. Deploy code that reads from display_name but falls back to full_name, then deploy code that writes both (dual-write). Backfill the existing rows so display_name is populated everywhere, and verify parity between the two columns. Finally, cut reads fully over to display_name. At every sub-step both versions of the code work against the data as it exists.
3. Contract. Once you are confident nothing reads or writes full_name — and you have baked long enough that a rollback would not need it — drop the old column. This is the only destructive step, and it happens last, deliberately, after the risk has been retired.

# Phase 1 — EXPAND: additive migration, both columns exist
ALTER TABLE users ADD COLUMN display_name TEXT NULL;   # old code ignores it

# Phase 2 — MIGRATE: dual-write + read-with-fallback in app code
function write_user(u):
  row.full_name    = u.name        # keep writing old (rollback safety)
  row.display_name = u.name        # also write new
  save(row)

function read_user(row):
  return row.display_name          # prefer new ...
      if row.display_name != null
      else row.full_name           # ... fall back to old

# backfill historical rows, then verify:  COUNT(display_name IS NULL) == 0
UPDATE users SET display_name = full_name WHERE display_name IS NULL;

# Phase 3 — CONTRACT: only after new code is fully baked & won't roll back
ALTER TABLE users DROP COLUMN full_name;               # irreversible — do it last

Two properties make this safe. First, each step is independently deployable — you deploy readers before writers so that no instance ever writes a shape another instance cannot read. Second, each step is reversible until the contract: if the dual-writing version regresses, you roll back to the read-with-fallback version, and the data it wrote is still readable because you never dropped the old column. The pattern converts one dangerous atomic change into a series of boring, safe ones.

4. Progressive Delivery

Compatibility makes coexistence possible; progressive delivery is how you actually move traffic from old to new while limiting blast radius. There are three canonical strategies, and a strong answer names all three and says when each fits.

• Rolling update. Replace instances a few at a time — drain, swap, health-check, repeat — until the whole fleet runs the new version. It needs no extra capacity, but old and new run simultaneously for the whole rollout (so compatibility is mandatory), and rollback means rolling the old version forward again, which is slow.
• Blue-green. Stand up a complete second environment (green) running the new version while the current one (blue) serves all traffic. Flip the router to green in one atomic switch once green is verified. Rollback is instant — flip back to blue. The cost is running two full environments at once, and any shared state (a database) must still be compatible across both.
• Canary. Route a small percentage of real traffic to the new version, compare its health against the baseline, and ramp up only if it holds. This is the gold standard for a critical platform because it exposes the change to production reality at minimal blast radius.

The engine that makes canaries safe is the health/metric gate: at each ramp step, an automated analysis compares the canary’s error rate, latency percentiles, and key business metrics against the stable baseline running the same traffic. If the canary is statistically no worse, it is promoted to the next percentage; if it regresses, the rollout aborts and traffic shifts back. A typical ramp is 1% → 10% → 50% → 100%, with a bake time at each step long enough to catch slow-burning problems (memory leaks, cache pollution) that a thirty-second check would miss.

5. Feature Flags & Dark Launches

The single most powerful idea in modern rollout is decoupling deploy from release. A deploy puts new code on the servers; a release turns a new behavior on for users. A feature flag is the switch between them: you ship the new code paths dark (present but off), deploy them normally, and then enable the behavior with a config change you can flip in seconds — no rebuild, no redeploy.

This buys several things at once. Rollback of a feature becomes flipping a flag off, which is faster and far lower-risk than rolling back a binary. You can enable a feature for internal users first, then a cohort, independent of the deploy cadence. And you separate the risk of the deploy (does the new binary run?) from the risk of the release (does the new behavior work?), so you never debug both at once.

• Dark launch. Ship and run the new code path in production with its user-visible effect suppressed, to prove it works at real scale before anyone sees it.
• Shadow traffic. Mirror a copy of live requests to the new version and compare its responses against the old one — without the shadow’s responses ever reaching users. This validates correctness and load behavior of a risky rewrite against production traffic with zero user-facing risk. Be careful that shadowed requests do not cause side effects (double-writes, duplicate charges); shadow reads, not writes, unless the write path is fully idempotent and sandboxed.
• Kill switch. A coarse flag that instantly disables a feature or subsystem under duress. Every risky launch should ship with one.

6. Rings, Dogfooding & the Bug Bar

Canary percentages control how much traffic sees the change; deployment rings control who sees it, in order of increasing blast radius and decreasing tolerance for bugs. The change graduates outward through the rings, and each boundary is a gate.

1. Ring 0 — internal / dogfood. The team and then the whole company use the new version first (“eating your own dog food”). Employees hit real edge cases and report them, and the cost of a bug here is an annoyed colleague, not a customer.
2. Ring 1 — small external cohort. A small percentage of real users, ideally ones who have opted into early releases or a region you can watch closely.
3. Ring 2 — general availability. The full population, reached only after the earlier rings held.

What makes rings a discipline rather than a hope is an explicit bug bar: a pre-agreed set of severity thresholds that block promotion to the next ring. For example — any Sev1/Sev2, any regression in a core SLI beyond a set margin, or an error-rate delta above X% halts the rollout automatically; a cosmetic Sev3 does not. Defining the bar before the rollout removes the in-the-moment temptation to wave through a problem because a deadline looms. The bug bar turns “is this good enough to proceed?” from a judgment call under pressure into a rule checked against objective signals.

7. Rollback Strategy

A deployment strategy is only as good as its undo button. There are two fundamentally different ways to recover, and the whole reason the earlier compatibility work matters is that it keeps the fast one available.

• Instant rollback via traffic shift. With blue-green or canary, the previous good version is still running (or one flip away). Rollback is a routing change — shift traffic back to it — and completes in seconds with no build and no data migration. This is the target state for a critical platform, and it is available only because you kept the change backward/forward compatible.
• Roll-forward. Sometimes you cannot go back — most often because an irreversible database migration has already run. Then recovery means fixing the problem and deploying forward: ship a hotfix, or flip a kill switch to disable the offending path while you do. Roll-forward is slower and higher-risk, which is exactly why you design to avoid needing it.

The key insight is that irreversible DB migrations are what make rollback hard, and expand/contract is the antidote. Because the destructive contract step happens last and separately, a rollback during the risky middle phases never needs the dropped data — the old shape is still there. A few more data-safety rules make rollback trustworthy:

• Backfill safely. Backfill in idempotent, resumable batches that throttle to avoid overloading the live datastore, and verify parity before cutting reads over. A half-finished backfill must be safe to re-run from the start.
• Idempotency everywhere. Dual-writes, retries during a rollout, and re-run backfills all mean an operation can execute more than once. Idempotency keys make a repeated write a no-op instead of a corruption, so rolling back and forward is safe. (See the idempotency discussion in Designing a Highly Reliable Service.)
• Decouple schema rollback from code rollback. You almost never roll a schema back; you roll code back onto a schema that is still compatible. Keeping the schema forward-compatible is what makes that possible.

8. Automated Guardrails

For a critical platform, humans watching a dashboard are too slow and too unreliable to be the safety mechanism. The rollout should defend itself with automated gates wired to objective signals.

• SLO / error-budget gates. Tie promotion to the same error budget that governs the service (see Designing a Highly Reliable Service). If the canary would burn the budget at an elevated rate, the gate refuses to promote — reliability and rollout share one currency.
• Automated canary analysis. A statistical comparison of the canary against a baseline fleet on the same traffic, over the metrics that matter (errors, latency percentiles, saturation, key business KPIs), producing a pass/fail rather than a human eyeballing graphs.
• Auto-rollback on regression. When a gate fails, the system shifts traffic back automatically — recovery does not wait on a pager being acknowledged. This is the difference between a two-minute blip and a two-hour outage.
• Kill switch. An always-available manual override to disable a feature or halt a rollout instantly, independent of the automated gates, for the cases the metrics do not catch.

9. Stateful vs Stateless & Tradeoffs

Everything above is easiest when the tier being deployed is stateless — any instance can serve any request, so you can freely add, drain, and remove instances and shift traffic at will. Stateful components need extra care, and calling that out separates a real answer from a checklist.

• Draining connections. Before an instance exits, stop routing new requests to it (mark it unready) but let in-flight requests finish within a grace period, then shut down. Without draining, every deploy severs live connections and manufactures the very errors you were trying to avoid.
• Stateful rollouts. For datastores and stateful services, deploy one replica at a time, respecting quorum so the cluster never loses availability, and lead with the schema-compatibility work above. Leader elections and rebalancing must be accounted for so a rolling restart does not trigger a storm.
• Version skew & sessions. During any rollout both versions serve simultaneously, so a client can hit v1 then v2 on consecutive requests. Sessions must not be pinned to a version’s in-memory state (keep session state external), and any wire protocol or cache format shared across versions must itself be forward/backward compatible — version-skew handling is compatibility applied to the in-flight request, not just the data at rest.

The tradeoffs to name:

• Speed vs safety. More ramp steps, longer bake times, and more rings catch more bugs but slow every release. Match the caution to the blast radius — a core platform earns slow, gated rollouts; a low-risk internal tool does not.
• Cost vs rollback speed. Blue-green buys instant rollback at 2× the capacity; canary buys most of the safety for a fraction of the cost but rolls back a hair slower. State the tradeoff and pick per criticality.
• Compatibility overhead. Expand/contract turns one change into several deploys and a window of dual-write complexity. That is real engineering cost — and it is the price of never having an unsafe moment. For a critical platform it is almost always worth paying.
• Flag debt. Feature flags proliferate; stale flags become a maintenance and correctness hazard. Treat flag cleanup as part of the release, not an afterthought.

10. Summary

Zero-downtime deployment is compatibility plus progressive delivery plus a fast, safe undo. You never run incompatible versions together, you change one variable at a time, you expose the change to production in widening slices gated on objective health, and you keep the previous good state one traffic-flip away.