Leading a Cross-functional Migration
Moving dozens of teams onto a new platform or architecture — phased, backward-compatible, reversible, and tracked to zero.
This question tests whether you can execute a large, risky migration that touches many teams without a big-bang outage and without losing the plot halfway through. Cloud moves, storage-engine swaps, and architecture migrations are where a lot of senior candidates reveal they’ve only ever done greenfield. The interviewer wants to see a phased plan, backward compatibility so there’s no forced flag day, shadow or dual-write to verify before cutover, incremental ramp with rollback at every step, and the coordination discipline to drive dozens of stakeholders all the way to zero on the old system. The answer follows the CARL shape.
What this question is really testing
Can you de-risk a large migration into safe, reversible increments, coordinate many teams over a long horizon, and actually finish — including the brutal last 10% and the decommission — rather than leaving two systems running forever?
How to answer- Phase the plan. Break the migration into milestones with owners and exit criteria — foundation, dual-run, ramp, decommission — so it’s a sequence of small verifiable steps, not one terrifying cutover.
- Preserve backward compatibility. Build the new path behind the same interface so consumers don’t have to change on your schedule. No forced flag day — teams migrate when ready, within a deadline.
- Shadow, then dual-write, to verify. Run the new system in shadow against production traffic and compare outputs, or dual-write to both stores and diff them, so you prove correctness on real load before anything depends on the new path.
- Cut over incrementally with rollback. Ramp traffic 1% → 10% → 50% → 100% behind a flag, with a fast rollback at every step and clear health metrics gating each increase. Reversibility is what makes the migration safe to run during business hours.
- Coordinate stakeholders and track to zero. One tracker, one status cadence, per-team migration state visible to all. And finish it — drive the long tail off the old system and decommission it, because a migration that stalls at 90% leaves you paying for two systems and owning double the on-call.
- Risk decomposed into phases with exit criteria, not a heroic cutover.
- Backward compatibility and dual-run so correctness is proven before dependence.
- Incremental ramp with rollback and health gates — reversibility as a first principle.
- Coordination machinery for many teams over a long horizon.
- The discipline to finish — the last 10% and the decommission, tracked to zero.
A worked example (CARL)
Context. Our ads events storage sat on an aging storage engine that was hitting scaling and cost limits, and we needed to migrate to a new tiered backend. The catch: more than 20 downstream teams read from this store, several with strict freshness SLAs, and a billing pipeline where any data loss or duplication was unacceptable. A big-bang cutover was off the table — a bad migration here would be a company-level incident.
Actions. I designed it as a phased, reversible program rather than a project with a launch date. Phase one was foundation and compatibility: my team put the new backend behind the exact same read and write interface the 20 teams already used, so no consumer had to change code to benefit — that removed the “forced flag day” that kills these efforts. Phase two was verification without dependence: we dual-wrote every event to both the old and new stores and ran a continuous diff job comparing them, so we caught correctness bugs against real production traffic while the old store was still the source of truth and nothing was at risk. We ran dual-write for weeks until the diff rate was effectively zero, including through peak load and a few gnarly edge cases the diff surfaced that we’d never have found in a test environment. Phase three was incremental cutover of reads: I moved read traffic behind a flag and ramped 1% → 10% → 50% → 100%, with each step gated on freshness and error-rate health metrics and a one-command rollback if anything regressed — and we did roll back once at 10% when a latency spike appeared, fixed it, and re-ramped. For the billing pipeline specifically, I kept it on the old store until every other consumer was cut over and stable, because its risk tolerance was lowest. Throughout, the coordination was as much of the work as the engineering: I ran a single migration tracker with per-team state, a biweekly stakeholder sync, and a shared dashboard so all 20+ teams could see the ramp and their own migration status. The last 10% was the hardest — a few low-priority teams that kept deprioritizing — so I concierge-migrated them myself and set a dated decommission of the old store to convert “someday” into “this sprint.”
Results. We migrated all 20+ teams with zero data-loss incidents and no consumer-facing SEV, decommissioned the old storage engine within the target half, and cut storage cost for the tier by roughly 35% once the old system was fully retired. Because we never had a forced flag day and could roll back at every step, the whole migration ran during normal business hours.
Learnings. The two moves that made it safe were backward compatibility (no consumer had to move on my clock) and dual-write-plus-diff (correctness proven on real traffic before anyone depended on it). And the migration wasn’t “done” at 90% — the discipline to drive the long tail to zero and actually decommission the old system is what turned it from a science project into realized cost savings.
Common follow-ups
How do you handle the last teams that keep deprioritizing the migration?
How to answer- Concierge the migration. Do the work for them so it’s a review, not a project on their backlog.
- Set a hard decommission date. A dated shutoff of the old system converts “later” into a real deadline.
- Make the cost of the old path visible. Show leadership the double on-call and dollar cost of keeping it alive.
- Escalate the true holdouts. If a team won’t move despite an easy path, that’s a priority call for shared leadership.
How do you verify data correctness during the migration?
How to answer- Dual-write and diff. Write to both systems and continuously compare outputs on real traffic before cutover.
- Shadow reads. Serve from the old path but replay against the new one and compare, so users are never at risk during validation.
- Gate ramp on the diff rate. Don’t increase traffic until the discrepancy rate is at or near zero, including at peak.
- Reconcile the edge cases. Investigate every diff — the last few percent are usually the real bugs.
Mid-migration you find the new system has a flaw. Do you roll back or push through?
How to answer- Roll back if users are at risk. Reversibility exists precisely for this — use it without ego.
- Assess blast radius and reversibility. A two-way-door flaw you fix forward; a one-way-door risk you stop.
- Fix, re-verify, re-ramp. Return to dual-write/diff to confirm the fix before resuming the ramp.
- Communicate the pause. Tell stakeholders why you paused — a controlled hold builds more trust than a hidden push-through.
- GSD — pull from the migration program, its phased milestones, and the per-team tracker driven to zero.
- Scuba / Unidash / ODS — pull the dual-write diff rate, ramp health metrics, and the cost reduction after decommission.
- Design-review notes — pull the migration design: compatibility layer, dual-write plan, and rollback strategy.
- Workplace posts / announcements — pull the ramp updates, the stakeholder syncs, and the decommission notice.
- Phabricator — pull the compatibility shim, dual-write, and cutover-flag diffs that made the migration reversible.