Collaborating During Major Incidents
Running a cross-team incident with a single commander, comms discipline, and follow-through that outlives the SEV.
This question tests how you lead when the system is on fire and the fix spans org boundaries. A major incident that touches many teams fails not for lack of talent but for lack of coordination — five people debugging in parallel, no one deciding, leadership asking for status in the same channel engineers are trying to work in. The interviewer wants to see that you know the incident-command model, that you prioritize mitigation over root-cause in the moment, that you keep communication disciplined, and that you drive a blameless postmortem whose action items actually get done across every org involved. The answer follows the CARL shape.
What this question is really testing
Can you impose order on a multi-team crisis — establish command, restore service fast, communicate cleanly — and then convert the incident into durable fixes that span organizations, without hunting for someone to blame?
How to answer- Establish a single incident commander. One person owns coordination and decisions — not the debugging, the direction. In a cross-team SEV the biggest failure mode is diffuse ownership; naming one commander (who may not be the most senior person) is the first move.
- Assign clear roles. Separate the commander from the operations lead doing the hands-on work, the communications lead handling stakeholders, and a scribe keeping the timeline. Roles prevent the two classic failures: everyone doing the same thing, or a critical thing no one owns.
- Enforce comms discipline. One incident channel, a regular status cadence, and a communications lead who shields the responders from the leadership firehose. Stakeholders get updates on a clock; engineers get to focus.
- Mitigate before root-cause. The first job is to stop the bleeding — roll back, fail over, shed load, flip the flag — and understand why afterward. Restore the user experience first; diagnose second.
- Run a blameless postmortem and drive action items to zero. Grade the system, not the person; find the real root cause; and — the part most orgs drop — track every action item to completion across all the teams involved, because the incident isn’t closed until the class can’t recur.
- Fluency with the incident-command model — single commander, distinct roles.
- Mitigation instinct — restore service before chasing the root cause.
- Communication treated as a first-class workstream, shielding responders.
- A blameless culture — the reflex to fix the system, not assign fault.
- Relentless follow-through on cross-org action items, not just a nice writeup.
A worked example (CARL)
Context. A schema change in an upstream producer combined with a config push in our ads events storage tier caused write failures that cascaded into two downstream teams — measurement dashboards went stale and a billing pipeline started dropping events. Three teams were paged at once. Within ten minutes there were four separate debugging threads, a director asking for status in the main channel, and no single person actually deciding what to do. I was the most senior engineer online across the affected teams.
Actions. My first move was not to debug — it was to impose structure. I declared a single SEV, took the incident commander role explicitly, and said so in the channel: “I’m IC, all coordination here.” Then I assigned roles rather than letting people self-organize into chaos: I named an operations lead on my storage team to drive the hands-on mitigation, an operations lead on the producer team since the trigger was theirs, a communications lead to own stakeholder updates and get the director out of the working channel and onto a status cadence, and asked one person to scribe the timeline. I made the priority explicit and unpopular: we would mitigate before we understood. Several engineers wanted to root-cause the schema interaction first; I ruled that we’d roll back the config push and the schema change immediately to restore writes, and diagnose the interaction afterward — stop the bleeding first. The comms lead posted status every 15 minutes on a clock so stakeholders stopped interrupting responders. Once writes recovered, I kept the SEV open until we’d confirmed the downstream billing pipeline had caught up with no permanent event loss, because “our part is fixed” isn’t “the incident is resolved” when it spans teams. Afterward I ran the postmortem blameless — the root cause was that the producer’s schema change and our config push were individually safe but unsafe together, with no gate that could see the combination. The action items spanned three orgs, which is exactly where follow-through usually dies, so I created a shared tracker, assigned each item a single owner with a date, and reviewed it in the weekly ops meeting until every item was closed.
Results. We cut time-to-mitigation to about 25 minutes by rolling back before diagnosing, and confirmed zero permanent event loss in billing. The blameless postmortem produced a cross-team compatibility check that catches unsafe producer-plus-storage change combinations in CI, and every one of the cross-org action items was closed within the quarter — the same class of incident hasn’t recurred since.
Learnings. In a multi-team incident, coordination is the scarce resource, not engineering skill — naming one commander and clear roles is worth more than another debugger. Mitigate-before-root-cause is a discipline you have to enforce against smart people’s instinct to understand first. And a cross-org postmortem is only real if someone owns driving the action items to zero after the adrenaline fades.
Common follow-ups
What if no one obvious should be incident commander, or teams argue over who owns it?
How to answer- Someone just takes it. An imperfect commander beats none — step in and declare it, seniority is not the criterion.
- IC coordinates, doesn’t own the fix. The role is direction and decisions, so it can sit above any one team.
- Hand off cleanly if needed. If a better-placed commander emerges, transfer explicitly with a state summary.
- Pre-agree the model. The time to decide who can be IC is before the SEV, in the runbook — not during it.
How do you handle leadership demanding updates during the incident?
How to answer- Route them to the comms lead. Shield responders — leadership talks to comms, not to the person mid-mitigation.
- Update on a cadence. A predictable status clock removes the need for individual pings.
- Give impact, not speculation. Report scope, mitigation status, and ETA to next update — not half-formed theories.
- Escalate for decisions, not narration. Pull leadership in when you need a business call (e.g. shed load vs degrade), not for play-by-play.
Postmortem action items always seem to rot. How do you make them stick?
How to answer- One owner, one date each. No shared ownership — every item has a name and a deadline.
- Track in a real system. Tasks or a tracker reviewed weekly, not a bullet list in a doc no one reopens.
- Prioritize the prevention items. The guardrail that stops recurrence outranks cosmetic cleanups.
- Report closure to leadership. Visibility on open action items creates the pressure that keeps them moving across orgs.
- SEV / postmortem records — pull the incident timeline, time-to-mitigation, root cause, and the blameless writeup.
- Scuba / Unidash / ODS — pull the impact and recovery graphs (error rate, event loss, freshness) that quantify the SEV.
- GSD / Tasks — pull the cross-org action-item tracker and its burn-down to zero.
- Workplace posts — pull the incident comms and the postmortem share-out to stakeholders.
- The internal wiki — pull from the incident-command runbook and the guardrail (e.g. compatibility check) that came out of the review.