← Back to System Internals

Reviewing API, Schema & Query Compatibility

The most expensive review miss isn't a bug — it's a breaking change that ships to a hundred consumers you didn't know about.

In a large system, interfaces are contracts, and a diff that changes a contract can break producers and consumers that the author never sees. This question tests whether you review schema and API changes with the whole ecosystem in mind: who reads this field, who writes it, what happens during the window when old and new code both run. For an events-infra engineer this is bread and butter — Thrift/protobuf/Avro schemas flow through pipelines and land in tables that dozens of teams query, so a careless field change is a silent data-corruption or outage event, not a local bug.

What this question is really testing

Whether you think in terms of rollout windows and independent deployment. The core insight is that producers and consumers are never upgraded atomically, so every change must be safe when old and new versions coexist. The tell is that you reach for “additive and backward-compatible” by default, know the concrete wire-format rules that make a change safe or breaking, and always ask “who are the consumers?” before approving.

How to answer

• Additive vs breaking, by default additive. Adding an optional field is safe; removing a field, renaming it, changing its type, or making an optional field required is breaking. Reordering positional fields or changing enum meaning is breaking. Classify the change first, then decide the rollout.
• Wire-format field rules. For protobuf/Thrift, the field number/tag is the identity — never reuse or renumber a tag, never change a field's type, and reserve removed tags/names so they can't be recycled. For Avro, compatibility rides on the reader/writer schema pair and on defaults. Names are cosmetic in tag-based formats but load-bearing in name-based ones (JSON, Avro).
• Defaults. A new field needs a sensible default so old readers and old data behave correctly, and so a missing value is distinguishable from a zero value where that matters. Watch proto3's no-explicit-presence trap — you can't tell “unset” from “default” without optional/wrappers.
• Schema evolution: forward and backward. Backward = new code reads old data; forward = old code reads new data. Pipelines usually need both, because data written yesterday is read by code deployed today and vice versa. Confirm the change survives both directions.
• Deprecation path. To remove or change a field, do it in stages: add the new, dual-write, migrate readers, verify no one reads the old, then remove. Never a big-bang rename. The diff should show which stage it's in.
• Downstream consumers. Ask explicitly: who consumes this schema/endpoint? Pipelines, dashboards, other services, external partners? Use code search and lineage to find them. “I couldn't find a caller” is not the same as “there is no caller.”
• Query-language and semantic changes. A change that keeps the schema but alters semantics — a filter's default, a join's cardinality, a null-handling rule, a unit or timezone — is a silent breaking change. It compiles, tests pass, and downstream numbers quietly shift.
• Data correctness and schema stability over time. The table or topic is a long-lived asset. Ask whether the change preserves the meaning of historical data, whether backfills are needed, and whether the schema stays stable enough to trust in year-over-year analysis.

What the interviewer is looking for

• “Additive and backward/forward compatible” as the default posture.
• Precise knowledge of wire-format rules — tag identity, no type changes, reserve removed fields.
• “Who are the consumers?” asked before approval, with lineage/code search to find them.
• A staged deprecation path instead of a big-bang rename or removal.
• Awareness that semantic changes are breaking even when the schema is unchanged.

The classic wire-format catch — a protobuf change that looks harmless and silently corrupts every consumer:

// WRONG: reused tag 2 and changed a type -- old readers misparse the bytes
message AdEvent {
  string ad_id = 1;
  int32  region_id = 2;   // was: string campaign_id = 2;  BREAKING
}

// RIGHT: reserve the old tag/name, add the new field with a fresh tag
message AdEvent {
  reserved 2;                 // never recycle tag 2
  reserved "campaign_id";
  string ad_id = 1;
  int32  region_id = 3;       // additive: new tag, optional, defaulted
}

Common follow-ups

Where to get your data (Meta)

• Code search + data lineage — to enumerate every producer and consumer of a schema, topic, or table before approving.
• Phabricator — a review where you caught a tag reuse, a type change, or a required-field addition before it shipped.
• SEV / postmortems — an incident from a breaking schema/API change and the compatibility guardrail it produced.
• Internal wiki — the Thrift/protobuf/Avro compatibility guidelines and the schema-change review checklist.
• Scuba / Unidash — field-population and downstream-metric dashboards that reveal a semantic change after the fact.

---