APIs & communication

An API is not a format question, it is a coupling question

Candidates treat "REST or GraphQL?" as a taste debate. It isn't. The API is the seam between teams that deploy independently, and the choice determines three things you cannot easily change later: how many round trips a client needs, who has to redeploy when a field changes, and whether an intermediary can cache the response. Everything else — JSON versus protobuf, verbs versus procedures — is downstream of those. The reason interviewers spend fifteen minutes here is that the answer reveals whether you have ever had to evolve an API that mobile clients from two years ago are still calling.

REST vs gRPC vs GraphQL, at the system level

The honest architecture at most large companies is all three at once: gRPC between internal services, a GraphQL or BFF layer for first-party clients, and REST at the public edge because that is what partners can integrate against without a codegen toolchain. Saying that — and saying why each boundary picked what it picked — is a stronger answer than defending one of them everywhere.

Versioning: you evolve an API, you rarely version it

Every version you ship you maintain forever, because the client that stops calling /v1 is the client that uninstalls the app. That makes additive, backward-compatible evolution the default and a version bump the failure case. The rules are boring and absolute: adding an optional field is safe; adding a required field, removing a field, renaming a field, narrowing a type, or changing the meaning of an existing value are all breaking, even when the tests pass.

Mobile is what makes this hard, and it is worth saying explicitly because it reframes the whole discussion. A web client is whatever you deployed thirty seconds ago. A mobile client is a distribution: an app-store rollout takes days, adoption tails for months, and some non-trivial slice of users never updates. So the design constraints are (1) the server must tolerate every client version simultaneously, forever, (2) you need per-version telemetry to know when a field is genuinely dead, and (3) a server-driven kill switch or forced-upgrade path is a feature you should have shipped in v1.

Sync vs async: when a call should have been an event

A synchronous call says "wait here while I do this, and while everything I depend on does its part." That is the right shape when the user's next action depends on the result. It is the wrong shape far more often than people write it.

Four symptoms tell you a request/response call should have been an event:

• The caller ignores the response. If the return value is discarded, or wrapped in a try/catch that swallows the error, you have written an event with extra steps and worse failure semantics.
• New consumers force a producer change. When "also notify the loyalty service" means editing checkout, the coupling is backwards. Publishing order.placed once and letting consumers subscribe is the entire point.
• The callee's availability became yours. Every synchronous hop enters the availability product. Five three-nines dependencies in series cap you at 99.50% before you write any of your own code.
• The work outlives the request budget. Video transcoding, PDF generation, bulk import. Return 202 Accepted with a status URL, and let the client poll or subscribe.

Then say the cost, unprompted, or you sound naive: asynchronous means at-least-once delivery, so consumers must be idempotent; it means ordering is only guaranteed within a partition key; it means the user's receipt now arrives "soon" rather than "now" and someone in product has to agree to that; and it means debugging a broken flow requires distributed tracing rather than a stack trace.

Real-time transports: the decision table you will need

Nearly every real-time design question — chat, notifications, live scores, collaborative editing, order tracking — bottoms out in this choice. Do the arithmetic before picking. One million clients short-polling every five seconds is 200,000 requests per second, and if only 2% of polls have anything to return, 196,000 of those per second are pure overhead: TLS, headers, auth, a database lookup, and a 204.

Idempotency keys, and retries that don't multiply

Any call that times out has three possible truths: it never arrived, it arrived and failed, or it arrived and succeeded and you lost the response. The client cannot tell them apart, so it retries, so every mutating endpoint must be safe to call twice. GET, PUT and DELETE are naturally idempotent. POST /payments is not, and that is where the money is.

The mechanism: the client generates a key per logical operation — not per attempt — and sends it as a header. The server makes the key part of the transaction.

BEGIN;
  -- UNIQUE (account_id, idempotency_key). This insert is the lock.
  INSERT INTO idempotency (account_id, key, request_hash, state)
  VALUES ($1, $2, $3, 'in_progress');   -- duplicate → unique violation

  -- the real work, in the SAME transaction, so it can never
  -- happen without the key being recorded, or vice versa
  INSERT INTO payments (...) VALUES (...);

  UPDATE idempotency SET state = 'done', response = $4 WHERE key = $2;
COMMIT;

Three details separate a working implementation from a plausible-sounding one, and interviewers probe all three:

• Bind the key to the request. Store a hash of the body. If the same key arrives with a different body, that is a client bug — return 422, never silently replay the old response or, worse, execute the new one.
• Handle the concurrent duplicate. The retry may arrive while the original is still running. The unique constraint rejects it, and the correct response is 409 in progress so the client backs off — not a wait, which converts one slow request into two held connections.
• Give keys a lifetime and say what it is. 24 hours to 7 days is typical. After that the record is reaped and a replayed request would execute again — which is fine, because no sane client retries a day later, but you should be the one to point that out.

Retry hygiene is the other half. Retry only on timeouts, connection errors, 429 and 5xx — never on 4xx, which will fail identically forever. Use exponential backoff with full jitter, cap attempts, and add a retry budget (abort retrying entirely if retries exceed ~10% of requests) so a struggling dependency doesn't receive 4× traffic at its worst moment. Honour Retry-After. And remember that every layer that retries multiplies: a client retrying 3× in front of a gateway retrying 3× in front of a service retrying 3× turns one user action into 27 backend calls. Retry at one layer, not every layer.

Pagination: why offset breaks and cursors don't

LIMIT 20 OFFSET 100000 does not skip 100,000 rows — the engine reads and discards 100,020 rows to hand you 20. Cost grows linearly with page depth: page 1 is a sub-millisecond index read, page 5,000 is hundreds of milliseconds and climbing, and a crawler walking to page 50,000 is an accidental denial-of-service you built yourself.

The correctness problem is worse than the performance one, because it is silent. Offsets address positions, and positions shift. On a feed where new rows arrive constantly, three items inserted between the user fetching page 1 and page 2 pushes three items they already saw onto page 2 — guaranteed duplicates. A deletion does the reverse and silently skips items. Every infinite scroll that repeats posts is this bug.

API gateways: what belongs there and what emphatically doesn't

A gateway is the one place every request passes through, which makes it the correct home for concerns that are identical for every request — and a catastrophic home for anything else, because it is also a single shared deployment across every team you have.

Two consequences to state out loud. First, the gateway is on the critical path for 100% of traffic, so its availability is the ceiling on everything behind it — it must be horizontally scaled, multi-AZ, and boringly simple. Second, the moment teams start putting logic in it you have built a distributed monolith with a shared release train: forty teams queueing behind one config repo, and an outage in the gateway is an outage in every product. The BFF pattern is the escape hatch — one thin aggregation service per client type (iOS, web, partner), owned by the team that consumes it, so mobile's needs never distort the public API and vice versa.

Recognizing it in an unseen problem

• The words "live", "real-time", "instant", "as soon as it happens" mean the interviewer wants the polling/long-poll/SSE/WebSocket decision — with the request-per-second arithmetic, not a preference. Ask first how stale is acceptable; "30 seconds" turns a WebSocket design into polling.
• "Mobile app" is a versioning and round-trip prompt. Say that old clients live forever, that you evolve additively, and that you would collapse the composite screen into one aggregated call.
• A naive design makes every internal call synchronous and never mentions timeouts, retries or idempotency — then designs a retry loop that quietly double-charges customers.
• Any mutating endpoint involving money, inventory or messages is an idempotency question in disguise. Volunteer the key before you're asked.
• "Show the user their history / feed / all their orders" with a large dataset is a cursor-pagination prompt. Offset in an interview reads as never having operated a table past a few million rows.
• Distinguish this from the message queue topic: this chapter is about the shape of a call between two parties; queues are about durability, ordering and buffering once you've decided the call should be asynchronous. Name the boundary and move on rather than re-deriving both.