APIs & communication

An API is not a format question, it is a coupling question

Candidates treat "REST or GraphQL?" as a taste debate. It isn't. The API is the seam between teams that deploy independently, and the choice determines three things you cannot easily change later: how many round trips a client needs, who has to redeploy when a field changes, and whether an intermediary can cache the response. Everything else — JSON versus protobuf, verbs versus procedures — is downstream of those. The reason interviewers spend fifteen minutes here is that the answer reveals whether you have ever had to evolve an API that mobile clients from two years ago are still calling.

REST vs gRPC vs GraphQL, at the system level

The honest architecture at most large companies is all three at once: gRPC between internal services, a GraphQL or BFF layer for first-party clients, and REST at the public edge because that is what partners can integrate against without a codegen toolchain. Saying that — and saying why each boundary picked what it picked — is a stronger answer than defending one of them everywhere.

Versioning: you evolve an API, you rarely version it

Every version you ship you maintain forever, because the client that stops calling /v1 is the client that uninstalls the app. That makes additive, backward-compatible evolution the default and a version bump the failure case. The rules are boring and absolute: adding an optional field is safe; adding a required field, removing a field, renaming a field, narrowing a type, or changing the meaning of an existing value are all breaking, even when the tests pass.

Mobile is what makes this hard, and it is worth saying explicitly because it reframes the whole discussion. A web client is whatever you deployed thirty seconds ago. A mobile client is a distribution: an app-store rollout takes days, adoption tails for months, and some non-trivial slice of users never updates. So the design constraints are (1) the server must tolerate every client version simultaneously, forever, (2) you need per-version telemetry to know when a field is genuinely dead, and (3) a server-driven kill switch or forced-upgrade path is a feature you should have shipped in v1.

Sync vs async: when a call should have been an event

A synchronous call says "wait here while I do this, and while everything I depend on does its part." That is the right shape when the user's next action depends on the result. It is the wrong shape far more often than people write it.

Four symptoms tell you a request/response call should have been an event:

• The caller ignores the response. If the return value is discarded, or wrapped in a try/catch that swallows the error, you have written an event with extra steps and worse failure semantics.
• New consumers force a producer change. When "also notify the loyalty service" means editing checkout, the coupling is backwards. Publishing order.placed once and letting consumers subscribe is the entire point.
• The callee's availability became yours. Every synchronous hop enters the availability product. Five three-nines dependencies in series cap you at 99.50% before you write any of your own code.
• The work outlives the request budget. Video transcoding, PDF generation, bulk import. Return 202 Accepted with a status URL, and let the client poll or subscribe.

Then say the cost, unprompted, or you sound naive: asynchronous means at-least-once delivery, so consumers must be idempotent; it means ordering is only guaranteed within a partition key; it means the user's receipt now arrives "soon" rather than "now" and someone in product has to agree to that; and it means debugging a broken flow requires distributed tracing rather than a stack trace. Say how the event is published, too. Charging a card and then publishing to the bus are two separate writes, and a crash between them loses the event (or, in the other order, announces an order that never happened). The standard fix is the transactional outbox: write the event into an outbox table in the same database transaction as the order and let a relay publish it. Delivery is then at-least-once, and "exactly once" in practice means at-least-once plus idempotent consumers.

Real-time transports: the decision table you will need

Nearly every real-time design question — chat, notifications, live scores, collaborative editing, order tracking — bottoms out in this choice. Do the arithmetic before picking. One million clients short-polling every five seconds is 200,000 requests per second, and if only 2% of polls have anything to return, 196,000 of those per second are pure overhead: TLS, headers, auth, a database lookup, and a 204.

Idempotency keys, and retries that don't multiply

Any call that times out has three possible truths: it never arrived, it arrived and failed, or it arrived and succeeded and you lost the response. The client cannot tell them apart, so it retries, so every mutating endpoint must be safe to call twice. GET, PUT and DELETE are naturally idempotent. POST /payments is not, and that is where the money is.

The mechanism: the client generates a key per logical operation — not per attempt — and sends it in an Idempotency-Key header, the convention popularised by APIs such as Stripe's (which keeps keys for at least 24 hours). The header is still an IETF Internet-Draft (draft-ietf-httpapi-idempotency-key-header, revision 07 of October 2025, since expired), not an RFC. The server records the key before doing the work, in its own short transaction that commits immediately.

-- Step 1: claim the key. Its own transaction, committed at once.
-- UNIQUE (account_id, key). 1 row inserted = we own it; 0 rows = someone
-- else does, so SELECT the row and branch on state and request_hash.
INSERT INTO idempotency (account_id, key, request_hash, state, locked_until)
VALUES ($1, $2, $3, 'in_progress', now() + interval '30 seconds')
ON CONFLICT (account_id, key) DO NOTHING;

-- Step 2: do the work. A card charge happens at an external processor, outside
-- any transaction of yours: pass the processor a key derived from ours, so it
-- deduplicates even if we crash and retry.

-- Step 3: record the outcome. Local effects and the key update are atomic.
BEGIN;
  INSERT INTO payments (...) VALUES (...);
  UPDATE idempotency SET state = 'done', response = $4
   WHERE account_id = $1 AND key = $2;
COMMIT;

Why not claim the key and do everything in one transaction? In Postgres the uncommitted claim row is invisible to other sessions, and a second inserter of the same key does not fail fast: it blocks until the first transaction ends, then gets a unique violation. The "fast 409" would never happen and a retry would just hang behind the original. Committing the claim first makes the duplicate visible immediately.

Four details separate a working implementation from a plausible-sounding one, and interviewers probe all of them:

• Bind the key to the request. Store a hash of the body. If the same key arrives with a different body, that is a client bug — return 422, never silently replay the old response or, worse, execute the new one.
• Handle the concurrent duplicate. The retry may arrive while the original is still running. Because the claim is already committed, the insert is a no-op, the lookup shows in_progress, and the correct response is 409 so the client backs off (the IETF draft also recommends 409) — not a wait, which converts one slow request into two held connections.
• Survive a crash mid-flight. If the server dies after the charge but before step 3, the row stays in_progress. The lease (locked_until) lets a later retry take over; it repeats the processor call with the same derived key, and the processor returns the original charge instead of making a second one. Without a lease, that key is stuck forever.
• Give keys a lifetime and say what it is. 24 hours to 7 days is typical. After that the record is reaped and a replayed request would execute again — which is fine, because no sane client retries a day later, but you should be the one to point that out.

Dry run: an idempotency key across a timeout, a retry, and a concurrent duplicate

Notice what makes this safe, and what does not. The local database write and the outcome record are atomic (step 3), but the card charge is not: it lives at an external processor, outside your transaction. The processor's own idempotency key is what stops a second charge, and the committed claim row plus its lease are what let the server resume after a crash. The client cannot tell a lost response from a lost request, but the server never has to guess — it just answers "what did I already decide for key K?"

Retry hygiene is the other half. Retry only on timeouts, connection errors, 408, 429 and 5xx — never on other 4xx codes, which will fail identically forever. Use exponential backoff with full jitter, cap attempts, and add a retry budget (for example, stop retrying once retries exceed ~10% of requests, a common starting point) so a struggling dependency doesn't see its traffic multiplied at its worst moment; with two retries per request and no budget, it sees up to 3×. Pair retries with timeouts and a circuit breaker, and honour Retry-After. And remember that every layer that retries multiplies. Count attempts, not retries: a client, a gateway and a service that each make up to 3 attempts (2 retries) turn one user action into 3 × 3 × 3 = 27 backend calls, and the Google SRE book's example, where each of three layers retries 3 times (4 attempts), reaches 4³ = 64. Retry at one layer, not every layer.

Pagination: why offset breaks and cursors don't

LIMIT 20 OFFSET 100000 does not skip 100,000 rows — the engine reads and discards 100,020 rows to hand you 20. Cost grows linearly with page depth: page 1 is a sub-millisecond index read, page 5,000 is hundreds of milliseconds and climbing, and a crawler walking to page 50,000 is an accidental denial-of-service you built yourself.

The correctness problem is worse than the performance one, because it is silent. Offsets address positions, and positions shift. On a feed where new rows arrive constantly, three items inserted between the user fetching page 1 and page 2 pushes three items they already saw onto page 2 — guaranteed duplicates. A deletion does the reverse and silently skips items. Every infinite scroll that repeats posts is this bug.

API gateways: what belongs there and what emphatically doesn't

A gateway is the one place every request passes through, which makes it the correct home for concerns that are identical for every request — and a catastrophic home for anything else, because it is also a single shared deployment across every team you have.

Two consequences to state out loud. First, the gateway is on the critical path for 100% of traffic, so its availability is the ceiling on everything behind it — it must be horizontally scaled, multi-AZ, and boringly simple. Second, the moment teams start putting logic in it you have built a distributed monolith with a shared release train: forty teams queueing behind one config repo, and an outage in the gateway is an outage in every product. The BFF pattern is the escape hatch — one thin aggregation service per client type (iOS, web, partner), owned by the team that consumes it, so mobile's needs never distort the public API and vice versa.

Recognizing it in an unseen problem

• The words "live", "real-time", "instant", "as soon as it happens" mean the interviewer wants the polling/long-poll/SSE/WebSocket decision — with the request-per-second arithmetic, not a preference. Ask first how stale is acceptable; "30 seconds" turns a WebSocket design into polling.
• "Mobile app" is a versioning and round-trip prompt. Say that old clients live forever, that you evolve additively, and that you would collapse the composite screen into one aggregated call.
• A naive design makes every internal call synchronous and never mentions timeouts, retries or idempotency — then designs a retry loop that quietly double-charges customers.
• Any mutating endpoint involving money, inventory or messages is an idempotency question in disguise. Volunteer the key before you're asked.
• "Show the user their history / feed / all their orders" with a large dataset is a cursor-pagination prompt. Offset in an interview reads as never having operated a table past a few million rows.
• Distinguish this from the message queue topic: this chapter is about the shape of a call between two parties; queues are about durability, ordering and buffering once you've decided the call should be asynchronous. Name the boundary and move on rather than re-deriving both.