Client-server basics
A senior candidate can narrate everything between the keypress and the pixel — and knows what each hop costs in milliseconds.
The one question that separates levels: "what happens when I hit enter?"
This looks like a trivia question and is actually a depth probe. Anyone can say "the browser sends a request to the server." The signal is in whether you can name each hop, say roughly what it costs, and identify which hops you get to influence as a designer. Every scaling technique in later chapters — caching, load balancing, CDNs, replicas — is an intervention at one specific point on this path. If the path is fuzzy, the interventions sound arbitrary.
DNS: four lookups you hope never to make
The browser needs an IP address before it can open a socket. It asks its
configured recursive resolver (your ISP's, or a public one like
8.8.8.8), and that resolver is the component that does the actual walking:
it asks a root nameserver who handles .com, asks that TLD
server who is authoritative for example.com, and asks the
authoritative nameserver for the record. Each step is a network round trip,
which is why a genuinely cold resolution can cost 100 ms or more, and why
in practice it costs zero — every layer caches.
Caching is governed by the record's TTL. A 60-second TTL means fast failover and heavy query load on your nameservers; a 24-hour TTL means the opposite. The critical property for design: the TTL is a hint, not a contract. Resolvers clamp it, operating systems cache on top of it, and browsers keep their own cache for a minute or two regardless. If you plan to move traffic by changing a DNS record, assume a long tail of clients keeps using the old answer for hours.
| DNS routing strategy | Mechanism | Reach for this when… |
|---|---|---|
| Multiple A records (round robin) | Return several IPs; clients pick roughly at random | You need crude spread across a handful of static endpoints and nothing better is available |
| Weighted records | Return IP A 95% of the time, IP B 5% | Canary deploys and gradual migrations between stacks or providers |
| GeoDNS / latency-based | Answer depends on the resolver's location or measured latency | Multi-region: send European users to the European stack, cutting 80-100 ms off every request |
| Health-checked failover | Provider probes endpoints and withdraws dead records | Regional disaster recovery — accept that failover takes TTL plus stubborn-cache time |
| Anycast | One IP announced from many locations; BGP routes to the nearest | What CDNs and large services actually use — failover in seconds, no client caching problem, but requires network-level infrastructure |
So DNS is a load balancer, but a bad one: it balances resolvers rather than requests, it can't see that a server is at 99% CPU, it can't do per-request decisions, and its failover is measured in minutes. It is the right tool for coarse geographic steering and the wrong tool for anything reactive.
TCP and TLS: the cost of a cold connection
With an IP in hand the client still cannot send a byte of HTTP. TCP needs a three-way handshake (one round trip before the client can send data), and TLS needs its own negotiation on top: two round trips in TLS 1.2, one in TLS 1.3, and zero on resumption if you accept the replay risk of 0-RTT data.
Two design consequences follow directly. First, connection reuse is not an optimisation, it's the baseline: HTTP keep-alive, connection pools in your service-to-service clients, and pooled database connections all exist to amortise this cost. A service that opens a fresh TLS connection per request has added several RTTs and a public-key operation to every call. Second, terminating TLS close to the user matters enormously. A CDN PoP 10 ms from the user absorbs the handshake round trips locally and reuses a warm connection back to origin, which is often a bigger win than caching the content itself.
HTTP/1.1 vs HTTP/2 vs HTTP/3, at the level that changes a design
| HTTP/1.1 | HTTP/2 | HTTP/3 | |
|---|---|---|---|
| Transport | TCP, one request in flight per connection | TCP, many streams multiplexed on one connection | QUIC over UDP, streams are independent |
| Head-of-line blocking | At the HTTP layer — a slow response blocks the connection | Fixed at HTTP layer, still present at TCP layer: one lost packet stalls every stream | Gone — a lost packet stalls only its own stream |
| Handshake | TCP + TLS, 2-3 RTT | Same | 1 RTT combined, 0-RTT on resumption |
| Headers | Plain text, repeated in full every request | HPACK compression | QPACK compression |
| Connection migration | Breaks on network change | Breaks on network change | Survives a Wi-Fi to cellular switch via connection ID |
| Reach for this when… | Simple internal service-to-service; still the default for many proxies to origin | Default for browser traffic and gRPC; many small resources over one connection | Lossy or mobile networks, and latency-sensitive global traffic — the win grows with packet loss |
The practical fallout you should be able to state: because HTTP/1.1 allows one outstanding request per connection, browsers open about six connections per origin, and the old trick of "domain sharding" existed to buy more. On HTTP/2 that trick is actively harmful — it defeats multiplexing and header compression and multiplies handshakes. And HTTP/2's remaining weakness is real: multiplexing many streams over one TCP connection means one dropped packet stalls all of them, which is exactly the case QUIC was built to fix.
Load balancer, reverse proxy, API gateway
These overlap enough that candidates use them interchangeably and get caught. A reverse proxy is any server that accepts a client connection and makes its own request to a backend on the client's behalf; a load balancer is a reverse proxy whose defining job is distributing across many backends; an API gateway is an L7 reverse proxy that additionally owns cross-cutting concerns — authentication, rate limiting, request shaping, per-route policy. One box often does all three.
| L4 (transport) | L7 (application) | |
|---|---|---|
| Sees | IPs, ports, TCP connections | Full HTTP: method, path, headers, cookies, body |
| Can do | Connection-level distribution, extremely high throughput, near-zero added latency | Path and header routing, TLS termination, retries, rate limiting, request rewriting, sticky sessions, per-request balancing |
| Cost | Blind to application health and to individual requests on a long-lived connection | More CPU, added latency (typically well under a millisecond), and it must hold the TLS keys |
| Reach for this when… | Raw TCP services, databases, extreme packet rates, or you want the backend to see the real client TLS | Essentially all HTTP traffic — the routing and observability are worth the overhead |
Beyond distribution, the load balancer is where three other things live, and naming them unprompted is a strong signal: health checks (active probes plus passive outlier ejection when a backend starts erroring), TLS termination, and the balancing algorithm itself. Round robin is fine when every request costs the same; least-outstanding-requests is materially better when they don't, because it routes away from a backend that has quietly become slow. Consistent hashing is the choice when backends hold a warm per-key cache and you want the same key to land on the same node.
Statelessness is the property everything else is built on
A stateless app server is one where any request can be served correctly by any instance, because no request depends on memory left behind by a previous request on that same box. Every request carries or fetches everything it needs. That single property is what makes horizontal scaling, rolling deploys, autoscaling, and instance failure all boring — you can add, remove, or kill a server without anyone noticing.
Statelessness does not mean the system has no state. It means the state has been moved somewhere purpose-built: a database, a cache, a blob store, or the client itself. The interview question is always where did you put it.
| Where session state lives | Cost | Reach for this when… |
|---|---|---|
| In app server memory + sticky sessions | Losing a node logs users out; deploys are disruptive; load skews toward whichever node holds the busy users; autoscaling barely helps | Almost never in a new design — know it mainly so you can explain why you rejected it |
| Shared session store (Redis, Memcached) | One extra network hop (~0.5 ms) per request; the store becomes a dependency you must make highly available | You need server-side revocation, large session payloads, or session data that changes mid-session |
| Signed token in a cookie (JWT and friends) | No lookup at all, but revocation is genuinely hard and every request pays the token size in bytes | Read-mostly identity claims with short expiry, plus a refresh token you can revoke server-side |
Numbers every engineer should know
You will be asked to justify a latency budget, and the justification has to be built from components. Memorise the orders of magnitude, not the digits. The single most useful mental jump is that each of these tiers is roughly 100x apart: nanoseconds in cache, microseconds in memory and SSD, milliseconds on the network, hundreds of milliseconds across the planet.
| Operation | Time | What it means for you |
|---|---|---|
| L1 cache reference | ~1 ns | Free; never a design consideration |
| Branch mispredict | ~3 ns | Free |
| Mutex lock/unlock | ~20 ns | Contention, not the lock itself, is what hurts |
| Main memory reference | ~100 ns | An in-process cache hit is ~1000x faster than a network cache hit |
| Read 1 MB sequentially from memory | ~3 µs | Serialisation and copying usually cost more than the read |
| SSD random read | ~16-100 µs | A cache miss to local NVMe is survivable; to a remote DB it is not |
| Round trip in the same datacenter | ~0.5 ms | Every internal service hop costs this at minimum — this is why chatty microservices die |
| Redis GET over the network | ~0.2-1 ms | Dominated by the round trip, not by Redis |
| Simple indexed DB query, warm | ~1-10 ms | Your typical read-path floor |
| HDD seek | ~2-10 ms | Why random I/O on spinning disks shaped a generation of storage design |
| Cross-region RTT (US East to US West) | ~60-70 ms | One synchronous cross-country call blows a 100 ms budget on its own |
| Cross-continent RTT (US East to Europe) | ~80-100 ms | Speed of light in fibre, not an engineering problem you can optimise |
| Cross-planet RTT (US to Singapore/Sydney) | ~200-250 ms | Multi-region read-local architecture is mandatory, not a nice-to-have |
Note that the speed of light in fibre is about 200,000 km/s, so New York to London (5,600 km) has a physical floor near 56 ms round trip; real paths run 1.5-2x that. When you say "we cannot make this call synchronous across regions", you are citing physics, and interviewers recognise the difference between that and an opinion.
Recognizing it in an unseen problem
- Any prompt with a latency target ("under 200 ms p99") is asking you to build a budget out of the table above — say where the milliseconds go before proposing optimisations
- "Users are global" means the physics number, ~100-250 ms RTT, is now in every request; the design answer is edge termination and read-local replicas, not faster servers
- If a component holds per-user state in memory, the interviewer will ask what happens when it restarts — decide up front whether that state is durable, reconstructible, or genuinely disposable
- "How would you do a zero-downtime deploy / autoscale this?" is a statelessness question wearing a costume
- DNS is the answer for coarse geographic routing and gradual migrations; it is never the answer for fast failover or per-request balancing — an interviewer probing failover wants anycast or a floating VIP
- Distinguish bandwidth problems from round-trip problems: large media is a bandwidth and CDN problem, chatty APIs are a round-trip problem, and they have completely different fixes