Client-server basics

The one question that separates levels: "what happens when I hit enter?"

This looks like a trivia question and is actually a depth probe. Anyone can say "the browser sends a request to the server." The signal is in whether you can name each hop, say roughly what it costs, and identify which hops you get to influence as a designer. Every scaling technique in later chapters — caching, load balancing, CDNs, replicas — is an intervention at one specific point on this path. If the path is fuzzy, the interventions sound arbitrary.

DNS: four lookups you hope never to make

The browser needs an IP address before it can open a socket. It asks its configured recursive resolver (your ISP's, or a public one like 8.8.8.8), and that resolver is the component that does the actual walking: it asks a root nameserver who handles .com, asks that TLD server who is authoritative for example.com, and asks the authoritative nameserver for the record. Each step is a network round trip, which is why a genuinely cold resolution can cost 100 ms or more, and why in practice it costs zero — every layer caches.

Caching is governed by the record's TTL. A 60-second TTL means fast failover and heavy query load on your nameservers; a 24-hour TTL means the opposite. The critical property for design: the TTL is a hint, not a contract. Resolvers clamp it, operating systems cache on top of it, and browsers keep their own cache for a minute or two regardless. If you plan to move traffic by changing a DNS record, assume a long tail of clients keeps using the old answer for hours.

The record types you will actually name in a design are few: A and AAAA map a name to an IPv4 or IPv6 address, CNAME aliases one name to another (not allowed at the zone apex, which is why providers invent "ALIAS" records), NS delegates a zone, MX routes mail, and TXT carries verification and policy strings. When a name returns both A and AAAA records, clients using Happy Eyeballs (RFC 8305) start IPv6 first and, if it has not connected after a short delay (RFC 8305 recommends 250 ms), start IPv4 and keep whichever completes first, so a broken IPv6 path costs a fraction of a second rather than a full timeout.

So DNS is a load balancer, but a bad one: it balances resolvers rather than requests, it can't see that a server is at 99% CPU, it can't do per-request decisions, and its failover is measured in minutes. It is the right tool for coarse geographic steering and the wrong tool for anything reactive.

TCP and TLS: the cost of a cold connection

With an IP in hand the client still cannot send a byte of HTTP. TCP needs a three-way handshake (one round trip before the client can send data), and TLS needs its own negotiation on top: two round trips in TLS 1.2, one in TLS 1.3, and zero on resumption if you accept the replay risk of 0-RTT data.

Two design consequences follow directly. First, connection reuse is not an optimisation, it's the baseline: HTTP keep-alive, connection pools in your service-to-service clients, and pooled database connections all exist to amortise this cost. A service that opens a fresh TLS connection per request has added several RTTs and a public-key operation to every call. A fresh TCP connection is also slow in a second way: congestion control starts it in slow start, with an initial window of about 10 segments (roughly 14 KB, RFC 6928), so a response bigger than that needs extra round trips to ramp up even after the handshakes. A warm, reused connection has already ramped. Second, terminating TLS close to the user matters enormously. A CDN PoP 10 ms from the user absorbs the handshake round trips locally and reuses a warm connection back to origin, which is often a bigger win than caching the content itself.

HTTP/1.1 vs HTTP/2 vs HTTP/3, at the level that changes a design

The practical fallout you should be able to state: because HTTP/1.1 allows one outstanding request per connection, browsers open about six connections per origin, and the old trick of "domain sharding" existed to buy more. On HTTP/2 that trick is actively harmful — it defeats multiplexing and header compression and multiplies handshakes. And HTTP/2's remaining weakness is real: multiplexing many streams over one TCP connection means one dropped packet stalls all of them, which is exactly the case QUIC was built to fix.

Load balancer, reverse proxy, API gateway

These overlap enough that candidates use them interchangeably and get caught. A reverse proxy is any server that accepts a client connection and makes its own request to a backend on the client's behalf; a load balancer is a reverse proxy whose defining job is distributing across many backends; an API gateway is an L7 reverse proxy that additionally owns cross-cutting concerns — authentication, rate limiting, request shaping, per-route policy. One box often does all three.

Beyond distribution, the load balancer is where three other things live, and naming them unprompted is a strong signal: health checks (active probes plus passive outlier ejection when a backend starts erroring), TLS termination, and the balancing algorithm itself. Round robin is fine when every request costs the same; least-outstanding-requests is materially better when they don't, because it routes away from a backend that has quietly become slow. Consistent hashing is the choice when backends hold a warm per-key cache and you want the same key to land on the same node.

Statelessness is the property everything else is built on

A stateless app server is one where any request can be served correctly by any instance, because no request depends on memory left behind by a previous request on that same box. Every request carries or fetches everything it needs. That single property is what makes horizontal scaling, rolling deploys, autoscaling, and instance failure all boring — you can add, remove, or kill a server without anyone noticing.

Statelessness does not mean the system has no state. It means the state has been moved somewhere purpose-built: a database, a cache, a blob store, or the client itself. The interview question is always where did you put it.

Numbers every engineer should know

You will be asked to justify a latency budget, and the justification has to be built from components. Memorise the orders of magnitude, not the digits. The single most useful mental jump is that each of these tiers is somewhere between 10x and 1,000x slower than the last: nanoseconds in cache and memory (about 1 ns in L1, about 100 ns for main memory), tens of microseconds on SSD, about half a millisecond for a datacenter round trip, and hundreds of milliseconds across the planet. Going from memory to a same-datacenter round trip alone is a factor of about 5,000.

These are rounded planning numbers, not measurements. The original table attributed to Jeff Dean (mutex 25 ns, 1 MB read from memory 250 µs, disk seek 10 ms) describes hardware of roughly that era; Colin Scott's interactive page extrapolates it to today and is where the optimistic 3 µs figure for a sequential memory read comes from. Where the two disagree this table gives a range.

Note that the speed of light in fibre is about 200,000 km/s, so New York to London (5,600 km) has a physical floor near 55 ms round trip. Measured paths on that route run only about 15% above it (around 64 ms); less direct routes run further above their floor. When you say "we cannot make this call synchronous across regions", you are citing physics, and interviewers recognise the difference between that and an opinion.

Dry run: a 200 ms page-load budget for a user 80 ms away

Three round trips for a cold connection cost more than the entire 200 ms budget by themselves — the server never gets to run. The identical request on a warm, reused connection has 120 ms of budget left before the server does anything, which is why connection reuse and edge TLS termination are not tuning knobs but the difference between meeting the budget and missing it entirely.

Recognizing it in an unseen problem

• Any prompt with a latency target ("under 200 ms p99") is asking you to build a budget out of the table above — say where the milliseconds go before proposing optimisations
• "Users are global" means the physics number, ~100-250 ms RTT, is now in every request; the design answer is edge termination and read-local replicas, not faster servers
• If a component holds per-user state in memory, the interviewer will ask what happens when it restarts — decide up front whether that state is durable, reconstructible, or genuinely disposable
• "How would you do a zero-downtime deploy / autoscale this?" is a statelessness question wearing a costume
• DNS is the answer for coarse geographic routing and gradual migrations; it is never the answer for fast failover or per-request balancing — an interviewer probing failover wants anycast or a floating VIP
• Distinguish bandwidth problems from round-trip problems: large media is a bandwidth and CDN problem, chatty APIs are a round-trip problem, and they have completely different fixes