Load balancing in depth

The most-drawn, least-understood box on the whiteboard

Every candidate draws a load balancer. Very few can say what layer it operates at, what it can and cannot see, how it decides where a request goes, how it learns a backend is dead, or what happens to the connections in flight when you deploy. Those five questions are the entire chapter, and they are asked because the load balancer is simultaneously the thing that makes a system horizontally scalable and the single component every request must survive. Get it wrong and your redundancy is decorative — a health check that is too eager can convert a slow backend into a dead fleet in under a minute.

L4 vs L7: what only L7 can do

An L4 balancer picks a backend once, per connection, from the 5-tuple, then shovels bytes. It never decrypts, never parses, and therefore costs almost nothing — a software L4 on commodity hardware handles millions of concurrent flows and tens of gigabits, and with direct server return the response never traverses the balancer at all, which matters enormously for video and download workloads. An L7 proxy terminates the connection, does the TLS handshake, parses the HTTP request, and makes a fresh decision per request.

Algorithms, and when each one is right

Power of two choices is worth naming explicitly because it sounds like a compromise and is actually close to optimal. Pure random assignment leaves the busiest server with roughly log n / log log n times the average queue; sampling just two and taking the shorter queue reduces that to about log log n — an exponential improvement bought with zero coordination. It is why modern proxies default to it rather than to true least-connections, and mentioning it is a cheap, genuine seniority signal.

Consistent hashing, properly

Suppose you route cache keys with hash(key) mod N. With four nodes you add a fifth, and every key whose home changes loses its cached value. How many change? Work it out with the Chinese remainder theorem: a key keeps its home only when k mod 4 == k mod 5, which for k mod 20 happens exactly for 0, 1, 2 and 3. That is 4 out of 20 — 20% stay, 80% move. You added capacity and instantly invalidated four fifths of your warm cache.

On a ring, both keys and nodes are hashed into the same space (say 0 to 2³²−1). A key is owned by the first node clockwise from it. Adding a node inserts one new point and it steals only the arc between itself and its predecessor: an expected 1/(N+1) of all keys. Going from four nodes to five moves 20% instead of 80%. At a hundred nodes, adding one moves 1% instead of essentially everything.

Tie that back to the cache arithmetic, because that is where the difference becomes an incident. A tier at a 95% hit rate, resizing from four nodes to five:

• mod-N: new miss rate = 0.80 + 0.20 × 0.05 = 0.81. Origin load jumps 16×, while you were adding capacity.
• Consistent hashing: new miss rate = 0.20 + 0.80 × 0.05 = 0.24. A 4.8× bump — still real, still worth ramping the new node in gradually, but survivable.

Two refinements you should mention unprompted. Virtual nodes: placing each machine at a single ring point gives arc lengths drawn from an exponential distribution, so with ten nodes one of them can easily own three times its fair share. Give every machine 100-200 ring points instead and the imbalance shrinks roughly as 1/√V — around ±10% at V = 100 — and you get weighting for free, since a machine with twice the capacity simply gets twice the points. Bounded-load consistent hashing: cap any node at (1 + ε) times the average load and spill the overflow to the next node clockwise, which fixes the one genuine weakness of the ring, namely that a single very hot key concentrates on a single node forever.

Health checks: active, passive, and how they amplify an outage

Two more health-check details that separate operators from readers. First, probe the instance, not the world: a readiness endpoint that checks the database will fail on every instance simultaneously during a 20-second database blip and drain your entire pool — the outage is then total rather than partial. Second, probing costs something: 200 balancer instances each probing 500 backends every 2 seconds is 50,000 probes/sec of pure overhead. Past a certain fleet size you stop having every proxy probe every backend and move to a control plane that distributes health state (xDS-style push), which is also what makes ejection decisions consistent across the fleet instead of each proxy holding its own private opinion.

Connection draining, and the deploy that drops requests

Removing a backend is not an event, it is a sequence, and skipping a step shows up as a small burst of 502s on every single deploy that nobody ever gets around to fixing.

• 1. Stop being advertised. Fail the readiness probe or deregister from the pool. New requests stop arriving — eventually.
• 2. Wait for that to propagate. This is the step everyone omits. The balancer may not notice for a full health-check interval, and in Kubernetes the SIGTERM and the endpoint removal happen concurrently, so a container that exits promptly on SIGTERM will drop requests that were routed a moment earlier. A preStop sleep of 5-10 seconds before you begin shutting down is the standard fix.
• 3. Drain in-flight work up to a deadline — 30 s for a web tier, longer for uploads. Reject anything new with a clean 503.
• 4. Close keep-alive connections deliberately. Send Connection: close on the last response (or an HTTP/2 GOAWAY), otherwise a client holds an idle socket to a process that is about to vanish and its next request fails.
• 5. Then exit.

Long-lived connections don't drain, they get evicted. Rolling a fleet holding a million WebSockets disconnects a million clients, all of which reconnect immediately and in unison unless the client backs off with jitter — a self-inflicted denial of service that arrives roughly one second after a successful deploy. Roll in small batches, and make jittered reconnect a client requirement, not a hope.

Sticky sessions and why they are a smell

Affinity — by source IP, by a balancer-issued cookie, or by hashing a header — pins a user to one backend. It works, which is the problem: it lets you keep server-side state that you should have externalised, and it converts a stateless tier into a stateful one without anybody deciding to.

• Load stops being balanced. Existing sessions never move, so scaling out during a spike adds instances that receive nothing until sessions churn. Autoscaling is least effective exactly when you need it most.
• Every deploy is a data-loss event. Restarting a backend destroys the sessions, carts and in-memory work of whoever was pinned to it.
• Draining becomes user-visible. You cannot remove a node gracefully when removal means logging its users out.
• Failure stops being graceful. One dead instance out of twenty means 5% of users are fully broken, rather than everyone losing one request that gets retried.
• Source-IP affinity in particular is broken by design. Carrier-grade NAT puts an entire mobile network behind a handful of IPs, and those hash to a handful of backends.

The legitimate uses are narrow and worth naming, so you don't sound dogmatic: a WebSocket or gRPC stream is inherently pinned for its lifetime; a multipart upload buffered on local disk has to finish where it started. And for cache locality — routing the same user to the instance that already has their data warm — the right tool is consistent hashing on the user id, not session affinity, because a ring degrades gracefully when a node dies and rebalances when you add one, whereas affinity does neither. Everything else is solved by putting the session in Redis or in a short-lived signed token, at which point any instance can serve any request and the whole category of problem disappears.

Multi-tier balancing: DNS, global, regional

At scale there is no "the load balancer" — there are four tiers, each choosing at a different granularity and failing over on a different timescale.

One practical detail from the bottom tier that interviewers like: zone-aware routing. Cross-AZ traffic costs real money (roughly one to two cents per GB, charged in both directions) and adds a millisecond or two. Configuring the L7 tier to prefer backends in its own availability zone, and only spill across zones when the local ones are unhealthy or saturated, cuts both. Say it as a cost decision, because it is one, and cost awareness is rare enough in system design interviews to be memorable.

Recognizing it in an unseen problem

• The moment a design has more than one instance of anything, the interviewer is entitled to ask how traffic reaches them. Have a default ready: L4 in front of L7, round robin or power-of-two at the bottom, active plus passive health checks, slow start on new instances.
• Any mention of gRPC, HTTP/2 or long-lived connections between services should make you say "per-request balancing" out loud — an L4 balancer in front of gRPC produces permanent, invisible hotspots and this is a favourite gotcha.
• A naive design draws one load balancer box, never says which layer, and leaves it as an unreplicated single point of failure in front of a carefully replicated everything-else.
• Distinguish this from sharding: load balancing spreads stateless work across interchangeable workers; sharding partitions state across non-interchangeable owners. Consistent hashing shows up in both, which is exactly why people confuse them — name which one you're doing.
• "Users must stay connected to the same server" is a prompt to push back. Ask what state lives there and whether it can be externalised, and only accept affinity for genuinely connection-scoped things.
• Pitfall: treating health checks as a checkbox. If you can describe the detection interval, what the probe does not check, the panic threshold, and what happens to in-flight requests during a deploy, you are answering at a level most candidates never reach.