Designing for fault tolerance

Failure is the steady state, not the exception

In a 10,000-server fleet with a two-year mean time between failures, you lose roughly 14 machines every day. Disk annualised failure rates of 1–2% mean 100–200 dead drives a year per 10,000. Add rack power events, kernel panics, bad deploys, expired certificates, and a dependency's dependency having a bad afternoon — and there is no hour in which everything is healthy. Fault tolerance is not a hardening pass at the end; it is the shape of the design.

The arithmetic that motivates all of it: serial dependencies multiply. A service that must call ten dependencies, each independently available 99.9% of the time, is available 0.999¹⁰ ≈ 99.0% — about 7 hours of downtime a month, built entirely out of "reliable" components. You get availability back only by making dependencies optional, redundant, or bounded.

Planning figure: one well-run region with multi-AZ lands somewhere around 99.95–99.99%, depending on what it shares (a single database, a control plane, a config pipeline). Multi-region active-active can exceed that but costs far more, and many outages are caused by a bad deploy or config push, which a second region simply receives too.

Timeouts: no timeout is a bug

Every network call must have a deadline. Without one, a hung dependency is converted into thread, connection, and memory exhaustion in your process — you die of someone else's slowness. The defaults are not on your side: many HTTP clients ship with no socket read timeout at all and fall back to OS-level TCP behaviour. On Linux that means a blocked read on a silent connection can wait indefinitely unless keepalive is on, and keepalive does not even start probing until the connection has been idle for 2 hours (the 7,200 s default). If the peer vanishes while you have unacknowledged data in flight, the kernel retransmits for about 15 minutes (tcp_retries2 = 15, roughly 925 s) before giving up. Either way your thread is parked far longer than any user will wait.

• Set the timeout from the dependency's healthy p99.9, not from a round number. If a call normally takes 20 ms at p99.9, a 30-second timeout is not a safety margin — it is 1,500× the useful waiting time, and it guarantees you hold resources for half a minute per hung request.
• Propagate deadlines, don't restart them. Five hops each with a fresh 1-second timeout can burn 5 seconds while the user gave up at 2. Pass the remaining budget down (gRPC deadlines, an x-request-deadline header) and have each hop subtract its own elapsed time. If the remaining budget is already less than the dependency's p50, fail immediately rather than starting work that cannot finish.
• Separate connect, read, and total timeouts. A connect timeout should be tight (a few hundred ms: a handshake to a healthy host in the same region takes about one round trip, and under SYN-queue pressure a lost SYN is only retried by the kernel after about 1 s, then 3 s, so waiting longer just means waiting for a host that is not coming back — fail over to another instance instead); a read timeout tracks the work; a total timeout caps the whole thing including retries.
• A timeout is not an error you understand. A timed-out write may still have committed. This is why the next two sections — retries and idempotency — are inseparable.

Retries, backoff, jitter — and how naive retries kill you

Retrying a transient failure is obviously correct and quietly one of the most dangerous things in distributed systems, because retries add load exactly when the system has the least capacity.

Exponential backoff spaces attempts as base × 2ⁿ up to a cap. Jitter is what stops every client from retrying in synchronised waves — without it, a blip trains all your clients onto the same clock and you get a self-inflicted DDoS every 2, 4, 8 seconds. Full jitter is the standard choice and is a one-liner:

// "Full jitter": sleep uniformly in [0, capped backoff). AWS-recommended.
function delayMs(attempt, base = 100, cap = 20000) {
  const window = Math.min(cap, base * 2 ** attempt);
  return Math.random() * window; // spreads a retry wave across the whole window
}

async function callWithRetry(fn, { attempts = 3, budget } = {}) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (!isRetryable(err)) throw err;        // 400s, validation, auth: never retry
      if (i === attempts - 1) throw err;
      if (budget && !budget.tryConsume()) throw err; // token bucket: retries capped at ~10% of traffic
      await sleep(delayMs(i));
    }
  }
}

Three rules that matter more than the backoff formula. Retry only what is retryable — a 400, a validation failure, or an auth error will fail identically every time and burning three attempts on it just adds latency. Retry at one layer only: AWS's guidance is to retry "at a single point in the stack", and every other layer fails fast and passes the error up. Retrying next to the failing dependency redoes the least work; retrying at the edge is simplest when the whole operation is idempotent. Either is defensible — the rule is to pick one and say which. And enforce a retry budget: a token bucket that allows retries to be only a fraction of successful requests. The numbers differ by system: the Google SRE book uses a per-client budget of 10% (plus at most three attempts per request), gRPC's retry-throttling example uses a token ratio of 0.1, Envoy's retry budget defaults to 20%, and Finagle's to 20% plus 10 retries a second. When the dependency is fully down, a budget makes retries stop automatically.

Circuit breakers

Once a dependency is clearly down, continuing to call it is pure harm: you burn your own threads waiting for timeouts, and you keep hammering something that needs quiet to recover. A circuit breaker is a small state machine on the caller side, one per dependency, that converts slow failures into instant ones.

The thresholds drawn here (50% errors, a minimum of 20 calls, a 10-second window, a 5-second cooldown) are the defaults of Netflix Hystrix, which has been in maintenance mode since 2018. Its recommended successor, Resilience4j, defaults differently: a 100-call window with a minimum of 100 calls, a 60-second wait in the open state, and 10 permitted calls in half-open. Treat all of these as starting points to tune per dependency, not as constants.

Dry run: a circuit breaker tripping, cooling down, and probing back open

Trip threshold >50% errors over a rolling 10s window, minimum 20 calls; 5s cooldown; 1 probe in half-open. A rolling window trips the moment it crosses the threshold, not on a round-number boundary.

The load-bearing detail is in the row at t=17-22s: the breaker frees every caller's thread in microseconds instead of letting five more calls each burn a full timeout against a dependency that was already failing more than half the time — that gap is the entire value of "fail instantly" over "fail slowly."

Bulkheads and resource isolation

Named after ship compartments: partition your resources so one flooded compartment doesn't sink the vessel. The failure it prevents is the most common shape of outage — one slow dependency consuming every thread or connection in a shared pool, taking down endpoints that never touched it.

• Per-dependency pools: a separate connection/thread pool per downstream, sized so no single one can exhaust the process.
• Separate critical from non-critical: checkout and recommendations must not share a pool. Ever.
• Cells: partition the whole stack — LB, app, cache, DB — into independent cells serving disjoint customer sets. A bad deploy or poison request takes out one cell, not the fleet. AWS documents this as cell-based architecture in its Well-Architected guidance.
• Shuffle sharding: assign each customer a random subset of workers rather than a single cell. With 100 workers and 5 per customer there are about 75 million distinct combinations, so the chance that any other customer shares all five with a noisy neighbour is vanishingly small — near-total isolation at almost no capacity cost.

Graceful degradation and load shedding

When you cannot serve everything, the choice is between deciding what to drop and letting the system decide randomly — and random means the checkout requests die alongside the avatar thumbnails. Decide in advance, and encode the decision in the request.

Tag every request with a priority class at the edge — critical (payments, auth, writes the user is watching), normal (reads on the main path), bulk (backfills, analytics, prefetch, recommendations) — and propagate it through every hop. Under pressure you shed bulk first, then normal, and only ever fail critical when there is nothing left. Google's internal version has four named classes (critical-plus, critical, sheddable-plus, sheddable), and backends reject the lowest class first.

Idempotency: the property that makes retries legal

Every mechanism above depends on being able to retry safely, and you can only retry safely if the operation is idempotent. This is not a detail — it is the load-bearing assumption. A timed-out request may have succeeded, so "retry" and "do it twice" are the same code path from the client's point of view.

• Client-generated idempotency keys. The caller mints a UUID per logical operation and sends it with every attempt. The server stores key → response and returns the stored response on replay. Store the response, not just a "seen" flag, so the retry gets the same order ID rather than a 409.
• Handle the concurrent duplicate. Two retries can race. Insert the key with a unique constraint first, in the same transaction as the effect; the loser of the insert either waits and returns the winner's result or fails fast with a retryable conflict (Stripe returns an error for a request that conflicts with one still executing). A read-then-write check is a race, not a solution.
• Give keys a TTL (Stripe's v1 API lets keys be pruned after 24 hours) and scope them per-customer per-endpoint so a key cannot be replayed against a different operation. Stripe also stores the first result including a 500 and replays it, and returns an error if a retry arrives with different parameters under the same key. The Idempotency-Key HTTP header is still only an expired IETF Internet-Draft, not an RFC, so every API defines its own rules; read them.
• Prefer natural idempotency where you can get it. "Set balance to X" is idempotent; "add 10 to balance" is not. A unique constraint on (order_id, item_id) makes double-insert a no-op for free.
• Downstream side effects need it too. Sending an email or charging a card twice is the actual customer harm — push the idempotency key all the way to the payment provider, which is exactly why every payment API has one.

Anatomy of a cascading failure, and where to cut it

Cascades follow the same script every time, and each arrow in it is a place you can insert a defence:

Failover has its own failure modes

Redundancy brings a new question: who is in charge now? A leader that is slow, paused by a long garbage collection, or cut off by a network partition looks dead to everyone else, so a replacement is promoted — and then the old leader wakes up still believing it is the leader. That is split brain. The standard defence is a fencing token: every promotion hands out a monotonically increasing number, and the storage or downstream service rejects any write carrying an older one. Failover also needs numbers agreed in advance: the RPO (how much recent data you can afford to lose) and the RTO (how long you can be down) decide whether you need synchronous replication, a hot standby, or just backups you have actually restored. Finally, make shutdown graceful too: stop accepting new requests, finish in-flight ones, then exit, so a routine deploy does not look like a failure to every caller.

Chaos engineering and game days

Every mechanism above is a code path that only executes during an incident, which means it is untested by default — and an untested fallback is usually broken. Chaos engineering is the practice of executing those paths deliberately, while people are watching.

• Hypothesis first. "If we kill one instance in the payments cell, error rate stays under 0.1% and recovery completes in under 60 seconds." A chaos experiment without a predicted outcome is just an outage you caused.
• Smallest blast radius, then widen. One instance, then one AZ, then a region evacuation. Netflix's ladder from Chaos Monkey (kill an instance) to Chaos Kong (evacuate a region) is the model (the wider Simian Army project is retired; Chaos Monkey continues as a standalone service configured through Spinnaker); AWS Fault Injection Service and Gremlin package the same idea.
• Inject latency, not just failure. Slow is harder than dead and far more common — most cascades start with a p99 that quietly went from 20 ms to 2 s.
• Run it in production, during business hours, with an abort button and a named person holding it. Staging does not have your traffic pattern, your cache state, or your on-call rotation.
• Game days test humans as much as systems. Does the alert fire? Does it page the right team? Is the runbook accurate? Can the on-call actually find the dashboard at 3 a.m.? Half the value found in a good game day is organisational, not technical.

Recognizing it in an unseen problem

• The prompt names an availability target ("99.99%", "always available", "handle Black Friday") or mentions money, safety, or regulatory consequences — all of them mean the interviewer wants explicit failure handling, not a happy-path diagram.
• Any arrow you draw between two boxes is a place to state a timeout, a retry policy, and a fallback. Walking the diagram once and annotating each arrow is a strong, structured way to spend five minutes.
• A naive design adds retries everywhere and calls it resilience. The distinguishing question is always "what happens when the dependency is down for ten minutes, not two seconds" — that is where breakers, budgets, and degradation separate from retries.
• Distinguish from replication and failover: those give you redundancy for component failure. This chapter is about overload and correlated failure, where redundancy alone makes things worse because every replica is failing for the same reason.
• Whenever you propose a retry, immediately say how the operation is made idempotent. Retries without idempotency are duplicate charges, and interviewers notice both when you say it and when you don't.
• Close by naming the degraded mode. "If the whole recommendation tier is gone, here is exactly what the user sees" is the answer that demonstrates you have run something in production.