Designing for fault tolerance

Failure is the steady state, not the exception

In a 10,000-server fleet with a two-year mean time between failures, you lose roughly 14 machines every day. Disk annualised failure rates of 1–2% mean 100–200 dead drives a year per 10,000. Add rack power events, kernel panics, bad deploys, expired certificates, and a dependency's dependency having a bad afternoon — and there is no hour in which everything is healthy. Fault tolerance is not a hardening pass at the end; it is the shape of the design.

The arithmetic that motivates all of it: serial dependencies multiply. A service that must call ten dependencies, each independently available 99.9% of the time, is available 0.999¹⁰ ≈ 99.0% — about 7 hours of downtime a month, built entirely out of "reliable" components. You get availability back only by making dependencies optional, redundant, or bounded.

Timeouts: no timeout is a bug

Every network call must have a deadline. Without one, a hung dependency is converted into thread, connection, and memory exhaustion in your process — you die of someone else's slowness. The defaults are not on your side: many HTTP clients ship with no socket read timeout at all and fall back to OS-level TCP behaviour, which can hold a connection for over two hours.

• Set the timeout from the dependency's healthy p99.9, not from a round number. If a call normally takes 20 ms at p99.9, a 30-second timeout is not a safety margin — it is 1,500× the useful waiting time, and it guarantees you hold resources for half a minute per hung request.
• Propagate deadlines, don't restart them. Five hops each with a fresh 1-second timeout can burn 5 seconds while the user gave up at 2. Pass the remaining budget down (gRPC deadlines, an x-request-deadline header) and have each hop subtract its own elapsed time. If the remaining budget is already less than the dependency's p50, fail immediately rather than starting work that cannot finish.
• Separate connect, read, and total timeouts. A connect timeout should be tight (a few hundred ms — TCP handshakes don't get slower under load, they just fail); a read timeout tracks the work; a total timeout caps the whole thing including retries.
• A timeout is not an error you understand. A timed-out write may still have committed. This is why the next two sections — retries and idempotency — are inseparable.

Retries, backoff, jitter — and how naive retries kill you

Retrying a transient failure is obviously correct and quietly one of the most dangerous things in distributed systems, because retries add load exactly when the system has the least capacity.

Exponential backoff spaces attempts as base × 2ⁿ up to a cap. Jitter is what stops every client from retrying in synchronised waves — without it, a blip trains all your clients onto the same clock and you get a self-inflicted DDoS every 2, 4, 8 seconds. Full jitter is the standard choice and is a one-liner:

// "Full jitter": sleep uniformly in [0, capped backoff). AWS-recommended.
function delayMs(attempt, base = 100, cap = 20000) {
  const window = Math.min(cap, base * 2 ** attempt);
  return Math.random() * window; // spreads a retry wave across the whole window
}

async function callWithRetry(fn, { attempts = 3, budget } = {}) {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      if (!isRetryable(err)) throw err;        // 400s, validation, auth: never retry
      if (i === attempts - 1) throw err;
      if (budget && !budget.tryConsume()) throw err; // token bucket: retries capped at ~10% of traffic
      await sleep(delayMs(i));
    }
  }
}

Three rules that matter more than the backoff formula. Retry only what is retryable — a 400, a validation failure, or an auth error will fail identically every time and burning three attempts on it just adds latency. Retry at one layer only, ideally the one closest to the failure or the outermost edge, never both. And enforce a retry budget: a token bucket that allows retries to be at most ~10% of successful requests, which is how Envoy, gRPC and Finagle bound the amplification. When the dependency is fully down, a budget makes retries stop automatically.

Circuit breakers

Once a dependency is clearly down, continuing to call it is pure harm: you burn your own threads waiting for timeouts, and you keep hammering something that needs quiet to recover. A circuit breaker is a small state machine on the caller side, one per dependency, that converts slow failures into instant ones.

Bulkheads and resource isolation

Named after ship compartments: partition your resources so one flooded compartment doesn't sink the vessel. The failure it prevents is the most common shape of outage — one slow dependency consuming every thread or connection in a shared pool, taking down endpoints that never touched it.

• Per-dependency pools: a separate connection/thread pool per downstream, sized so no single one can exhaust the process.
• Separate critical from non-critical: checkout and recommendations must not share a pool. Ever.
• Cells: partition the whole stack — LB, app, cache, DB — into independent cells serving disjoint customer sets. A bad deploy or poison request takes out one cell, not the fleet. This is how AWS builds most services.
• Shuffle sharding: assign each customer a random subset of workers rather than a single cell. With 100 workers and 5 per customer there are about 75 million distinct combinations, so the chance that any other customer shares all five with a noisy neighbour is vanishingly small — near-total isolation at almost no capacity cost.

Graceful degradation and load shedding

When you cannot serve everything, the choice is between deciding what to drop and letting the system decide randomly — and random means the checkout requests die alongside the avatar thumbnails. Decide in advance, and encode the decision in the request.

Tag every request with a priority class at the edge — critical (payments, auth, writes the user is watching), normal (reads on the main path), bulk (backfills, analytics, prefetch, recommendations) — and propagate it through every hop. Under pressure you shed bulk first, then normal, and only ever fail critical when there is nothing left.

Idempotency: the property that makes retries legal

Every mechanism above depends on being able to retry safely, and you can only retry safely if the operation is idempotent. This is not a detail — it is the load-bearing assumption. A timed-out request may have succeeded, so "retry" and "do it twice" are the same code path from the client's point of view.

• Client-generated idempotency keys. The caller mints a UUID per logical operation and sends it with every attempt. The server stores key → response and returns the stored response on replay. Store the response, not just a "seen" flag, so the retry gets the same order ID rather than a 409.
• Handle the concurrent duplicate. Two retries can race. Insert the key with a unique constraint first, in the same transaction as the effect; the loser of the insert waits and returns the winner's result. A read-then-write check is a race, not a solution.
• Give keys a TTL (Stripe uses 24 hours) and scope them per-customer per-endpoint so a key cannot be replayed against a different operation.
• Prefer natural idempotency where you can get it. "Set balance to X" is idempotent; "add 10 to balance" is not. A unique constraint on (order_id, item_id) makes double-insert a no-op for free.
• Downstream side effects need it too. Sending an email or charging a card twice is the actual customer harm — push the idempotency key all the way to the payment provider, which is exactly why every payment API has one.

Anatomy of a cascading failure, and where to cut it

Cascades follow the same script every time, and each arrow in it is a place you can insert a defence:

Chaos engineering and game days

Every mechanism above is a code path that only executes during an incident, which means it is untested by default — and an untested fallback is usually broken. Chaos engineering is the practice of executing those paths deliberately, while people are watching.

• Hypothesis first. "If we kill one instance in the payments cell, error rate stays under 0.1% and recovery completes in under 60 seconds." A chaos experiment without a predicted outcome is just an outage you caused.
• Smallest blast radius, then widen. One instance, then one AZ, then a region evacuation. Netflix's ladder from Chaos Monkey (kill an instance) to Chaos Kong (evacuate a region) is the model; AWS Fault Injection Service and Gremlin package the same idea.
• Inject latency, not just failure. Slow is harder than dead and far more common — most cascades start with a p99 that quietly went from 20 ms to 2 s.
• Run it in production, during business hours, with an abort button and a named person holding it. Staging does not have your traffic pattern, your cache state, or your on-call rotation.
• Game days test humans as much as systems. Does the alert fire? Does it page the right team? Is the runbook accurate? Can the on-call actually find the dashboard at 3 a.m.? Half the value found in a good game day is organisational, not technical.

Recognizing it in an unseen problem

• The prompt names an availability target ("99.99%", "always available", "handle Black Friday") or mentions money, safety, or regulatory consequences — all of them mean the interviewer wants explicit failure handling, not a happy-path diagram.
• Any arrow you draw between two boxes is a place to state a timeout, a retry policy, and a fallback. Walking the diagram once and annotating each arrow is a strong, structured way to spend five minutes.
• A naive design adds retries everywhere and calls it resilience. The distinguishing question is always "what happens when the dependency is down for ten minutes, not two seconds" — that is where breakers, budgets, and degradation separate from retries.
• Distinguish from replication and failover: those give you redundancy for component failure. This chapter is about overload and correlated failure, where redundancy alone makes things worse because every replica is failing for the same reason.
• Whenever you propose a retry, immediately say how the operation is made idempotent. Retries without idempotency are duplicate charges, and interviewers notice both when you say it and when you don't.
• Close by naming the degraded mode. "If the whole recommendation tier is gone, here is exactly what the user sees" is the answer that demonstrates you have run something in production.