Designing for availability

Availability is arithmetic, and the arithmetic is unforgiving

Every candidate says "I'll make it highly available." Almost nobody can say what that costs. Availability is a measured number — the fraction of requests (or of wall-clock time) during which the system did the thing it promised — and the single most valuable fact about it is that it multiplies down a serial dependency chain. Your service cannot be more available than the product of everything it must reach to answer a request. Show that multiplication on the whiteboard unprompted and you have signalled seniority in about fifteen seconds.

Dry run: adding one 99.9% dependency at a time

(Downtime figures use a 365.25-day year, 8,766 hours.) Each additional hard dependency costs roughly another 8-9 hours a year at three nines, regardless of which one it is — the multiplication doesn't care about names. Step 6 is the only row that recovers availability, and it does it without touching a single server: converting payments from a hard dependency to an optional one (queue the order, confirm asynchronously) simply removes its factor from the product.

Include your own tier in that product and five hard dependencies at three nines put you at 99.40% — about 52 hours a year. Notice what that means: you cannot buy availability you did not design for. If the target is four nines end-to-end, either every hop is five nines (expensive) or most hops stop being hard dependencies (design work).

The parallel case runs the other way: if any one of n redundant copies is enough, availability is 1 − (1 − a)ⁿ, so two independent 99.9% copies give 99.9999% on paper. The caveat is the word independent. Copies that share a deploy pipeline, a config push, a region, a dependency or a bug fail together, so the real figure is far lower than the formula, which is why shared-fate failures dominate real outages. Availability is also MTBF / (MTBF + MTTR) (mean time between failures over that plus mean time to repair), which shows the two levers: fail less often, or recover faster. Automatic recovery attacks MTTR, which is usually the cheaper lever.

What the nines actually cost you in wall-clock time

Read the four-nines row again. 52 minutes a year is your entire budget for every bad deploy, every dependency incident, every certificate expiry and every disk failure combined. A single human paging in, opening a laptop and finding the runbook has already spent a quarter of the annual budget. That is the real threshold: at four nines and above, recovery has to be automatic.

Redundancy: active-active vs active-passive

Redundancy is the only actual mechanism for availability; everything else is plumbing around it. The design question is whether the spare capacity is serving traffic right now.

Active-active is not simply "better". Two nodes both accepting writes to the same row is the whole consistency problem from the consistency-models chapter, showing up in your availability design. Most real systems are active-active at the stateless tier and active-passive (or quorum-based) at the storage tier, and saying exactly that is the correct answer.

Failover, and the three ways it betrays you

Failover is the moment a system decides a component is dead and acts on that belief. The belief can be wrong, and the action can be worse than the fault.

• Split brain. A partition, not a crash. Both replicas promote and accept writes; when the network heals you have two divergent histories and no principled way to merge them. The fix is not a smarter heartbeat — it is quorum: an odd number of voters, and a node refuses to serve as primary unless it can see a majority. A two-node cluster cannot do this: if the rule is "a majority must agree", losing either node stops the cluster, so it is less available than one node (and without a quorum rule it split-brains instead). That is why "two nodes for HA" is often worse than one.
• Failback storms. The recovered node comes back, is declared healthy, and instantly receives its full share of traffic — with cold caches, cold connection pools and cold JIT. It falls over, gets marked unhealthy, and the loop repeats while the healthy nodes absorb the oscillation. The fix is slow-start / connection ramping: bring a returning node up to full weight over 30-120 seconds, and require it to pass health checks for a sustained window, not a single probe.
• Retry amplification. One dependency slows down and every caller retries; with 3 retries (4 attempts) the dependency now receives up to 4× the traffic at exactly the moment it can least handle it. Across layers it compounds: if three stacked layers each make up to 3 attempts (2 retries), one user action becomes 3³ = 27 attempts at the bottom, and the Google SRE book's example of 3 retries per layer over three layers is 4³ = 64. Retries need exponential backoff with jitter, a retry budget (e.g. retries capped at 10% of requests), and a circuit breaker that stops calling a failing dependency entirely rather than politely queueing for it.

Health checks are a design decision, not a checkbox

A health check is the sensor your entire availability story depends on, and most candidates specify it in three words. There are two distinct questions and they need different endpoints.

Graceful degradation: shrinking the blast radius on purpose

The most leverage in availability design is not making dependencies more reliable — it is making fewer of them hard. Every dependency you can convert from "request fails without it" to "page renders with a slightly worse experience" is removed from the multiplication.

Degradation only works with aggressive timeouts. A dependency that hangs for 30 seconds is worse than one that fails in 50 ms, because the hang consumes your own threads, connections and memory until you fall over too. Set every outbound call a timeout meaningfully tighter than your own SLO, and treat "no timeout configured" as a bug.

Hunting single points of failure

A single point of failure is any component whose loss takes the system with it. They are rarely the obvious boxes — nobody forgets to replicate the database. They hide in the parts of the diagram people don't draw.

• The load balancer itself. Redundant app servers behind one LB instance is a SPOF with extra steps. Managed LBs are already redundant; self-managed ones need a floating IP or DNS-level failover.
• DNS and TLS certificates. An expired cert is a total outage that no amount of replication prevents. So is a single authoritative DNS provider.
• Shared configuration and feature flags. If every instance fetches config at startup from one service, that service is a SPOF for every deploy and every autoscale event. Cache config locally and serve stale on failure.
• The deployment pipeline. If you cannot roll back because CI is down, your recovery time is now bounded by someone else's uptime.
• Correlated failure. Three replicas in one rack, one AZ, or on one storage volume are one failure domain wearing a costume. Ask "what is the smallest event that kills all N of these at once?"
• Shared state you forgot is shared. One Redis holding sessions for a stateless fleet makes that fleet stateful. One shared connection pool, one shared secret store, one leader-election service.

Multi-AZ vs multi-region, honestly costed

Be blunt about the cost of multi-region: it is not "deploy twice." It is a second copy of your data with a replication strategy, a global traffic routing layer, a story for cross-region consistency, and an organisation-wide discipline that every new service is region-aware from day one. Teams frequently build it and then discover their failover has never been tested, giving them all the cost and none of the availability. Untested failover is decoration. Say you would run scheduled region-evacuation drills — that one sentence separates people who have operated multi-region from people who have read about it.

Limiting blast radius and recovering from disaster

Redundancy handles the failures you expected. These techniques limit the damage from the ones you did not.

• Bulkheads. Give each dependency or customer tier its own thread pool, connection pool or queue, so a slow payments call exhausts only the payments pool, not every worker.
• Load shedding and deadlines. Reject excess work early and cheaply (a 503 beats a 30-second queue), and propagate one absolute deadline down the call tree so no hop works on a request whose caller has already given up. Both are described in the Google SRE book chapter on cascading failures.
• Cells. Partition the fleet into independent cells that each serve a slice of users with their own copy of the stack. A bad deploy or poisoned input then takes out one cell, say 5% of users, instead of everyone.
• Safe change. Most outages follow a change, so ship through canaries and gradual rollouts with automatic rollback, and rehearse failure with game days and chaos experiments.
• Measure and learn. Watch the four golden signals (latency, traffic, errors, saturation) and run blameless postmortems, so the fix targets the system and not the person.

For regional disaster recovery, two numbers set the design: the recovery point objective (how much data you can lose) and the recovery time objective (how long you can be down). AWS's disaster recovery guidance describes four tiers in rising cost and falling recovery time: backup and restore (restore from copies in another region, hours), pilot light (data replicated and databases running, with the application servers defined but switched off), warm standby (a scaled-down but fully running copy that can take traffic right away), and multi-site active-active (every region serves traffic). Pick the cheapest tier that meets the RTO and RPO the business signed up for, and test the restore; an untested backup is a hope.

SLA, SLO, SLI and the error budget

The SLA is always looser than the SLO — deliberately. You want to be breaching your internal target and fixing it long before you owe anyone money. If your SLO equals your SLA you have no warning zone.

The error budget (a Google SRE idea) is the inverse of the SLO, and it is the most useful idea here because it turns reliability from an argument into a number. A 99.95% SLO over 28 days permits about 20 minutes of failure. That budget is a resource: spend it on risky deploys, experiments and migrations. The operating rule is mechanical — budget remaining, ship features; budget exhausted, feature work stops and reliability work starts until the rolling window recovers. It also stops the opposite failure mode: if you finish every month with 100% of the budget unspent, you are over-invested in reliability and shipping too slowly.

Recognizing it in an unseen problem

• The prompt contains a reliability number ("99.99%", "always available", "cannot go down during Black Friday") — that is an invitation to do the dependency multiplication out loud before designing anything.
• A naive design adds a second server, calls it highly available, and never mentions how failure is detected, how long detection takes, or what happens when the two servers disagree.
• Distinguish from fault tolerance (the advanced sibling chapter): availability is about staying up during expected failures; fault tolerance is about correctness and recovery when things fail in unexpected, partial and byzantine ways.
• Distinguish from consistency: if the prompt stresses "users must never see stale data", you are being asked a CAP question, not an availability question — and the two pull in opposite directions during a partition.
• Anything with a hard dependency on a third party you don't control (payments, mapping, SMS) should trigger the degradation conversation: what does the product do when that vendor is down for 20 minutes?
• Pitfall: treating redundancy as sufficient. Redundancy without automated, tested, ramped failover raises your cost and leaves your availability roughly where it was.