Designing for availability
Availability multiplies down your dependency chain — five nines in one box is four nines in a system.
Availability is arithmetic, and the arithmetic is unforgiving
Every candidate says "I'll make it highly available." Almost nobody can say what that costs. Availability is a measured number — the fraction of requests (or of wall-clock time) during which the system did the thing it promised — and the single most valuable fact about it is that it multiplies down a serial dependency chain. Your service cannot be more available than the product of everything it must reach to answer a request. Show that multiplication on the whiteboard unprompted and you have signalled seniority in about fifteen seconds.
Include your own tier in that product and five hard dependencies at three nines put you at 99.40% — about 52 hours a year. Notice what that means: you cannot buy availability you did not design for. If the target is four nines end-to-end, either every hop is five nines (expensive) or most hops stop being hard dependencies (design work).
What the nines actually cost you in wall-clock time
| Availability | Downtime / year | Downtime / month | What it realistically implies |
|---|---|---|---|
| 99% | 3.65 days | 7.3 hours | One box, business-hours on-call, deploys cause outages |
| 99.9% ("three nines") | 8.8 hours | 43.8 minutes | Redundant instances, automated deploys, someone paged 24/7 |
| 99.95% | 4.4 hours | 21.9 minutes | Multi-AZ, health-checked failover, tested rollbacks |
| 99.99% ("four nines") | 52.6 minutes | 4.4 minutes | No manual step in the recovery path — humans cannot respond that fast |
| 99.999% ("five nines") | 5.3 minutes | 26 seconds | Multi-region active-active, no dependency below five nines, very few systems truly have it |
Read the four-nines row again. 52 minutes a year is your entire budget for every bad deploy, every dependency incident, every certificate expiry and every disk failure combined. A single human paging in, opening a laptop and finding the runbook has already spent a quarter of the annual budget. That is the real threshold: at four nines and above, recovery has to be automatic.
Redundancy: active-active vs active-passive
Redundancy is the only actual mechanism for availability; everything else is plumbing around it. The design question is whether the spare capacity is serving traffic right now.
| Active-active | Active-passive | |
|---|---|---|
| Failure behaviour | Remaining nodes absorb the load — no transition | Detect, promote, redirect — a transition that can fail |
| Recovery time | Effectively zero (a few in-flight requests) | Detection window + promotion + DNS/connection churn |
| Cost | You pay for N+1 but you use it | You pay for the standby and get nothing back |
| Hard part | Every node writes: conflicts, split state, sticky sessions | Keeping the standby warm, current, and actually working |
| Reach for this when… | The work is stateless or the store already does multi-writer (Cassandra, Dynamo-style) | There is a single logical writer — a relational primary, a leader-elected coordinator |
Active-active is not simply "better". Two nodes both accepting writes to the same row is the whole consistency problem from the consistency-models chapter, showing up in your availability design. Most real systems are active-active at the stateless tier and active-passive (or quorum-based) at the storage tier, and saying exactly that is the correct answer.
Failover, and the three ways it betrays you
Failover is the moment a system decides a component is dead and acts on that belief. The belief can be wrong, and the action can be worse than the fault.
- Split brain. A partition, not a crash. Both replicas promote and accept writes; when the network heals you have two divergent histories and no principled way to merge them. The fix is not a smarter heartbeat — it is quorum: an odd number of voters, and a node refuses to serve as primary unless it can see a majority. A two-node cluster cannot do this, which is why "two nodes for HA" is often worse than one.
- Failback storms. The recovered node comes back, is declared healthy, and instantly receives its full share of traffic — with cold caches, cold connection pools and cold JIT. It falls over, gets marked unhealthy, and the loop repeats while the healthy nodes absorb the oscillation. The fix is slow-start / connection ramping: bring a returning node up to full weight over 30-120 seconds, and require it to pass health checks for a sustained window, not a single probe.
- Retry amplification. One dependency slows down, every caller retries three times, and the dependency now receives 4× the traffic at exactly the moment it can least handle it. Retries need exponential backoff with jitter, a retry budget (e.g. retries capped at 10% of requests), and a circuit breaker that stops calling a failing dependency entirely rather than politely queueing for it.
Health checks are a design decision, not a checkbox
A health check is the sensor your entire availability story depends on, and most candidates specify it in three words. There are two distinct questions and they need different endpoints.
| Probe | Asks | Failure action | Must NOT check |
|---|---|---|---|
| Liveness | Is this process wedged and unrecoverable? | Restart the instance | Dependencies — a DB outage would restart your whole fleet |
| Readiness | Can this instance serve a request right now? | Remove from the load-balancer pool | Anything slow — the probe itself must be cheap |
| Deep / synthetic | Does a real user journey still work end to end? | Page a human, drive dashboards | Nothing — this one is allowed to be expensive, run it from outside |
Graceful degradation: shrinking the blast radius on purpose
The most leverage in availability design is not making dependencies more reliable — it is making fewer of them hard. Every dependency you can convert from "request fails without it" to "page renders with a slightly worse experience" is removed from the multiplication.
| Dependency down | Naive result | Degraded result |
|---|---|---|
| Recommendation service | 500 on the home page | Serve a cached or globally-popular list; page still renders |
| Personalized ranking | Empty feed | Fall back to reverse-chronological |
| Search cluster | Search page errors | Fall back to a prefix lookup on the primary store, or show recent items |
| Cache tier | Origin collapses under 100% miss rate | Shed load: serve a smaller page, admit a fraction of traffic, keep the origin alive |
| Payments | Checkout unavailable | Accept the order into a queue, confirm asynchronously — only if the business allows it |
Degradation only works with aggressive timeouts. A dependency that hangs for 30 seconds is worse than one that fails in 50 ms, because the hang consumes your own threads, connections and memory until you fall over too. Set every outbound call a timeout meaningfully tighter than your own SLO, and treat "no timeout configured" as a bug.
Hunting single points of failure
A single point of failure is any component whose loss takes the system with it. They are rarely the obvious boxes — nobody forgets to replicate the database. They hide in the parts of the diagram people don't draw.
- The load balancer itself. Redundant app servers behind one LB instance is a SPOF with extra steps. Managed LBs are already redundant; self-managed ones need a floating IP or DNS-level failover.
- DNS and TLS certificates. An expired cert is a total outage that no amount of replication prevents. So is a single authoritative DNS provider.
- Shared configuration and feature flags. If every instance fetches config at startup from one service, that service is a SPOF for every deploy and every autoscale event. Cache config locally and serve stale on failure.
- The deployment pipeline. If you cannot roll back because CI is down, your recovery time is now bounded by someone else's uptime.
- Correlated failure. Three replicas in one rack, one AZ, or on one storage volume are one failure domain wearing a costume. Ask "what is the smallest event that kills all N of these at once?"
- Shared state you forgot is shared. One Redis holding sessions for a stateless fleet makes that fleet stateful. One shared connection pool, one shared secret store, one leader-election service.
Multi-AZ vs multi-region, honestly costed
| Multi-AZ | Multi-region | |
|---|---|---|
| Protects against | Rack, power, cooling, one datacenter | Regional outage, regional network, fibre cut, regulatory isolation |
| Inter-node latency | ~1-2 ms — synchronous replication is fine | 30-150 ms cross-continent — synchronous replication is not fine |
| Data cost | Cross-AZ transfer, roughly 1-2 cents per GB | Cross-region transfer plus a full second copy of everything |
| Consistency impact | Essentially none; you keep a single primary | You must choose: single writer with slow remote writes, or multi-writer with conflict resolution |
| Realistic ceiling | 99.95-99.99% | 99.99%+ — and only if failover is genuinely automatic |
| Reach for this when… | Almost always — it is close to free and the default answer | The business loses serious money per minute of downtime, or law requires data residency |
Be blunt about the cost of multi-region: it is not "deploy twice." It is a second copy of your data with a replication strategy, a global traffic routing layer, a story for cross-region consistency, and an organisation-wide discipline that every new service is region-aware from day one. Teams frequently build it and then discover their failover has never been tested, giving them all the cost and none of the availability. Untested failover is decoration. Say you would run scheduled region-evacuation drills — that one sentence separates people who have operated multi-region from people who have read about it.
SLA, SLO, SLI and the error budget
| Term | What it is | Audience | Example |
|---|---|---|---|
| SLI | The measurement itself | Engineers | Fraction of requests returning 2xx/3xx in under 300 ms, per minute |
| SLO | The internal target for that SLI | Your team | 99.95% of those requests, measured over 28 rolling days |
| SLA | The contractual promise, with a penalty | Customers, lawyers | 99.9%, or service credits are owed |
The SLA is always looser than the SLO — deliberately. You want to be breaching your internal target and fixing it long before you owe anyone money. If your SLO equals your SLA you have no warning zone.
The error budget is the inverse of the SLO, and it is the most useful idea here because it turns reliability from an argument into a number. A 99.95% SLO over 28 days permits about 20 minutes of failure. That budget is a resource: spend it on risky deploys, experiments and migrations. The operating rule is mechanical — budget remaining, ship features; budget exhausted, feature work stops and reliability work starts until the rolling window recovers. It also stops the opposite failure mode: if you finish every month with 100% of the budget unspent, you are over-invested in reliability and shipping too slowly.
Recognizing it in an unseen problem
- The prompt contains a reliability number ("99.99%", "always available", "cannot go down during Black Friday") — that is an invitation to do the dependency multiplication out loud before designing anything.
- A naive design adds a second server, calls it highly available, and never mentions how failure is detected, how long detection takes, or what happens when the two servers disagree.
- Distinguish from fault tolerance (the advanced sibling chapter): availability is about staying up during expected failures; fault tolerance is about correctness and recovery when things fail in unexpected, partial and byzantine ways.
- Distinguish from consistency: if the prompt stresses "users must never see stale data", you are being asked a CAP question, not an availability question — and the two pull in opposite directions during a partition.
- Anything with a hard dependency on a third party you don't control (payments, mapping, SMS) should trigger the degradation conversation: what does the product do when that vendor is down for 20 minutes?
- Pitfall: treating redundancy as sufficient. Redundancy without automated, tested, ramped failover raises your cost and leaves your availability roughly where it was.