Release safety and incident response

Most outages are a change someone made

Google's SRE book states it plainly: roughly 70% of outages are due to changes in a live system. Hardware fails and traffic spikes, but the most common cause of a bad day is a deploy, a configuration push, a schema migration or a flag flip. That makes safe change a reliability feature in its own right, and it is the answer to the question after "how do you keep it up?", which is "and how do you change it without taking it down?"

The unit to think about is blast radius: how many users a bad change touches before you notice, and how fast you can undo it. Every technique in this chapter either shrinks the first number or speeds up the second. Note what counts as a change: a new binary is only one kind. A config file, a certificate, a database migration, a dependency upgrade and a feature flag all change production behaviour, and a config push often reaches every server at once, so it deserves the same staged rollout as code.

Deploy is not release

Deploying puts new code on servers. Releasing makes users see it. Keeping them separate is the most useful trick in the chapter. Ship the code dark behind a feature flag, turn it on for staff, then one percent, then everyone, and turn it off in seconds if it misbehaves. Fowler's catalogue of flags names four jobs worth distinguishing: release flags hide unfinished work, experiment flags split users for an A/B test, ops flags are kill switches an operator flips during an incident, and permission flags expose features to some users.

Rollout strategies compared

They combine. A common production shape is waves of regions, a canary inside the first wave, and a rolling replacement inside each region. A shadow run suits a read path, where a bug costs little (the mirrored reads still add load, warm caches and touch rate-limited dependencies); on a write path a mirrored copy with real side effects would charge cards and send emails, so you mirror only with its side effects stubbed out.

Canary analysis: what to compare, and for how long

The Google SRE Workbook's guidance on canaries reads like a checklist. Compare the canary with a control running at the same time on the same traffic, not with "before the deploy", because the world changes every hour. Start with a handful of metrics tied to the user-visible SLIs: error rate and latency first, since CPU can rise without hurting anyone. Leave client-error status codes out or watch them with black-box probes, because a 404 can be a user's typo. Look at the canary and the control separately, because the aggregate can hide a failure in the small group. Add an absolute check against the SLO as well, since canary and control can degrade together. Keep the metric window no longer than the canary, run one canary at a time, and make it large and long enough to be representative: performance bugs often show only under peak load, so a canary at 3 a.m. proves little.

Dry run: what a canary saves in error budget

Take a 99.9% availability objective over 30 days. The error budget is 0.1% of 30 days, which is 0.001 × 30 × 24 × 60 = 43.2 full-outage minutes. A bad build fails 20% of the requests it serves. At 1,000 requests a second, a 5% canary sees 50 a second, 3,000 a minute.

The detection times are assumptions, so look at the third row: even if both rollouts were caught in ten minutes, the canary costs about twenty times less, because the share of traffic is the lever. What a canary adds for noticing is a control group. On 3,000 canary requests a minute, 20% failure means 600 errors against about 3 expected from the 0.1% baseline, and the comparison against the old version on the same traffic says the new build caused it. A subtler regression of 0.2 percentage points means 9 errors a minute against 3. That is already unlikely by chance, but once you run many comparisons across many metrics one minute is borderline, so you wait several minutes (ten minutes gives 90 against 30) before you trust it. Small canaries are cheap, and the smaller the regression you want to catch the longer you wait.

Rollback is a feature you have to design

"We can roll back" is true only if the old version still works with whatever the new version left behind. Three places that goes wrong, and the standard defences:

• New data formats. Amazon's guidance is a two-step release. First deploy a version that can read the new format but still writes the old one; rolling that back is safe because no new data exists. Wait until every host has finished that step and a bake period has passed (Amazon says usually a few days, long enough to be sure you will not go back to the old version), then deploy the version that writes the new format. Readers go before writers going forward, and writers are removed before readers going back.
• Database schema. Use parallel change (also called expand and contract): add the new column, deploy code that writes both, backfill, switch reads, stop writing the old one, and only then drop it in a later release. Never rename or drop in the same release as the code that stops using the old name. A destructive migration cannot be rolled back, so the real plan is rolling forward with a fix.
• Configuration. Version it, validate it before it ships, keep the last known good copy, push it in stages like code, and make the service refuse a bad file and keep running with the old one instead of crashing at startup.

Gates, bake time and automatic rollback

A human watching a dashboard is not a control. Mature pipelines make the checks mechanical: deploy to one box or one small cell, wait a bake time while alarms watch the user-visible signals, promote to the next wave only if they stay quiet, and roll back automatically when one fires. Amazon describes pipelines of this shape, with waves ordered so that the least risky or smallest cells go first. Two practical rules follow. Do not promote on the clock alone, because a bug that appears only at the nightly peak survives a 30-minute bake. And keep the rollback path as exercised as the forward path: a rollback you have never run is a hope.

Connect this to the error-budget policy from the availability chapter. When the budget for the period is spent, the standard response is to pause feature releases and spend the time on reliability work, which turns "ship safely" from advice into a rule the team agreed in advance.

When it breaks anyway: incident response

Some bad changes get through. What decides the length of the outage is the first few minutes, and the rule that shortens them is mitigate first, diagnose later. Roll back, turn the flag off, drain the bad region or shed load; understanding the root cause can wait until users are no longer being hurt.

• Clear roles. Borrowed from emergency services, the SRE book's incident management uses an incident commander who coordinates and decides, an operations lead who is the only one changing the system, and a communications lead who updates stakeholders, with a planning role for large incidents. One person in charge prevents ten engineers pushing ten fixes at once.
• One channel and a running log. Everyone watches the same place, and decisions are written down as they happen, which is also the raw material of the postmortem.
• Severity levels set who is paged and how often updates go out, so a minor issue does not wake the executives and a major one does not wait for someone to notice.
• Hand off cleanly on a long incident, so the person who has been awake for eight hours is not the one making the next call.

Blameless postmortems

After the incident, the goal is that this class of failure cannot recur, not that someone is punished. The SRE book's postmortem chapter is firm on blamelessness: if people expect to be blamed, they hide the facts, and the next outage is worse. Treat "human error" as where the analysis starts, not where it ends (a view from the wider safety literature rather than from the SRE book): the question is why the system let one slip become an outage. The SRE book's example postmortem shows the shape, and a good one has a short summary and impact, a timeline, the contributing causes, what went well, what went badly, and action items with an owner and a priority, tracked to completion. An action list nobody reads is the failure mode, so review the previous ones in the next postmortem.

Recognizing it in an unseen problem

• Signals: "deploy without downtime", "zero-downtime migration", "how do you roll this out", "what if the new version is buggy", or any change to a schema, a format or an API that has live clients.
• The naive design pushes to every server at once and relies on a redeploy to undo. Say the blast radius and the rollback time before the interviewer asks.
• Pick the strategy for the risk: canary for user-facing behaviour, rolling for a routine low-risk bump, blue/green when a fast cutover matters and you can afford twice the capacity, shadow for a read path or a new engine.
• Name the compatibility rule: new version and old version run side by side for a while, so both must read and write the same data.
• Quantify: "a 5% canary caps the bad build to a 1% error rate, about a quarter of one percent of the monthly budget at a ten-minute detection" lands better than "we canary".
• The pitfall: treating configuration and flags as outside the release process. Several of the largest recent outages were configuration changes: Cloudflare's 18 November 2025 outage traced to a change in a feature file used by its bot management system, and Google Cloud's 12 June 2025 outage traced to policy data with blank fields that reached production without flag protection.