Observability at scale

Three pillars, three questions, one common order

Observability is usually taught as a list of three data types, which is useless at a whiteboard. The useful framing is that each answers a different question, and during an incident you usually reach for them in a familiar order. Metrics answer "is something wrong, and since when?" Traces answer "where in the call graph is it wrong?" Logs answer "what exactly happened to the request that failed?" Run that order backwards and you spend the outage grepping. It is a default drill-down, not a law: teams that store one wide, structured event per request (what some call "observability 2.0") can slice by any field and jump straight to the evidence, and "three pillars" is itself a contested label. Interviewers ask about this because a design that cannot be debugged is not a finished design — and because the candidate who says "and here is how I'd know it broke" is visibly a different animal from one who draws boxes and stops.

The three pillars, priced

A fourth thing is worth naming because senior candidates do: events — a durable, structured record of a business fact (order placed, payment captured). Those are not telemetry, they belong in the same queue infrastructure as everything else in the message-queue chapter, and they are the thing you reconcile against when metrics and reality disagree.

Averages lie, and they lie in the direction that hurts

A mean latency is a single number summarising a distribution that is never symmetric. Request latency is always right-skewed: it has a hard floor (you cannot be faster than the network) and no ceiling (a lock, a GC pause, a cold cache, a retry). Averaging a floor with a fat tail produces a number that describes no actual user. Worse, the mean blends the fast majority and the slow tail into one number that hides both: if the slowest 1% of requests all got twice as slow (p99 going from 1.9 s to 3.8 s), the mean would rise by only about 19 ms, from 205 ms to roughly 224 ms, while the p99 doubled.

Fan-out: why p99 per service is not p99 per user

Here is the arithmetic that separates people who have run a distributed system from people who have read about one. A user request that touches 10 backend services is only fast if all ten were fast. If each service independently has a 1% chance of being in its slow tail, the chance the request avoids every one of them is 0.99 to the tenth power.

Read the last row again: with a hundred-way fan-out, a per-service p99 means most requests hit at least one slow path. This is why Jeff Dean and Luiz Barroso's "The Tail at Scale" (CACM, 2013) matters — as you decompose into more services, the user-visible p99 degrades even though every individual service's p99 is unchanged. Turn it around to get the design rule: to hold a user-facing p99, each of ten backends needs roughly p99.9, because 0.999¹⁰ is 0.990. Every layer you add moves the percentile you must engineer for one notch further out.

Cardinality is the thing that makes metrics expensive

A metrics system does not charge you per event. Incrementing a counter a billion times costs essentially nothing. What it charges for is unique time series — one series per distinct combination of metric name and label values, each of which needs its own in-memory index entry and its own compressed chunk on disk. Cardinality is multiplicative:

http_requests_total{service, endpoint, status, region}

  services   50
  endpoints  200
  statuses   15
  regions    5
  ----------------------------------------------
  50 * 200 * 15 * 5 = 750,000 active series   // fine — a mid-size Prometheus handles this

at a 15s scrape interval:
  750,000 * 4 samples/min * 1,440 min = 4.3 * 10^9 samples/day
  at ~2 bytes/sample compressed = ~8.6 GB/day   // affordable

now add one label: user_id, 20,000,000 values
  750,000 * 20,000,000 = 1.5 * 10^13 series    // the cluster is dead

The rule is mechanical: a metric label must be bounded and low cardinality — something you could write out on a whiteboard. User ids, request ids, session ids, raw URLs with path parameters, email addresses, and error strings all belong in logs or traces, where you pay per event but the schema is free. "Which user?" is a logs question. "How many users?" is a metrics question.

Distributed tracing and context propagation

A trace is one tree per user request. Each unit of work is a span carrying a trace id (shared across the whole request), its own span id, and a parent span id. The only genuinely hard part is context propagation: every hop — HTTP call, gRPC call, queue publish, thread handoff — must carry the ids forward, or the tree silently breaks into disconnected fragments. The W3C Trace Context standard (a W3C Recommendation, updated November 2021) defines a traceparent header, plus an optional tracestate header for vendor-specific data:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^   ^                                ^                ^
             |   trace-id (16 bytes)              parent span-id   flags
             version                              (8 bytes)        (01 = sampled)

Structured logging, and the volume problem behind it

A log line that is a sentence is a log line you can only grep. A log line that is an object is a log line you can query, aggregate and join to a trace. The single most valuable field is the trace id — it is what turns three independent stores into one investigation.

{"ts":"2026-08-08T11:04:22.184Z","level":"error","svc":"ranking",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7",
 "user_id":"u_88214","route":"/v1/feed","status":503,"dur_ms":1841,
 "err":"model_pool_exhausted","pool_size":32,"queue_depth":417}

Now the volume. At 100,000 requests/sec with one 500-byte log line each:

100,000 * 500 B          = 50 MB/s
50 MB/s * 86,400 s       = 4.3 TB/day
4.3 TB/day at $0.50/GB   ~ $2,150/day  ~ $780,000/year   // $0.50/GB is illustrative; one log line per request

That number is why sampling is not optional at scale, and why the sampling policy should be asymmetric: keep 100% of errors and warnings, keep 100% of anything on a trace that was already sampled, and keep 1% of successful requests. You lose almost nothing diagnostically and drop roughly 99% of the bill. Add a per-service log budget so one team's debug statement cannot consume the org's ingest quota — a noisy-neighbour problem identical in shape to the one solved in the rate-limiting chapter.

Alert on symptoms, not causes

The instinct is to alert on everything you can measure: CPU above 80%, disk above 70%, replica lag above 5 seconds. Every one of those is a cause alert, and cause alerts have two failure modes. They fire when nothing is wrong (CPU at 90% during a nightly batch job is correct behaviour), and they fail to fire when something is wrong in a way you did not predict. After six months of false pages, the team mutes the channel, and the monitoring system is now decorative.

Alert on symptoms instead: the things a user would complain about. Google's four golden signals are latency, traffic, errors, and saturation; the first three are symptoms, and saturation is the early warning. The SRE book's rule is to page when one signal is problematic or, in the case of saturation, nearly problematic: a disk or quota about to fill deserves a page before it causes errors. Page on that kind of imminent exhaustion, and leave a generic "CPU above 80%" on the dashboard. Keep cause metrics — you need them the moment you start debugging — but put them on dashboards, not on the pager. Every page should link to a runbook and have an owner, and every serious incident should end in a blameless postmortem; an alert that nobody can act on is noise, however well it was measured.

SLOs and error budgets: making the alert threshold non-arbitrary

An SLI is the measurement ("proportion of requests served in under 300 ms"). An SLO is the target on it ("99.9% over a rolling 30 days"). The error budget is the leftover: 0.1% of requests, which over 30 days is a concrete allowance you may spend on deploys, experiments and bad luck. This is the mechanism that converts a philosophical argument about reliability into arithmetic.

Alerting on the raw SLI ("error rate above 0.1%") is too twitchy: a 30-second blip trips it. Alerting on the whole window is too slow: you find out you blew the budget on day 29. The standard answer is multi-window burn-rate alerting — page on how fast the budget is being consumed, measured over both a long and a short window so a recovered blip stops paging. The table below is the recommended policy for a 99.9% SLO from the Google SRE Workbook chapter "Alerting on SLOs"; the short window is about one twelfth of the long one.

Dry run: a 20-minute incident through a multi-window burn-rate alert

SLO: 99.9% over 30 days → an allowed error rate of 0.1%. Page when both the 1-hour and the 5-minute burn rate exceed 14.4× (an error rate of 1.44%). Traffic is steady; baseline error rate is 0.05%; the incident runs 10:00–10:20 at a 5% error rate.

The alert only fires once both windows agree, which is why it waits until about 10:17 rather than paging on the first bad minute — and it clears at about 10:24 even though the 1-hour window is still showing a 17× burn, because the 5-minute window is what tells the alert the incident is actually over. Without that short window, this page would stay lit for most of the remaining hour after the problem was already fixed. The price of the long window is the 17-minute wait; a total outage burns the budget far faster than 14.4× and trips the same rule in about 1 minute (a 100% error rate gives a burn rate of 1000×, and the 1-hour window crosses 14.4× after about 0.83 bad minutes).

The error budget also has a second, political job — when it is exhausted, feature launches stop until reliability work restores it. That is the only mechanism anyone has found that makes "we should invest in reliability" an automatic decision rather than a quarterly argument.

Recognizing it in an unseen problem

• The prompt says "highly available", "five nines", "SLA", or asks how you would operate the system — that is an invitation to talk SLI/SLO/error budget, and most candidates skip it entirely.
• A naive design monitors hosts (CPU, memory, disk) and calls it observability. The tell of a senior answer is monitoring user-visible symptoms and treating host metrics as debugging aids.
• Any design with more than about five services on the critical path has a tail-latency problem by construction — bring up the 0.99ⁿ arithmetic unprompted; it is one of the highest-signal thirty seconds available to you.
• If you propose adding a label to a metric, immediately state its cardinality bound. If it is unbounded, it is a log field, not a label.
• Distinguish it from the fault-tolerance chapter: fault tolerance is about the system surviving a failure; observability is about humans finding out and locating it. A system can be perfectly redundant and completely un-debuggable.
• The pitfall: proposing logging everything. At 100k rps that is a multi-hundred-thousand-dollar annual line item. Name the sampling policy (all errors, 1% of successes) before the interviewer has to ask what it costs.