Observability at scale
Metrics say something is wrong, traces say where, logs say what — and p99 says whether anyone noticed.
Three pillars, three questions, one fixed order
Observability is usually taught as a list of three data types, which is useless at a whiteboard. The useful framing is that each answers a different question, and during an incident you reach for them in a fixed order. Metrics answer "is something wrong, and since when?" Traces answer "where in the call graph is it wrong?" Logs answer "what exactly happened to the request that failed?" Run that order backwards and you spend the outage grepping. Interviewers ask about this because a design that cannot be debugged is not a finished design — and because the candidate who says "and here is how I'd know it broke" is visibly a different animal from one who draws boxes and stops.
The three pillars, priced
| Pillar | Question it answers | Shape | Cost driver | Typical retention | How it fails you |
|---|---|---|---|---|---|
| Metrics | Is something wrong, and when did it start? | Numeric time series, fixed label set, pre-aggregated at write | Unique time series (cardinality), not event count | 13-15 months (downsampled) | Tells you the error rate is 4%. Cannot tell you which 4%. |
| Logs | What exactly happened to this one request? | Semi-structured records, unbounded fields | Bytes ingested and indexed — roughly $0.30-$1.00 per GB on managed platforms | 7-30 days hot, then cold storage | Volume. At real scale you cannot afford to keep them all, so the one you needed was sampled away. |
| Traces | Where in the call graph did the latency go? | A tree of timed spans sharing a trace id | Spans retained — so almost always sampled at 0.1%-10% | 3-14 days | Head-based sampling throws away the slow trace before it knows it was slow. |
A fourth thing is worth naming because senior candidates do: events — a durable, structured record of a business fact (order placed, payment captured). Those are not telemetry, they belong in the same queue infrastructure as everything else in the message-queue chapter, and they are the thing you reconcile against when metrics and reality disagree.
Averages lie, and they lie in the direction that hurts
A mean latency is a single number summarising a distribution that is never symmetric. Request latency is always right-skewed: it has a hard floor (you cannot be faster than the network) and no ceiling (a lock, a GC pause, a cold cache, a retry). Averaging a floor with a fat tail produces a number that describes no actual user. Worse, the mean is dominated by the tail it is hiding — you can double your p99 and barely move the mean.
Fan-out: why p99 per service is not p99 per user
Here is the arithmetic that separates people who have run a distributed system from people who have read about one. A user request that touches 10 backend services is only fast if all ten were fast. If each service independently has a 1% chance of being in its slow tail, the chance the request avoids every one of them is 0.99 to the tenth power.
| Services on the critical path | P(all fast) = 0.99n | P(at least one slow path) |
|---|---|---|
| 1 | 0.990 | 1.0% |
| 5 | 0.951 | 4.9% |
| 10 | 0.904 | 9.6% |
| 100 | 0.366 | 63.4% |
Read the last row again: with a hundred-way fan-out, a per-service p99 means most requests hit at least one slow path. This is why Jeff Dean's "tail at scale" framing matters — as you decompose into more services, the user-visible p99 degrades even though every individual service's p99 is unchanged. Turn it around to get the design rule: to hold a user-facing p99, each of ten backends needs roughly p99.9, because 0.99910 is 0.990. Every layer you add moves the percentile you must engineer for one notch further out.
Cardinality is the thing that makes metrics expensive
A metrics system does not charge you per event. Incrementing a counter a billion times costs essentially nothing. What it charges for is unique time series — one series per distinct combination of metric name and label values, each of which needs its own in-memory index entry and its own compressed chunk on disk. Cardinality is multiplicative:
http_requests_total{service, endpoint, status, region}
services 50
endpoints 200
statuses 15
regions 5
----------------------------------------------
50 * 200 * 15 * 5 = 750,000 active series // fine — a mid-size Prometheus handles this
at a 15s scrape interval:
750,000 * 4 samples/min * 1,440 min = 4.3 * 10^9 samples/day
at ~2 bytes/sample compressed = ~8.6 GB/day // affordable
now add one label: user_id, 20,000,000 values
750,000 * 20,000,000 = 1.5 * 10^13 series // the cluster is deadThe rule is mechanical: a metric label must be bounded and low cardinality — something you could write out on a whiteboard. User ids, request ids, session ids, raw URLs with path parameters, email addresses, and error strings all belong in logs or traces, where you pay per event but the schema is free. "Which user?" is a logs question. "How many users?" is a metrics question.
/orders/8814 instead of
/orders/:id); the raw exception message as a label when it
contains an id or a timestamp; and a pod name or container id as a label in
an autoscaling deployment, which quietly creates a new series every deploy
and never garbage-collects the old ones. Each one has taken down a real
monitoring cluster.
Distributed tracing and context propagation
A trace is one tree per user request. Each unit of work is a span carrying a trace id (shared across the whole request), its own span id, and a parent span id. The only genuinely hard part is context propagation: every hop — HTTP call, gRPC call, queue publish, thread handoff — must carry the ids forward, or the tree silently breaks into disconnected fragments. The W3C standard is a single header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
^ ^ ^ ^
| trace-id (16 bytes) parent span-id flags
version (8 bytes) (01 = sampled)| Sampling strategy | How it decides | Cost | Reach for this when… |
|---|---|---|---|
| Head-based, fixed rate | Coin flip at the entry point; the decision rides in the traceparent flags so the whole tree agrees | Cheapest; trivially stateless | Default. You want a representative sample of normal traffic and you accept losing most rare events. |
| Head-based, rate-limited per route | N traces per second per endpoint | Cheap; needs a per-route counter | Traffic is wildly skewed and a fixed rate would give you a million traces of the health check and none of checkout. |
| Tail-based | Buffer all spans of a trace at the collector, decide once it completes: keep if slow, errored, or rare | Collector must hold every in-flight trace in memory for the duration; needs all spans of a trace routed to the same collector | You specifically need the pathological traces — which, since those are the ones you debug, is most serious deployments. |
Structured logging, and the volume problem behind it
A log line that is a sentence is a log line you can only grep. A log line that is an object is a log line you can query, aggregate and join to a trace. The single most valuable field is the trace id — it is what turns three independent stores into one investigation.
{"ts":"2026-08-08T11:04:22.184Z","level":"error","svc":"ranking",
"trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7",
"user_id":"u_88214","route":"/v1/feed","status":503,"dur_ms":1841,
"err":"model_pool_exhausted","pool_size":32,"queue_depth":417}Now the volume. At 100,000 requests/sec with one 500-byte log line each:
100,000 * 500 B = 50 MB/s
50 MB/s * 86,400 s = 4.3 TB/day
4.3 TB/day at $0.50/GB ~ $2,150/day ~ $780,000/year // for one log line per requestThat number is why sampling is not optional at scale, and why the sampling policy should be asymmetric: keep 100% of errors and warnings, keep 100% of anything on a trace that was already sampled, and keep 1% of successful requests. You lose almost nothing diagnostically and drop roughly 99% of the bill. Add a per-service log budget so one team's debug statement cannot consume the org's ingest quota — a noisy-neighbour problem identical in shape to the one solved in the rate-limiting chapter.
Alert on symptoms, not causes
The instinct is to alert on everything you can measure: CPU above 80%, disk above 70%, replica lag above 5 seconds. Every one of those is a cause alert, and cause alerts have two failure modes. They fire when nothing is wrong (CPU at 90% during a nightly batch job is correct behaviour), and they fail to fire when something is wrong in a way you did not predict. After six months of false pages, the team mutes the channel, and the monitoring system is now decorative.
Alert on symptoms instead: the things a user would complain about. Google's four golden signals are latency, traffic, errors, and saturation; the first three are symptoms and the fourth is the leading indicator you use for capacity planning, not for paging. Keep cause metrics — you need them the moment you start debugging — but put them on dashboards, not on the pager.
| Don't page on this (cause) | Page on this instead (symptom) | Why |
|---|---|---|
| CPU above 80% on app tier | p99 request latency above the SLO threshold | High CPU with acceptable latency is a well-utilised fleet, not an incident. |
| Read replica lag above 5 s | Rate of stale-read complaints / stale-read guard trips | Lag only matters if a user reads their own write and doesn't see it. |
| Queue depth above 10,000 | Oldest-message age above 5 minutes | Depth depends on message size and consumer count; age is directly the user-visible delay. |
| A single host is unreachable | Error rate above the error budget burn threshold | If losing one host pages you, you did not build the fault tolerance you claimed. |
SLOs and error budgets: making the alert threshold non-arbitrary
An SLI is the measurement ("proportion of requests served in under 300 ms"). An SLO is the target on it ("99.9% over a rolling 30 days"). The error budget is the leftover: 0.1% of requests, which over 30 days is a concrete allowance you may spend on deploys, experiments and bad luck. This is the mechanism that converts a philosophical argument about reliability into arithmetic.
| Availability target | Budget per 30 days | Budget per year | What it implies |
|---|---|---|---|
| 99% | 7.2 hours | 3.65 days | An internal tool. One long maintenance window is fine. |
| 99.9% ("three nines") | 43.2 minutes | 8.8 hours | A normal SaaS product. Achievable with a single region done well. |
| 99.95% | 21.6 minutes | 4.4 hours | Multi-AZ, automated failover, no manual step in the recovery path. |
| 99.99% | 4.3 minutes | 52.6 minutes | Multi-region active-active. A human cannot even read the page in the budget. |
| 99.999% | 26 seconds | 5.3 minutes | Almost never the right answer. Say so out loud when someone asks for it. |
Alerting on the raw SLI ("error rate above 0.1%") is too twitchy: a 30-second blip trips it. Alerting on the whole window is too slow: you find out you blew the budget on day 29. The standard answer is multi-window burn-rate alerting — page on how fast the budget is being consumed, measured over both a long and a short window so a recovered blip stops paging.
| Burn rate | Budget consumed | Windows (long + short) | Response |
|---|---|---|---|
| 14.4× | 2% in 1 hour | 1 hour + 5 minutes | Page immediately. At this rate the month is gone in ~2 days. |
| 6× | 5% in 6 hours | 6 hours + 30 minutes | Page. |
| 1× | 10% in 3 days | 3 days + 6 hours | File a ticket. This is a slow leak, not an outage. |
The short window is the part people forget: it exists so that when the incident ends, the alert clears within minutes instead of staying lit for the remaining hour of the long window. And the error budget has a second, political job — when it is exhausted, feature launches stop until reliability work restores it. That is the only mechanism anyone has found that makes "we should invest in reliability" an automatic decision rather than a quarterly argument.
Recognizing it in an unseen problem
- The prompt says "highly available", "five nines", "SLA", or asks how you would operate the system — that is an invitation to talk SLI/SLO/error budget, and most candidates skip it entirely.
- A naive design monitors hosts (CPU, memory, disk) and calls it observability. The tell of a senior answer is monitoring user-visible symptoms and treating host metrics as debugging aids.
- Any design with more than about five services on the critical path has a tail-latency problem by construction — bring up the 0.99n arithmetic unprompted; it is one of the highest-signal thirty seconds available to you.
- If you propose adding a label to a metric, immediately state its cardinality bound. If it is unbounded, it is a log field, not a label.
- Distinguish it from the fault-tolerance chapter: fault tolerance is about the system surviving a failure; observability is about humans finding out and locating it. A system can be perfectly redundant and completely un-debuggable.
- The pitfall: proposing logging everything. At 100k rps that is a multi-hundred-thousand-dollar annual line item. Name the sampling policy (all errors, 1% of successes) before the interviewer has to ask what it costs.