Real-world case studies

How to read these — and how not to use them

Everything below is drawn from publicly discussed architecture: conference talks, engineering blog posts, papers. Each is a snapshot of a system at a point in time, usually years ago, and in several cases the company has since rebuilt the thing described. None of it is current internal truth, and you should say so if you cite it. The value is not the architecture; it is the reasoning, and specifically the part that most retellings skip: what the clever decision cost.

Used badly, case studies are name-dropping — "I'd do it like Twitter" signals that you have read a blog post, not that you can design. Used well, they are a compressed argument: "there is a known pattern for the celebrity problem; it is a hybrid, and the threshold is a tunable I'd set at around a million followers." Each section below ends with the one transferable move.

Twitter timelines — the hybrid fan-out

The problem, as described in Twitter's engineering talks around 2013: home timeline reads ran at a few hundred thousand requests per second against tweet writes in the low thousands per second. A read-time query — "fetch every tweet from everyone this user follows, merge, sort" — is a scatter across thousands of partitions on the hottest path in the product. The fix was to invert it: on write, push the tweet id into a precomputed per-follower timeline list held in memory (Redis), capped at roughly 800 entries. The read then becomes a single list fetch. Reads went from a distributed join to an O(1) lookup.

The cost is write amplification, and it is brutal at the tail of the follower distribution. A user with 30 million followers generates 30 million list insertions per tweet. At a few thousand tweets per second across the network, total fan-out delivery ran into tens of billions of timeline writes per day. So the design became a hybrid: fan out on write for ordinary accounts, and for the small set of very-high-follower accounts, skip fan-out entirely and merge their recent tweets in at read time.

What to steal in an interview: whenever a workload has a power-law distribution, propose treating the head and the tail differently, and name the threshold as a tunable. "Fan-out on write below a million followers, fan-out on read above it, and I'd make that a config value because the right number depends on the follower histogram" is a complete answer to a whole family of feed, notification and subscription questions.

Netflix — precomputed placement, and failure as a habit

Netflix's Open Connect programme, publicly described from around 2012, inverts the usual CDN model. Instead of caching reactively on a miss, Netflix ships physical appliances into ISP networks and internet exchanges, and fills them overnight during off-peak hours with content chosen by a popularity prediction per region. Video is unusually well suited to this: the catalogue is finite, changes slowly, and next-Tuesday's demand is genuinely predictable from this-Tuesday's. So the cache miss — the expensive event that dominates ordinary CDN design — is mostly engineered out of existence rather than optimised.

The cost is that you are now running a hardware logistics operation: manufacturing, shipping, ISP contracts, remote diagnostics for boxes you cannot physically reach. That is a real organisational commitment, and it only pencils out because streaming video is a large enough fraction of internet traffic to justify it. Do not propose it for a service serving 200 KB images.

The second Netflix idea is cultural rather than architectural: Chaos Monkey and the broader Simian Army, which terminate production instances during business hours on purpose. The reasoning is that redundancy you have never exercised is a claim, not a property — failover paths rot, runbooks go stale, and the first real test of your multi-AZ story should not be at 3am. The cost is that you must build the resilience first; switching on failure injection in a system that isn't ready is just an outage you scheduled.

What to steal: two sentences. "If demand is predictable, precompute placement instead of caching reactively." And, at the end of any availability discussion, "I'd verify the failover path with scheduled failure injection, because an untested failover is a broken failover." The second one costs you fifteen seconds and reliably lands.

Uber — a spatial index is a sharding decision

Matching riders to drivers is a continuous geospatial query: given a point, find nearby available supply, fast, while every driver's position updates every few seconds. Uber's publicly described answer (the H3 library, open sourced in 2018) indexes the world as a hierarchy of hexagonal cells, each addressed by a 64-bit id, at sixteen resolutions from continent-sized down to about a square metre. A proximity query becomes "look up this cell and its ring of neighbours" — a key lookup, not a geometric search.

Hexagons rather than squares for a specific reason: a hexagon has six neighbours and all six centroids are equidistant. A square grid has four edge-neighbours at distance d and four corner-neighbours at d√2, so "expand the search by one ring" means different things in different directions, and any smoothing or gradient over the grid is distorted. The cost is that hexagons do not tile hierarchically: you cannot subdivide a hexagon into smaller hexagons exactly, so containment across resolutions is approximate, and wrapping a sphere in hexagons mathematically requires exactly twelve pentagons (H3 places them over ocean). That is a genuine correctness footnote you inherit.

The deeper point is that the spatial index doubles as the shard key. Uber's dispatch system was sharded geographically, which is the natural choice and which imports the natural failure: geography is not a uniform load distribution. A stadium at kickoff is a hot cell, and no amount of consistent hashing helps because the load genuinely is in one place. The mitigations are the usual ones from the sharding chapter — split hot cells to a finer resolution dynamically, and keep the dispatch tier able to rebalance ownership between nodes.

What to steal: for any "find things near me" prompt, name a concrete index (H3, S2, geohash, or a PostGIS R-tree) and immediately name the resolution tradeoff — coarse cells mean fewer lookups but more candidates to filter; fine cells mean more lookups but tighter results. Then volunteer the hot-cell problem before the interviewer does.

WhatsApp — constraining the product is a scaling lever

The widely cited figures: in 2015 WhatsApp served roughly 900 million users with an engineering team of about fifty. Their 2012 blog post described holding two million concurrent TCP connections on a single machine, running Erlang on FreeBSD. Erlang's runtime is the reason it is even plausible — millions of cheap preemptively-scheduled processes, per-process heaps so garbage collection is per-connection and measured in microseconds rather than a stop-the-world pause, and supervision trees that restart a crashed connection process without touching its neighbours. Add heavy kernel tuning of socket buffers and file descriptor limits, and the per-connection cost lands in the low tens of kilobytes.

But the runtime is only half of it. The other half is that WhatsApp refused features. For most of that period the server did not store message history — a message was queued only until delivered, then deleted. That single product decision is worth revisiting with the arithmetic from the capacity chapter: retaining history costs petabytes a year; retaining only undelivered messages costs tens of gigabytes. No ads, no timeline, no media transcoding pipeline of significance, minimal server-side state. The small team was not despite the scale; it was possible because the surface area was tiny.

The cost is severe specialisation. Hiring Erlang engineers is hard, the ecosystem is small, and the architecture is unusually inflexible in the direction of adding features — which is precisely what a company adds after an acquisition.

What to steal: when the requirements phase is happening, ask whether a feature is actually required, and put a number on what it costs if it is. "Do we need server-side history? If yes I need a petabyte-scale message store; if no, a 40 GB queue covers it" is the highest-leverage question in the whole interview, and almost nobody asks it. Also: name your per-connection memory budget on any real-time design.

Instagram — boring database, aggressively applied

Instagram is the standing counterexample to "web scale needs a new database." Through its hypergrowth years it ran on PostgreSQL, sharded by user id, with two decisions that made it work.

First, blobs left the database immediately — photos went to object storage behind a CDN, and Postgres held only metadata rows. Re-run the capacity arithmetic and it is obvious why: the blob path is measured in petabytes and the metadata path in terabytes. Keeping them in the same system means sizing your relational store for the wrong workload.

Second, logical shards. Rather than mapping users onto physical machines, they mapped users onto a few thousand logical shards (Postgres schemas), then mapped many logical shards onto each physical machine. Growing the cluster means moving logical shards between machines — no re-hashing, no application change. Their id generation scheme, described in a 2012 post, packs the shard into the primary key itself:

64-bit id =  41 bits  millisecond timestamp   // ~69 years of range from a custom epoch
           + 13 bits  logical shard id        // 8,192 logical shards
           + 10 bits  per-shard sequence      // 1,024 ids per ms per shard

  ids are roughly time-sortable  -> good index locality on an append-heavy table
  the shard is IN the id         -> route a request without a lookup table
  generated in the database      -> no separate id service to keep available

The cost is everything you give up by sharding a relational database: no cross-shard joins, no cross-shard transactions, no global secondary indexes, and application code that must always know the shard key. Connection management becomes its own project — thousands of app processes against a Postgres backend that forks per connection means PgBouncer is not optional.

What to steal: "I'd start with Postgres" is a legitimate and often correct answer, and the way to make it credible is to pair it with the scaling path — blobs out to object storage, logical shards from day one so rebalancing is a data move rather than a rewrite, and a shard-encoded id. Say the sentence "sharding costs me joins and cross-shard transactions" before the interviewer asks.

Discord — coalescing in front of a hot partition

Discord's two posts on message storage — billions of messages on Cassandra in 2017, trillions on ScyllaDB in 2023 — are unusually honest about what went wrong in between, which makes them the most instructive pair on this list.

The original data model is worth memorising as a template: partition key of (channel_id, bucket) where the bucket is a fixed time window, clustered by a snowflake message id in descending order. The bucket exists because a partition keyed on channel alone grows without bound; bucketing caps partition size and makes "load the most recent messages" a single partition read with no sorting.

What broke was not throughput but tail latency, and the causes are a tour of distributed-storage failure modes. Hot partitions: a handful of enormous channels received a wildly disproportionate share of reads. Tombstones: deletes in an LSM store are writes, and a range scan must read past every tombstone in the range, so a heavily-moderated channel became slow to read. And JVM garbage collection: stop-the-world pauses on Cassandra nodes produced multi-second stalls that showed up purely as p99 spikes — invisible in an average, invisible in the error rate, exactly the pattern the observability chapter warns about.

The 2023 rebuild had two parts. ScyllaDB — a C++ reimplementation of Cassandra with a shard-per-core architecture and no garbage collector — removed the GC pauses. More interesting is what they put in front: intermediate "data services" written in Rust that coalesce concurrent identical requests. When ten thousand clients ask for the same channel page in the same instant, the first request goes to the database and the rest wait on its result.

What to steal: request coalescing (also called single-flight or request collapsing) is a genuinely reusable move, and mentioning it marks you as someone who has fought a thundering herd. It is also the correct answer to cache-stampede questions from the caching chapter: on a miss, one request refills while the others wait, instead of ten thousand simultaneous refills.

The six, on one line each

System	The one interesting decision	What it cost	The transferable move
Twitter timelines (~2013)	Fan-out on write, hybrid for high-follower accounts	Enormous write amplification; a threshold that must be tuned	Split head and tail of a power-law workload; name the threshold
Netflix Open Connect (~2012+)	Predictive overnight cache fill into ISP-hosted appliances	Running a hardware logistics business	If demand is predictable, precompute placement rather than caching reactively
Netflix Chaos Monkey	Inject failure in production, continuously	Requires the resilience to exist first	An untested failover is a broken failover
Uber H3 (open sourced 2018)	Hexagonal hierarchical spatial index as both index and shard key	Approximate containment across resolutions; twelve pentagons; hot cells at events	Name the spatial index and its resolution tradeoff; volunteer the hotspot
WhatsApp (~2012-2015)	Erlang + kernel tuning for millions of sockets per box; no server-side history	Extreme specialisation; a deliberately tiny feature surface	Cutting a requirement is a scaling lever — put a number on it
Instagram (~2012+)	Sharded Postgres with logical shards and shard-encoded ids; blobs in object storage	No cross-shard joins or transactions; connection pooling as a project	Boring database plus an explicit scaling path beats an exotic one
Discord (2017 → 2023)	Bucketed partitions; then request coalescing in front of ScyllaDB	A trillion-row migration; a bespoke service tier to maintain	Coalesce concurrent identical reads; GC pauses surface as p99, not as errors

Using a case study without sounding like you read a blog post

• Lead with the problem, not the company. "This has the celebrity fan-out shape" is analysis; "Twitter uses Redis" is trivia. Name the company second, as a citation.
• Date it and hedge it. "As of their 2017 write-up" costs you four words and protects you completely — the interviewer may work there and know it changed.
• Always state the cost. A candidate who says "and that cost them cross-shard joins" has understood the decision. A candidate who lists only the benefit has memorised it.
• Never argue from authority. "Netflix does it" is not a reason. "Our demand is predictable in the same way theirs is, which is what makes precomputed placement work here" is a reason.
• Scale-check the analogy out loud. Most of these solve problems that appear above roughly 10 million users. If the prompt is a 100k-user product, the correct use of the case study is to explain why you are not doing it yet — which sets up the tradeoff-thinking chapter's central point.
• The pitfall: importing a whole architecture instead of one idea. These companies each had a specific constraint that justified a specific cost. Steal the reasoning; leave the org chart.