Walkthrough: a news feed / chat app
Fan-out on write, fan-out on read, and the hybrid that exists because one account has 150 million followers.
The problem, and what makes it a real interview
"Design a news feed" is the most-asked medium system design question in the industry, and it is asked because it has one genuinely hard decision buried under a lot of easy ones. Everything about it — the API, the data model, the caching — is routine. The decision is where the work happens: at write time or at read time, and the fact that neither answer works alone at scale. Candidates who reach the hybrid, and can say why the threshold exists, pass. Candidates who describe a beautiful CRUD service and never state the tradeoff do not.
Spend the first four minutes on scope. Not because interviewers reward ceremony, but because the fan-out decision is entirely determined by numbers you have to extract from them.
| In scope | Explicitly out of scope |
|---|---|
| Post text and images; follow / unfollow | Ads, monetization, DMs |
| Home feed: posts from accounts you follow | Search and hashtag discovery (see the search chapter) |
| Infinite scroll, newest-first with light ranking | Full ML ranking infrastructure and training pipelines |
| Likes and comment counts on feed items | Comment threads themselves |
Non-functional targets, stated as numbers because vague ones cannot drive a design: feed load p99 under 200 ms; a new post visible to followers within a few seconds (eventual consistency is fine and the product does not need better); read-heavy by roughly 100:1; availability 99.9% — a briefly stale feed is acceptable, an unavailable one is not. That last sentence already tells you the system should favour availability over consistency, which is the CAP-chapter tradeoff arriving as a product requirement rather than a theory question.
Capacity estimation, with the arithmetic shown
Do this on the board, out loud, rounding aggressively. The goal is not accuracy to two significant figures; it is to surface the one or two numbers that eliminate an entire design.
| Quantity | Arithmetic | Result |
|---|---|---|
| Daily active users | given | 500 M |
| Feed opens | 500 M × 15 per day ÷ 86,400 s | ~87,000 reads/s average |
| Peak reads | × 3 for daily peak | ~250,000 reads/s |
| Posts | 10% of DAU × 1.5 posts ÷ 86,400 s | ~870 writes/s average, ~2,500/s peak |
| Read : write ratio | 87,000 ÷ 870 | ~100 : 1 |
| Average accounts followed | given (mean; median far lower) | 200 |
| Post metadata storage | 75 M/day × 500 B × 365 | ~14 TB/year — trivial |
| Media storage | 30% of 75 M × ~800 KB (all renditions) × 365 | ~6.6 PB/year — object storage, not a database |
| Media egress at peak | 250,000 feeds/s × ~6 images × 150 KB | ~200 GB/s — a CDN problem, not an origin problem |
Two of those numbers decide the architecture. 100:1 read-heavy says precompute: it is worth doing substantially more work per write to make reads cheap. And 6.6 PB/year of media at 200 GB/s says the media path is completely separate from the feed path — object storage behind a CDN, with the feed API returning URLs, never bytes. Neither of those is a close call, and saying so quickly buys you time for the decision that is.
The core services and the data model
Four services, deliberately boring, so that the interesting part stands out. Post service owns the canonical posts table. Graph service owns follows, and answers "who follows X" and "who does X follow" — both directions, because fan-out needs the first and read-time merge needs the second. Feed service owns per-user materialized feeds. Media service issues presigned upload URLs and runs the transcode pipeline from the storage chapter.
| Store | Key | Holds | Why this shape |
|---|---|---|---|
| posts | post_id (snowflake: time-ordered) | author_id, text, media keys, created_at, counters | Time-ordered IDs mean a sort by ID is a sort by time — no extra index |
| follows | (follower_id, followee_id) | created_at | Stored twice, once per direction, because both lookups are hot |
| feed | user_id → sorted list of (score, post_id) | capped at ~500 entries | IDs only. Hydrate post bodies separately so an edited post is never stale in a million copies |
| counters | post_id | likes, comments | Separate, because they change orders of magnitude more often than the post does |
The single most important line in that table is "IDs only". If you copy the post body into every follower's feed, a post edit or deletion means chasing a million copies, and your feed store balloons by a factor of 50. Store references; hydrate at read time from a cache keyed by post_id, where a single copy serves every reader.
Fan-out on write: pay at post time
When a user posts, immediately push the post ID into the materialized feed of every one of their followers. Reading a feed then costs one range read of a precomputed list.
The arithmetic: 870 posts/s × 200 followers = ~174,000 feed appends per second average, and roughly 500,000/s at peak. That is a lot but it is entirely tractable — these are tiny appends to a sorted structure, spread across a sharded Redis or Cassandra cluster, and they are asynchronous so they can absorb a queue.
Two optimizations cut it substantially and both are worth volunteering. Fan out only to active users: with 2 B registered accounts and 500 M daily actives, most followers will not open the app today, so writing to them is pure waste. Fan out to accounts active in the last 30 days and lazily build the feed for anyone else on their next login. And cap the feed at ~500 entries — nobody scrolls past that, and an uncapped list grows without bound.
Fan-out on read: pay at feed time
| Fan-out on write | Fan-out on read | |
|---|---|---|
| Write cost | O(followers) — 200 appends, up to 150 M for a celebrity | O(1) — one insert |
| Read cost | O(1) — one range scan, p99 under 10 ms from cache | O(following) — 200 lookups plus a merge, per request |
| Total ops at our numbers | 174 k writes/s, 87 k cheap reads/s | 870 writes/s, 17.4 M lookups/s |
| Storage | Duplicated per follower | Single copy of each post |
| Freshness | Seconds of lag while fan-out drains | Perfectly fresh by construction |
| Unfollow / delete | Requires cleanup, or filtering at read | Free — the merge just stops including them |
| Reach for this when… | Read-heavy, bounded follower counts | Write-heavy, or the reader follows very few, very prolific accounts |
17.4 million lookups per second, with a p99 gated by the slowest of 200 parallel calls, is not a system you can operate. At a 100:1 read ratio, fan-out on read loses on arithmetic. But look at the top-right cell of that table again — that is where fan-out on write dies.
The celebrity problem, and the hybrid that answers it
An account with 150 million followers posts. Under pure fan-out on write that is 150 million feed appends for one tweet. Even at a dedicated 100,000 appends/s you are looking at 1,500 seconds — 25 minutes before the last follower sees it. Worse, it is bursty and it starves everyone else's fan-out behind it in the same queue. And ten such accounts posting in the same minute is a self-inflicted denial of service.
The answer every real system converges on: fan out on write for normal accounts, fan out on read for the handful of accounts above a follower threshold, and merge the two at request time.
Why this is cheap: the number of accounts above 100,000 followers is small, and crucially the number of them any one user follows is small — typically under ten. So a feed read becomes one range scan of the materialized feed plus perhaps six cached "recent posts by author" lookups, merged and re-sorted. That is a bounded, predictable cost, unlike the 200 lookups of pure fan-out on read.
Two refinements, if the interviewer digs. The threshold should ideally be dynamic rather than a magic constant — the real cost driver is followers × posting rate, so an account with 80,000 followers posting 50 times a day may deserve pull treatment while a 200,000-follower account posting monthly does not. And the celebrity's recent posts are a perfect cache target: one list, read by tens of millions of people, with a hit rate approaching 100%.
Ranking, cursors, and the caching stack
Ranking. Reverse-chronological is a legitimate v1 and you should say so. When ranking arrives, use the same two-stage shape as search: candidate generation pulls ~500 entries from the materialized feed plus celebrity pulls, then a scorer ranks them on affinity × recency_decay × predicted_engagement. Affinity is how much this reader interacts with this author; recency decay is exponential with a half-life of hours. Keep the score out of the stored feed if you want to change the model without a backfill.
Pagination must use cursors, never OFFSET. The feed has new items
inserted at the head constantly. With LIMIT 20 OFFSET 20, if
five posts arrive between page 1 and page 2, everything shifts down by five
and the user sees five duplicates and misses nothing — or, scrolling the
other way, misses items entirely. A cursor encodes the position in the
ordering itself: (last_score, last_post_id) for a ranked feed,
or just the last post ID for a chronological one, base64-encoded so clients
treat it as opaque. For a ranked feed also pin a session seed and
snapshot timestamp in the cursor, so the ranking model does not reshuffle
under the user mid-scroll.
OFFSET 10000 makes the database
produce and discard 10,000 rows before returning yours. Deep pagination
costs grow linearly with depth. Cursors are O(1) regardless of how far the
user has scrolled, because they seek directly into the index.
| Layer | Key | Hit rate | Why it exists |
|---|---|---|---|
| CDN | media URL | >95% | Media is 200 GB/s at peak — this is the only layer that can carry it |
| Feed list cache | feed:user_id | ~90% for active users | The top ~100 IDs for active users; ~2 TB across the fleet, the tail lives in Cassandra |
| Post hydration cache | post:post_id | >95% | One copy read by everyone. A multi-get of 20 IDs replaces 20 database reads |
| Celebrity timeline cache | author:recent | ~100% | Makes the pull half of the hybrid essentially free |
| Counter cache | counts:post_id | high, written back in batches | Like counts change far faster than posts; do not make the post cache churn for them |
Feed store sizing, since interviewers ask: 500 M active users × 500 entries × 40 bytes = ~10 TB if you keep everything hot. Keeping only the top 100 entries in memory is 500 M × 100 × 40 = 2 TB, which is a manageable Redis cluster, with the remainder paged in from a disk-backed store on the rare deep scroll.
Part two: the same reasoning applied to chat
Chat looks like a different problem and is largely the same one with the latency requirement tightened and the connection made persistent. Targets: message delivery p99 under 500 ms, 10 M concurrent connections, ~20 B messages/day (~230,000/s average, ~600,000/s peak), and — unlike the feed — ordering and delivery guarantees actually matter. A feed showing posts slightly out of order is fine. A conversation showing the answer before the question is broken.
Connection management. A tuned Linux box holds 250,000-500,000 idle
WebSockets — the constraint is memory (roughly 20-50 KB per connection in
kernel and userspace buffers) and file descriptors, not the mythical 65,535
port limit, which applies to outbound tuples rather than accepted
connections. So 10 M concurrent needs about 40 gateway nodes plus headroom.
Gateways must be as thin as possible: terminate TLS, hold the socket,
translate frames to internal messages. The routing state lives in a
registry (Redis: user_id → gateway_node_id, with a TTL
refreshed by heartbeat) so that any service can find any user's socket in
one lookup. Sending to a user is then: look up the node, publish to that
node's topic, gateway writes the frame.
Ordering. Never order by client timestamp — clocks are skewed by seconds and users can set them arbitrarily. Assign a monotonic per-conversation sequence number at a single owner (the shard that owns that conversation), so ordering is total within the conversation, which is the only place it is observable. Global ordering across conversations is neither needed nor affordable. The client also attaches a client-generated message UUID so a retry after an ambiguous timeout is deduplicated server-side rather than posting twice — the cheapest idempotency mechanism there is, and its absence is a very visible gap.
Delivery receipts are a four-state machine — sent, server-acked, delivered to device, read — and each transition is itself a message that must be stored and routed. That doubles or triples your message volume, so batch them: acknowledge up to a sequence number rather than per message. Presence looks trivial and is not: 10 M online users heartbeating every 30 seconds is ~330,000 writes/s just to say "still here," and a single online/offline transition must notify everyone watching. Only subscribe to presence for contacts currently visible on screen, and batch transitions into a periodic digest rather than pushing each one.
And then group chat brings the fan-out question straight back. A message to a 5-person group is 5 deliveries — push it into each member's inbox, exactly like fan-out on write. A message to a 100,000-member community is 100,000 deliveries per message, which is the celebrity problem wearing a different hat. The same answer applies: store the message once per conversation and let clients pull by (conversation_id, seq > last_seen_seq), with a lightweight "something changed" nudge instead of a full delivery. Small groups push, large groups pull, and the threshold is again about member count times message rate. Read receipts in a large group get degraded to an aggregate count or removed entirely, because N readers × M messages of receipt traffic exceeds the traffic of the conversation itself.
What the interviewer was actually scoring
- Did you find the real decision? Everything in this design is easy except fan-out. Time spent on the posts table schema is time not spent on the thing being evaluated. Getting to the fan-out tradeoff inside ten minutes is itself the signal.
- Did the numbers drive the design, or decorate it? Estimating 87,000 reads/s and then not using it is worse than not estimating. The chain that scores is: 100:1 read ratio → precompute → 150 M followers breaks precompute → hybrid. Each number must eliminate an option.
- Did you reach the hybrid unprompted? Proposing pure fan-out on write and defending it until the interviewer says "what about a celebrity" is a mid-level performance. Anticipating the tail of the follower distribution yourself is a senior one.
- Did you state what you gave up? Every choice here costs something: the hybrid adds merge complexity and two code paths that can disagree; capped feeds mean deep scroll needs a fallback; async fan-out means seconds of staleness. Naming the cost of your own design is the strongest single behaviour in the whole loop.
- Did you separate the media path? Candidates who route 200 GB/s of images through their API tier have quietly designed something that cannot exist. Object storage plus CDN, with the feed returning URLs, should be a throwaway sentence — but it has to be said.
- Did you handle the boring correctness details? Cursor pagination instead of OFFSET, idempotent writes, IDs-not-bodies in the feed. These separate people who have shipped this from people who have read about it, and they cost one sentence each.
- Did you scope, and did you push back? Saying "reverse-chronological for v1, and here's where ranking would slot in" is stronger than hand-waving an ML system you cannot describe. Interviewers are calibrating judgement about what to build now, not enthusiasm for building everything.