Design an engagement-based popular content feed

Questions first: what is the unit being ranked (page, post, project)? Is "popular" global, per organization or per team? What window: the last hour, 24 hours, 7 days? How fresh must it be, seconds or minutes? Do different actions count differently? Must the feed hide content the viewer cannot access? Is it personalized?

Assumed scope: a non-personalized "popular" list per scope (an organization or a space), with a recency-weighted score, refreshed within a few minutes, filtered by what the viewer may see. Out of scope: personalized ranking and machine-learned recommendations.

Planning figures for the exercise, not facts about any real product.

• Events: 100 million a day ÷ 86,400 = 1,157 a second; a 5× peak is about 5,800 a second. At 100 B an event, that is 10 GB a day of raw log. The peak is above the 5,000 write line, so events go through a log and a stream job, not row updates in a database.
• Hourly buckets: one row per (item, hour) with a weighted count, 40 B. The worst case is one row per event: 100 million × 40 B = 4 GB a day, 28 GB for a 7-day window. Real data is far smaller because many events hit the same item and hour.
• Active items: assume 10 million items get engagement in a day. A score record per item (score, last update, counts by type) at 100 B is 1 GB. That fits in memory.
• Reads: 5 million users × 4 feed loads = 20 million a day ÷ 86,400 = 231 a second; a 5× peak is about 1,200 a second. That is far below the 20,000 line, so the read side needs a cache for latency and protection, not for capacity.
• Decay: with a 12-hour half-life, an event counts 100% now, 50% after 12 hours, and 25% after 24 hours.

The conclusion: the write side is the stream job; the read side is a small precomputed list per scope. Computing the ranking per request would be both unnecessary and wasteful.

Pipeline.

1. Capture. Services emit an engagement event (event_id, user_id, item_id, scope_id, type, ts) to a log, partitioned by item_id so one task sees all of an item's events.
2. Stream job. It drops duplicates by (user_id, item_id, type) within an hour (one person refreshing a page 100 times counts once), ignores the author's own actions, and applies weights (for example view 1, reaction 3, comment 5; tunable). It sums into one-hour event-time windows and emits the running total for the open hour about every minute.
3. Bucket store. Each emission is an upsert to buckets(item_id, hour) = weighted_count. Because it overwrites with a cumulative total, a replay or a late update is harmless, the same idea as the ad-click example in the stream chapter. Events later than the allowed lateness go to a side output and are ignored for the live feed.
4. Ranker. Every few minutes, for items whose bucket changed, compute score = sum(bucket × 2^((bucket_hour − ref_hour) / 12)) over the last 168 hourly buckets (7 days), a cheap read of at most 168 rows. This is the 12-hour half-life decay written against a fixed reference hour, so scores written at different times compare directly and an item that did not change keeps a correct place without being re-scored; rebase ref_hour and rewrite the scores now and then, because they only grow. Write the score into a sorted set per scope.
5. Publish. A refresher writes the top 200 for each scope into the cache as one list. Readers never compute anything.

Read path. GET /popular?scope=... reads the cached top 200, filters out items the viewer cannot see (a batched permission check, as in the search chapter), and returns the first 20 with a cursor. The list holds 200, not 20, because filtering can remove many. If fewer than 20 survive, fetch the next 200 for that scope straight from the sorted set.

Hard problem 1: hot items. A viral item receives a large share of events. Because the log is partitioned by item id one task could be overloaded; use the two-stage pattern (each task pre-aggregates, then a merge step sums partial totals per item), as in the stream chapter. Hard problem 2: gaming and bias. Count unique users, cap what any one user adds, and discount bots and the author's own actions. Normalize by audience when comparing scopes of very different sizes, or each scope is ranked only within itself, which is what this design does.

Staleness and cache. A few minutes of staleness is fine for "popular". Refresh the cache in the background on a timer instead of letting the first reader after expiry compute it, which avoids a stampede. If the search system needs the score as a ranking boost, update that field in the index in bulk every few minutes, not on every event.

Exact windows. For a "top this week" view that must be explainable, sum the 168 buckets with no decay. Same data, different scoring function.

• Decayed score vs exact sliding window. A decayed score is cheap, smooth and needs no window boundaries; an exact window is easier to explain ("views in the last 24 hours") but needs the buckets and has cliffs when old events drop out. Use decay for the default feed; flip to exact windows when users or auditors need the literal count.
• Stream aggregation vs counter increments in a database. Per-event increments are simple but cause hot rows and break under replay. A stream job with idempotent upserts costs more machinery and handles replays and hot items. Pick the stream job past a few thousand events a second.
• Precomputed top list vs on-demand query. Precomputing makes reads trivial and stale by a few minutes; on-demand is fresher but runs a sort per request. Precompute; flip only if the feed must be live to the second.
• Filter at read time vs per-user lists. Read-time filtering keeps one list per scope; per-user lists make each page exact but multiply storage by users. Filter at read time with an oversized candidate list.

• Doing count(*) over an events table on every request, or ranking in the database at read time.
• Incrementing one counter row per item from every request, creating hot rows and double counting on retry.
• Ignoring time: ranking by all-time counts, so old content never leaves the feed.
• Counting every raw view, so one refreshing user or a bot can push an item to the top.
• Forgetting permissions, then showing a title the viewer should not see; or filtering after taking only 20 candidates and returning an empty page.
• No plan for one item that gets a large share of all events.
• Updating a search index on every event instead of in periodic bulk batches.