Design an ad frequency-capping system

Clarifying questions. What identifies a viewer: an account, a profile, a device? What does a cap count: ads chosen, ads rendered, or ads watched? Are the windows calendar days and weeks, or rolling? Is the cap a hard promise to the advertiser, or best effort? Is a slight over-delivery or under-delivery worse? What do we do if the cap check is unavailable?

Assumed scope: caps are per profile and per campaign, daily and weekly on calendar buckets, enforced when the ad is chosen, a hard-ish cap with small tolerated overshoot, 100 million viewers, 30 active campaigns per viewer, and a pod of up to three ads chosen together. Out of scope: campaign budget pacing, auction logic, targeting, and billing.

Planning figures for the exercise, not facts about any company.

• Decisions: 50 million viewers a day × 20 ad slots = 1 billion decisions ÷ 86,400 s = about 11,600 a second; at a 5× peak, 58,000 a second.
• Lookups: if each decision has 20 candidate campaigns and we look up each cap separately, that is 58,000 × 20 = 1.16 million lookups a second. One record per viewer holding every campaign's counters turns it into 58,000 reads a second, 20 times fewer.
• Viewer record: 30 campaigns × 16 B of counters (campaign id, day count, week count) = 480 B, about 600 B with a short list of recent reservation ids. 100 million × 600 B = 60 GB.
• Writes: each decision reserves a slot (one write). Assume 5% of decisions never play and are refunded later, so 58,000 × 1.05 = about 61,000 writes a second. At the 5,000-a-second planning figure, 61,000 ÷ 5,000 = 12.2, so use 16 shards: about 3,800 writes a second each and 60 ÷ 16 = 3.75 GB each.
• Latency: a same-AZ round trip is 0.5 ms, so one call fits a 10 ms budget with room. A cross-region call at 70 to 90 ms does not, so the check happens in the viewer's region.
• Rolling windows: a log of up to 10 timestamps at 8 B per campaign is 80 B, × 30 = 2.4 KB per viewer, 100 million × 2.4 KB = 240 GB, 4 times the counter design.
• Retries: 3 attempts over 3 layers is 27× load if the store slows, so the hot path makes one attempt.

1. Identity. Key everything by profile id, not device id, so a cap follows the viewer across a TV, a phone and a laptop. The decision request carries the profile id. Signed-out or unknown traffic falls back to a device id and gets a looser, separate cap.
2. Store. A sharded key-value store, 16 shards by hash(profile_id). One record per profile: for each campaign, {day_bucket, day_count, week_bucket, week_count}, plus a short list of recent reservation ids with expiry times. A viewer's whole record is on one shard, so a decision touches one shard. The record has a TTL of 8 days idle, because nothing in it matters after the longest window.
3. Lazy reset. The bucket label (for example 2026-10-19 and 2026-W43) is stored beside the count. If the label differs from the current one, the count is treated as zero and overwritten on the next write. No midnight job is needed.
4. Campaign rules. The cap values (3 a day, 10 a week) are campaign metadata, cached in the decision service for about a minute. They change rarely and are not stored per viewer.
5. Decision path. The decision service gets candidates from targeting, then reads the viewer record in one call and drops every candidate that is at its cap. The auction then picks winners for the pod.
6. Reserve. For the chosen campaigns, one atomic script on that shard re-checks count < cap for each and increments, recording a reservation id with a 2-minute expiry. If a campaign fails the re-check because another decision for the same viewer just took the slot, the service falls back to the next-best candidate once. This closes the race where two concurrent decisions both read 2 of 3 and both serve.
7. Refund. The player sends an impression beacon with the reservation id to an event log. A stream job joins decisions to beacons; a reservation without a beacon after a timeout shorter than the reservation's 2-minute expiry is refunded with a decrement, applied once per reservation id. About 5% of reservations take this path.

Why count at decision time. The next decision for the same viewer may come seconds later, long before any beacon is processed. Counting at the moment of choice means it sees the previous choice. The refund repairs the cases where counting early was wrong.

Hard problem 1: failure. The store call has a 5 ms timeout and one attempt. On failure, fail open: serve the ad without a cap check, and emit a metric for every uncapped decision, because that is over-delivery you owe someone an explanation for. A circuit breaker stops calling a sick shard. The alternative, serving no ads, loses revenue and is worse for most campaigns.

Hard problem 2: regions and travel. Each viewer has a home region, and the cap store is regional. A viewer using the service abroad is served by a nearby region; reading their home record would cost 70 to 90 ms, so the system either uses an asynchronously replicated copy and accepts small over-delivery, or pins decisions to the home region if the budget allows.

Rebuild. The decision and beacon logs are the source of truth. After a store loss, load the last snapshot and replay the log tail.

• Count at decision versus count at impression. Pick decision-time with a refund, because rapid back-to-back decisions see each other. Flip to impression-time counting when advertisers insist that only rendered ads count and over-delivery risk is tolerable.
• Atomic reserve versus check then write. Pick the atomic script: it lives on one shard anyway and removes the double-serve race. Flip to plain check-then-write only if a small overshoot is acceptable and the script's cost becomes the bottleneck.
• Fail open versus fail closed. Pick fail open for revenue, with a metric and an alert on uncapped decisions. Flip to fail closed for campaigns with a contractual hard cap, or serve only those campaigns that have no cap.
• Calendar buckets versus rolling windows. Pick calendar: 60 GB against 240 GB, with a lazy reset. Flip to rolling when the contract says "in any 24 hours", and pay 4 times the state.
• One record per viewer versus one key per campaign. Pick one record: one read, one shard, 58,000 reads a second instead of 1.16 million. Flip if a viewer's record grows beyond a few KB.

Mistakes to watch for:

• Keying by device id, so a viewer on three devices sees the ad nine times.
• One lookup per candidate campaign, which multiplies load by the candidate count.
• Read-then-write without an atomic reserve, so concurrent pod decisions overshoot.
• Counting only when the beacon arrives, so the next decision does not see this one.
• Scheduling a nightly job to reset counters instead of stamping a bucket label.
• Retrying the store call at several layers when it is slow.
• Leaving the failure policy unsaid, then being asked "what if the store is down?" with no answer.
• Forgetting that a refund must be idempotent too.