Design a promotional offer notification system

Questions first. How big is the largest audience and how is it defined: a fixed list, or a rule such as "ordered biryani in the last 30 days in this city"? Which channels, and is the choice per user or per campaign? Is the offer capped by budget, for example the first 50,000 redemptions? Must sends be at the user's local time? What is the consent model per channel, and who owns it? Is there a need for a holdout group to measure real lift?

Scope I will assume: campaign creation and scheduling, audience resolution, eligibility checks, frequency caps, push as the main channel with SMS and in-app as options, offer validity and redemption budget, delivery and conversion tracking. Out of scope: the campaign UI, creative design, and the provider connections already covered by the notification platform underneath.

Planning figures for the exercise, not facts about any company. I reuse the notifications chapter's campaign of 20 million devices.

• Send rate: 20 million over one hour is 20,000,000 ÷ 3,600 = about 5,556 a second. Over 5 minutes it would be about 66,700 a second, well past the push quota arithmetic in that chapter (check current docs).
• Work units: chunks of 1,000 users give 20,000 chunks, about 5.6 a second at the one-hour pace. One queue message per chunk, not per user.
• Audience snapshot: 20M × 50 B = 1 GB. Send ledger (one dedupe key per campaign and user, 100 B): 20M × 100 B = 2 GB. Cap counters (one per user per day, 50 B): 1 GB.
• The downstream spike: assume 5% open the push, so 1 million opens. Over an hour that is about 280 opens a second, or 1,400 reads a second at 5 reads each. If the push goes in 5 minutes and opens bunch, it is 1,000,000 ÷ 300 = about 3,300 opens a second, about 16,700 reads a second, close to the 20,000 one database serves.
• Redemptions: if 10% of openers order, that is 100,000 orders, about 28 a second over an hour. One atomic budget counter is enough.

The bottlenecks are provider throughput and the app-open spike, so pace the send and protect the backend.

1. Campaign service. A campaign has an audience definition, a message template (with locale variants), a channel plan, a schedule, an offer id, an expiry and a state: DRAFT, APPROVED, SCHEDULED, RUNNING, PAUSED, DONE, CANCELLED. Approval is a separate step so a typo cannot go to 20 million people.
2. Offer service. offers(offer_id, rule, valid_from, valid_to, budget_total) and offer_grants(offer_id, user_id) with a unique key. Checkout is the source of truth: it validates the grant, the validity window and the remaining budget with an atomic decrement of a counter. The notification only advertises the offer; it never confers it.
3. Audience builder. A batch query against the warehouse writes user ids into a snapshot table or file in object storage, partitioned into chunks of 1,000, and records the snapshot id on the campaign. Freezing the audience makes the campaign repeatable and auditable.
4. Chunk workers. Each queue message is one chunk. For each user the worker runs filters in a fixed order, cheapest first: campaign state (stop if paused), marketing consent for the channel, suppression and quiet hours, already-redeemed, holdout membership, frequency cap. Survivors get a send-ledger entry and a job in the bulk lane of the notification pipeline, with an expiry equal to the offer end.
5. Notification pipeline. The existing per-channel queues, workers and provider adapters from the notifications chapter. The promotional system is a producer, not a second pipeline.
6. Tracking. Delivery receipts, opens, and redemptions (matched through the offer id in the deep link) flow into an event log and a warehouse. Redemption counts per campaign come from checkout, not from clicks.

Idempotency. The send key is campaign_id:user_id, and it is also the notification job id, so the pipeline drops a job whose key it has already accepted. The worker enqueues first and then writes the ledger entry, so a crash between the two can only cause a repeat enqueue, which that de-duplication absorbs, and never a silently skipped user. A chunk redelivered after a crash skips users already in the ledger and re-enqueues the rest. The key's TTL outlives the campaign by a safe margin.

Frequency caps. A counter per (user, day, channel) in a key-value store, incremented when a job is accepted. Caps apply to marketing only. If two campaigns compete for one user's slot, priority and a global daily cap decide; a campaign tagged "high priority" is evaluated first. The cap is an approximate guard: a few extra sends in a race are acceptable, while consent is exact.

Consent is checked twice, as in the notifications chapter: at chunk time and again by the sending worker just before the provider call, because an hour-long campaign outlives many unsubscribes. When the preference store is unreachable, marketing fails closed.

Pacing. The scheduler releases chunks at a configured rate (for example 5,556 users a second), ramps up over the first minute, and treats the first chunks as a canary: if the failure rate jumps, it pauses itself. For local-time sends, the audience is bucketed by time zone and each bucket is scheduled separately, which also spreads load.

Protecting the backend. Because a promotional push is a deliberate load generator, product and platform agree on a rate before launch. Pace sends so the expected open rate stays inside the read headroom, serve the offer landing page from a cache, and put a per-campaign kill switch on the campaign state that every worker checks per chunk.

Expiry. A message about an offer ending at 11 p.m. is worthless at 11:05. Every job carries expires_at; workers drop expired jobs and checkout rejects an expired grant, so both ends agree.

• Frozen audience vs live evaluation. A snapshot is cheap, repeatable and auditable but can go stale by send time. Evaluating the segment per user at send time is fresh but expensive and unrepeatable. Pick a snapshot plus live checks on the few fields that matter (consent, already redeemed); flip to live evaluation for behaviour-triggered messages such as an abandoned cart.
• Fast blast vs paced send. Blasting finishes sooner and lets the offer reach everyone while it is hot, but spikes both the provider and the backend. Pace over an hour or more; flip to a faster pace only for a short flash offer and only after load-testing the open spike.
• Strict vs approximate caps. Exact caps need a transactional check per user and slow the pipeline. Approximate counters allow a rare extra send. Pick approximate for frequency caps; pick strict for consent and for the offer budget, where an error costs money or compliance.
• Push first vs SMS. Push is cheap and fast but only reaches users with the app and permission. SMS reaches more people at a higher price and a much lower per-sender rate. Pick push, with SMS for a small high-value slice; flip toward SMS when the push opt-in rate is low.

• Looping over users in the request. A 20-million-row loop inside one API call; no chunks, no queue, no resume.
• Checking consent only at audience time, not at send time.
• No send ledger, so a crashed worker re-sends the first half of a chunk.
• Letting the notification carry the offer. The checkout service must validate the grant, validity and budget.
• Ignoring the open spike, protecting the sender but not the ordering backend.
• Forgetting expiry, so late messages advertise dead offers.
• Sharing queues with transactional traffic, so a campaign delays a password reset.
• Treating the send as one big job with no pause, resume or canary.