Walkthrough: a notification system

What the question is really about

"Design a notification system" (a chapter of Alex Xu's first volume) is a question about fan-out through other people's services. You will never deliver a push notification yourself: Apple and Google do, a carrier does for SMS, a mail provider does for email. Your system is the part in front of them. It decides whether to send (preferences, rate limits, duplicates), what to send (templates, locale, size limits), when (priority, quiet hours, expiry), and what to do when the provider says no or says nothing. Every one of those is a design decision with a number attached, and the walkthrough below attaches them.

Scope for this chapter: push (iOS and Android), SMS and email, triggered by other services (an order shipped, a password reset) or by a campaign tool (a promotion to millions). Out of scope: building a mail server or a carrier connection. Requirements worth saying aloud:

• Soft real-time. A one-time passcode should arrive in seconds; a marketing push can wait minutes.
• Never send what the user refused. This is the one requirement with legal weight, so it is checked as late as possible.
• At-least-once with de-duplication. Losing a shipping notice is bad, sending it three times is also bad; both are prevented by design, not by hope.
• Isolation. A slow SMS provider must not delay push, and a 20-million-user campaign must not delay a passcode.

Capacity estimation

The inputs are assumptions chosen to be round; copy the method. Provider figures are cited in the next section.

• Transactional baseline: 10 million notifications a day ÷ 86,400 s = 116 a second; at a 5× peak, 579 a second.
• Campaign burst: one marketing push to 20 million devices, sent over one hour: 20,000,000 ÷ 3,600 = 5,556 a second. Throughput here is set by the campaign, not by the baseline.
• Provider ceiling check (push): FCM's default quota is 600,000 messages a minute per project, which is 10,000 a second. The campaign plus the baseline peak is 5,556 + 579 = 6,135 a second, or 61% of quota. Sending the same 20 million, one message per device token, as fast as the default quota allows would take 20,000,000 ÷ 10,000 = 2,000 s, about 33 minutes. Google lets you request an increase of up to 25% when you meet its criteria, which would still leave a floor of about 27 minutes. The estimate also treats all 20 million devices as FCM sends, an upper bound: iOS devices can be sent to APNs directly.
• Concurrency to hold that rate: by Little's law, in-flight requests = rate × latency. At an assumed 50 ms per provider call, 5,556 × 0.05 = 278 requests in flight, which a modest worker pool with multiplexed connections handles.
• Email campaign: 5 million emails over two hours: 5,000,000 ÷ 7,200 = 694 a second.
• SMS is the bottleneck. For 1 million message segments: at 1 segment a second (Twilio's example for a US long code) that is 1,000,000 s, 11.6 days; at a short code's 100 a second, 10,000 s, 2.8 hours. Twilio also queues for at most 10 hours, which at 1 a second is 36,000 segments, so one long code cannot even hold a 1-million-message backlog. The sender type is a requirements decision, not a tuning knob.
• Device tokens: 20 million users × 2.5 devices = 50 million tokens × 200 B = 10 GB. It fits in memory on one cache cluster.
• De-duplication keys: 30 million notifications a day (10 million baseline plus the 20 million campaign) × an assumed 100 bytes a key = 3 GB for a 24-hour window.
• Delivery log: 30 million notifications × 4 status events (queued, sent, delivered, opened) × 100 B = 12 GB a day; 90 days is 1.08 TB, which assumes a campaign that size runs every day, so it is an upper bound. Small enough for one partitioned store, large enough that you will want a retention policy.

The architecture

Why a queue per channel (the queues chapter covers the mechanics): the channels have different throughput ceilings by orders of magnitude (the SMS arithmetic above), different failure modes and different retry policies, so a backlog in one must not block the others. Why lanes: a shared queue means a 4-hour campaign backlog delays a password reset. Put passcodes and security alerts in a high lane with reserved worker capacity, transactional in normal, and campaigns in bulk. Why the service does the checking, not the workers: preferences, de-duplication and rate limits are decisions about the user, made once per notification; workers only know how to talk to a provider.

The four providers, and what each one forces on you

A caution on all four rows: provider limits change, and several are account-specific (quotas, sender throughput). Quote them as "the documented default as of this writing", and design the sender as a thin adapter so replacing a provider changes one module.

Templates: render late, check size, localize

A producer should send an event, not a finished message: {type: "order_shipped", user_id, order_id, carrier, eta}. The service maps the type to a template per channel and locale, with a version. Three decisions:

• Render at the worker, store the template id, version and variables. You keep an auditable record of what was sent without storing millions of rendered bodies, and a fix to a typo applies to jobs still in the queue. The cost is that the worker must have the template; cache it by version.
• Validate at render time, not at the provider. A missing variable should fail the job to the dead-letter queue, not send "Hi {{name}}". Check the push payload against 4,096 bytes and the SMS against your segment budget; an oversized payload is a permanent failure, so it must never enter the retry loop.
• Locale falls back, never fails. User locale, then language, then default. Email carries an HTML part and a plain-text part.

Preferences and opt-outs: checked at the last moment

The preference store is a map from (user, category, channel) to allowed or not, plus quiet hours and a time zone. It has a few properties that matter more than its schema:

• Category matters more than channel. A user can silence promotions on every channel and still want a shipping notice. Mark each notification type as transactional, marketing or security. As a design policy (chosen to satisfy the strictest regimes: US law lets marketing email run on opt-out, while SMS marketing and EU rules need prior consent), marketing requires an opt-in and honours every opt-out; transactional ignores marketing opt-outs (but not a channel-level STOP at the SMS provider, which is blocked regardless); security is not user-silenceable.
• Check twice, and the second check is the real one. The service checks when it creates jobs, which is cheap and saves queue space. The worker checks again just before it calls the provider, because a campaign at 5,556 a second takes an hour to drain and someone will unsubscribe in minute ten. That second check is the last point at which an opt-out can still be honoured.
• Read path is hot, write path is rare. Cache preferences by user; invalidate on write; fail closed (do not send marketing) if the store is unreachable, fail open for security messages.
• Provider-level state wins. Mirror STOP replies, email unsubscribes and complaints into the same store, so one source of truth feeds all channels.
• Quiet hours mean a scheduled job, not a sleeping worker: if it is 2 a.m. for the user, enqueue with a delayed visibility time for 8 a.m., and apply an expiry so a stale message is dropped rather than delivered a day late.

Reliability: retries, duplicates and unknown outcomes

The queue gives at-least-once delivery, so every layer of this system must tolerate a repeat. There are two sources of duplicates: the producer calling twice (it timed out waiting) and the worker retrying after an ambiguous failure.

• De-duplicate at the front door. Every event carries an event id (or you derive a key from user, type and business id, such as the order id). The service does SET dedupe:{id} NX EX 86400 in a key-value store: if the key exists, drop the event. The 24-hour window costs 3 GB at the estimate above. It cannot cover a producer that replays a week later, which is a correctness limit to state.
• Classify errors before retrying. Retryable: timeouts, 5xx, and 429 (with the provider's delay). Permanent: bad requests, an oversized payload (APNs 413, FCM's 4,096-byte limit), 410 Unregistered or a hard bounce (remove the token or address), a STOP block. Permanent errors skip the backoff ladder and update state (delete the token, suppress the address).
• Back off with jitter, and cap the budget. The backoff and jitter maths are in the rate-limiting chapter. The retry rules are per provider. For FCM, Google's guidance is to wait at least 10 seconds before a retry, use exponential backoff with jitter, respect retry-after on a 429, and set a maximum age after which a request is dropped instead of retried forever (it gives no number, and says a request still failing an hour later points to a misclassified error or an outage). For APNs, Apple says 5xx responses can be retried after 15 minutes, with back-off if you like, that a 429 TooManyRequests can be retried with a delay, and that BadDeviceToken, DeviceTokenNotForTopic, Forbidden, ExpiredToken, Unregistered and PayloadTooLarge must not be retried. A passcode should expire much sooner than any of this: set the expiry on the job and have the worker drop it when it is stale, instead of delivering a code that no longer works.
• An unknown outcome is possible. A provider call that times out may have succeeded. Providers differ in whether they de-duplicate; push and email duplicates are annoying, an SMS duplicate costs money, and a duplicate OTP is harmless because the newest code works. Say the trade-off: you accept a rare duplicate rather than a lost message, and you reduce it by keeping the retry count low for non-idempotent channels and using the provider's idempotency or message-id feature where it exists.
• Dead-letter queue, alerting and replay, exactly as in the queues chapter: bounded retries, a DLQ with the error, an alert on depth and a replay path.

Rate limiting, priority and expiry

• Per user. Cap marketing at, say, three pushes a day and one email, so a bug or an overeager campaign cannot flood a person. A token bucket per (user, channel, category) held in the same key-value store does it (the algorithm and Redis implementation are in the rate-limiting chapter). The cap applies to marketing and bulk; never to security.
• Per provider. A global limiter in front of each provider adapter keeps you under its quota. The FCM quota above is 10,000 a second; give the bulk lane a share and reserve the rest so a campaign cannot starve transactional sends.
• Per campaign. Ramp up. Google's own guidance for FCM traffic is to ramp from zero to the target rate over at least 60 seconds, not to spike on the hour, and to step large increases in stages. Treat the first thousand sends of a campaign as a canary: if the failure rate jumps, pause.
• Priority is a scheduling policy. Workers pull high first, with a floor of capacity for normal and bulk so nothing starves, or run separate pools. Pass priority through to the provider too: APNs priority 10 for something the user is waiting for, 5 otherwise.
• Expiry is part of the message. Set a time to live on every job and pass it to the provider (the APNs expiration header, the FCM TTL). A notification that is late enough to be wrong should not be sent.

Delivery tracking and analytics

The honest distinction is between accepted and delivered. A 200 from APNs or FCM, or a 202 from an email API, means the provider took responsibility, not that a person saw anything. Delivery receipts come back later by webhook for SMS and email (delivered, bounced, complained), and push gives you partial signals at best. Design consequences: callbacks are idempotent and monotonic (take the furthest state); store the provider's message id on the "sent" row so a callback can find its job; and record opens and clicks from your own tracking links, which are approximate and affected by mail clients that prefetch. Analytics is then a separate consumer of the delivery log, not a feature of the sending path: stream the log to a warehouse, and keep the send path free of reporting queries.

Dry run: one shipping notice, with a provider error, a duplicate and a bounce

Event E-9001 (order shipped) for user u42, who allows push and email for transactional messages and has SMS off. Retry delay is the 10 second Google minimum plus jitter (the 13 s is illustrative).

Count the outcomes: the user got exactly one push and no email, the duplicate event cost one key lookup, the transient FCM error cost a 13-second delay rather than a lost message, and the bounce permanently improved future sends. What the log cannot tell you is whether the phone showed the push, which is why "delivered" is reported for email and SMS and "sent" for push.

Failure modes and follow-ups to volunteer

• A provider is down. Its queue grows, retries back off, others are unaffected. Fall back to a second provider where the channel allows (a second email or SMS vendor), and never fall back across channels for marketing: an unsent push is not permission to send an SMS.
• Thundering campaign. The ramp, per-provider limiter and bulk lane above. Add a kill switch per campaign.
• Stale tokens. FCM considers a registration stale after about a month without a connection and garbage collects Android registrations after 270 days. Refresh token timestamps from the client, drop stale tokens before sending, and delete on UNREGISTERED. Sending to dead tokens wastes quota and hurts delivery statistics.
• Time zones. "9 a.m. local" for 20 million users is a rolling send across 24 hours, not a burst: it helps with every limit above.
• Observability. Alert on queue depth and age per lane, provider error rates, DLQ depth, bounce and complaint rates (against the 0.10% and 0.30% marks) and the fraction of events dropped as duplicates.

Recognizing it in an unseen problem

• Signals: "send an email or SMS when...", "push notifications", "alerts", "reminders", "marketing campaign to millions", "OTP", or any design that has to talk to an external messaging provider.
• The structure to reach for: events in, preference and rate checks, queue per channel, adapters, callbacks to a delivery log.
• The numbers an interviewer wants: peak rate from the campaign rather than the average, a provider ceiling compared with that rate, and the SMS throughput shock.
• The pitfalls to name unprompted: checking consent only at enqueue time, one shared queue for passcodes and promotions, retrying permanent errors, treating "accepted" as "delivered", and ignoring expiry.
• The follow-up you will get: "how do you avoid sending the same notification twice?" Event-id de-duplication with a TTL, idempotent state updates, and an honest statement that the provider call itself can still produce a rare duplicate.