Notify users about their lineup 30 minutes before a match

Questions I would ask: is the trigger a clock (30 minutes before the scheduled start) or an event (the official playing lineups being published)? What does "their lineup" mean: the user's own team, or the real teams' playing eleven? Which channels: push only, or also SMS and email? When do teams lock? How late may a reminder be before it is worthless? Can the start time move, for rain or a delayed toss?

Assumptions: the main trigger is a timer at start minus 30 minutes, and the message carries lineup status if the real lineups are out. A second, event-driven nudge goes to users with a player who is not playing. Push only. Teams lock at the start. In scope: scheduling, fan-out, dedupe, expiry. Out of scope: the contest engine, the content of the lineup feed, and the push provider.

Planning figures for the exercise, not facts about any company.

• Audience: 5,000,000 users, about 100 B per entry row: 500 MB per match. With 3 teams each, 15,000,000 teams × 500 B = 7.5 GB. Both are small enough to scan quickly.
• Delivery window: a reminder is useful for the 30 minutes before lock, but we want most of them out in 5 minutes (300 s): 5,000,000 ÷ 300 = about 16,700 sends a second. Dumping all of it in 30 s would be 166,700 a second, more than I would ask a provider or our own workers to take in one go.
• Work units: in chunks of 1,000 users there are 5,000 chunks, or about 17 chunks a second. If a worker needs about 2 s per chunk, each worker finishes 0.5 chunks a second, so about 34 workers; plan 50 for headroom.
• Timers: one per match, not one per user. A thousand matches a season is a thousand timers; 5,000,000 timers is a storage and polling problem.
• Open spike: if 20% of users open the app within 10 minutes, that is 1,000,000 ÷ 600 = about 1,700 opens a second, at 3 reads each about 5,000 reads a second. Compressed into one minute it is 1,000,000 ÷ 60 × 3 = 50,000 reads a second, past the 20,000 line.

The numbers say: one timer per match, chunked fan-out, a spread-out send, and cached reads for the stampede that follows.

Components. A match schedule table; a scheduler; a fan-out planner; two log topics (match events, and notification tasks); stateless workers; a dedupe store; and the shared notification pipeline that owns preferences, rate limits and the provider calls, as in the notification walkthrough.

Data model.

• match_jobs(match_id, kind, fire_at, version, status) with an index on (status, fire_at). The version increments when the schedule changes.
• entries(match_id, user_id, team_count), partitioned by match and user, so a chunk is a key-range scan.
• sent(dedupe_key) where the key is match_id:user_id:kind, with a TTL of a day.

Scheduling. A scheduler polls every second for rows where fire_at is due and the status is pending. It claims a row with a conditional update that sets a short lease, publishes a reminder.due(match_id, version) event, and only then marks the row done; if the scheduler dies in between, the lease expires and another instance re-claims the row, and a duplicate event is harmless because of the dedupe key. If the match is moved, update fire_at and bump the version; a stale event with an old version is discarded by the planner. Several scheduler instances can run, because the claim is atomic.

Fan-out.

1. The planner receives the event, checks the version and the current time, and loads the lineup if it exists.
2. It splits the entry table into key-range chunks of about 1,000 users and publishes one task per chunk onto the tasks topic, partitioned so the chunks of a big match spread across all partitions.
3. A worker takes a chunk, loads each user's preferences and teams, builds the message (including "2 of your players are not in the eleven" when known), checks the dedupe key, and hands the batch to the notification pipeline, throttled to the provider's rate.
4. The dedupe key is recorded when the provider accepts the message. A crash after sending and before recording can send twice; that is a rare, accepted cost.

Every task carries an expires_at equal to the match start. A worker that picks it up late drops it, since a reminder after lock helps nobody.

The lineup stream. When the real lineup is published, an event on the match-events topic triggers a second planner pass. It joins the lineup with team data and notifies only the users whose team has an absent player, under a different dedupe kind. That is a smaller audience and a much more useful message.

Isolation, small and large. A small contest goes through the same path with one chunk. Give each match its own quota of in-flight chunks so a 5-million-user match cannot starve a small one. Put the reminder at higher priority than marketing, but below one-time passcodes.

The stampede after the push. The lineup is one document per match, cached and served from memory. User teams are cached for a short time. The send is spread over the 5-minute window to flatten the open spike. Per-user limits stop reminders from stacking.

• One timer per match vs per-user scheduled tasks. One timer plus a fan-out at fire time reads fresh data and handles late entrants, but the fire moment is a burst. Per-user tasks spread out naturally but cost millions of rows and go stale when the match moves. Pick one timer; flip to pre-created tasks only if the scan cannot finish inside the window.
• Personalised vs one generic broadcast. A generic message is trivially cheap but ignores whether the user is affected. Personalised costs a join per user. Send the reminder to everyone, with the lineup line added only when it is known, and the separate nudge only to affected users.
• At-least-once with dedupe vs at-most-once. A missed reminder is worse than a rare duplicate, so retry and dedupe. Flip to at-most-once if the channel is expensive or a duplicate is harmful, such as SMS.
• Spread the send vs send as fast as possible. Spreading flattens the follow-up traffic but delays the last user by minutes. Pick a 5-minute window; flip to a faster send for small matches where the stampede is no risk.

• Creating a timer or a scheduled job per user. The scheduler becomes the bottleneck and a reschedule means rewriting millions of rows.
• Looping over all 5 million users in a single process with no chunks, no checkpoint, and no way to resume.
• No expiry, so a delayed worker sends "starts in 30 minutes" after the match began.
• No dedupe key, so a retry sends the same push twice.
• Forgetting that a notification causes traffic: 5 million pushes in 30 seconds produce an app stampede.
• Letting marketing sends share the same queue and block time-critical reminders.