Design the job-alert feature of a careers site

Questions I would ask: is an alert a saved search (keywords, location, level, remote) or a subscription to a category? How fast must a match arrive: within minutes, or in a daily digest? Email only, or push too? Does an edit to an existing job count as new? How many alerts can one person hold? Do jobs close, and should a closed job still be sent? What consent do we need: double opt-in, one-click unsubscribe?

Assumed scope: create, edit, pause and delete alerts; match new and materially changed jobs; send an immediate email or a daily digest; unsubscribe. Out of scope: search ranking on the site, applications, recruiter tools, and running a mail server.

Planning figures for the exercise, not facts about any real careers site.

• Alerts: 5 million active alerts held by about 3.3 million people (1.5 each).
• Jobs: 50,000 posted or materially changed a day. 50,000 / 86,400 = about 0.6 a second. A recruiter bulk import of 5,000 jobs in 10 minutes is 8 a second.
• Matches: assume the average job matches 200 alerts. 50,000 x 200 = 10 million match rows a day, about 116 a second, or 350 a second at 3x. The bulk import makes 5,000 x 200 = 1,000,000 matches in 600 seconds, about 1,670 a second.
• Alert index: 5 million x 300 bytes = 1.5 GB. It fits in memory on one machine.
• Match table: 10 million rows x 50 bytes = 500 MB a day; 30 days is 15 GB.
• Emails: after digesting, say 1 million a day, about 12 a second. If all digests go out in a two-hour window, 1,000,000 / 7,200 = about 139 a second.
• The polling alternative: run every alert once a day as a search query. 5,000,000 / 86,400 = 58 a second on average, but 5,000,000 / 3,600 = 1,389 a second if they all run in one morning hour, competing with interactive search, and nearly all return nothing.

What the numbers force: nothing here needs sharding; the peak of 1,670 match writes a second is under the 5,000 a second one primary handles. The ratio that matters is 5 million alerts to 50,000 jobs, 100 to 1. Do the matching per job, not per alert.

1. Alert service. CRUD on alerts(alert_id, user_id, query_json, frequency, state, created_at, last_sent_at, unsubscribe_token), one relational primary with replicas. Normalize the query into structured filters (country, city, level, remote, team) plus keyword terms. New alerts start as pending until the email address is confirmed (double opt-in); only confirmed alerts match.
2. Job events. The jobs system publishes job_created, job_changed and job_closed to a durable log, partitioned by job id. The event carries the full job document, so the matcher never races the search index.
3. Matcher (reverse search). Hold the alerts in an inverted index in memory, keyed by facet and keyword, so a job looks up candidate alerts by its location, level and terms and then verifies the full predicate on each candidate. A percolator-style reverse index in a search engine does the same job; the 1.5 GB size means either is easy to host. Rebuild from the alert table at start and apply alert changes from a change feed.
4. Match writes. For each match insert matches(alert_id, job_id, version_class, matched_at, state) with a unique key on (alert_id, job_id, version_class). A replayed event or a second matcher instance inserts nothing new. That unique key is the dedupe for replays.
5. What counts as changed. Only a material change (title, location, level, reopening) bumps the job's version_class, which allows one new match row and so one more notification per alert. A typo fix in the description changes nothing and re-matches nothing.
6. Scheduler. A periodic job selects pending matches per user. Immediate alerts send after a short batching delay (say 2 minutes) so one bulk import becomes one email. Daily digests run at the user's local morning, spread across time zones, which also smooths the 139 a second peak.
7. Send step. Group by user, not by alert: one email with a section per alert. Re-read job state just before sending and drop closed jobs. Cap each alert at, say, 25 jobs per email with a "see all" link. Hand the message to the notification pipeline (preferences, rate limit, provider adapter) with an idempotency key of user_id + send_window.
8. Unsubscribe. Every email carries a one-click link and the standard list-unsubscribe headers, using a signed token that carries the alert id; it works without logging in. Suppression is written first, so a pending send checks it again.
9. Safety net. A nightly reconciliation job searches the job index for postings newer than a watermark and checks they all have match rows for the alerts that should match. This catches matcher outages and event loss; it can use the slow polling approach because it runs once.

Hard problem one: broad alerts. An alert such as "engineer, anywhere" matches almost everything and would send hundreds of jobs a day. Rate-limit per alert (the per-email cap above, plus a daily ceiling) and suggest narrowing in the email.

Hard problem two: duplicates under at-least-once delivery. The log may deliver an event twice and a send may time out after the provider accepted it. The unique match key stops duplicate matches (one per alert, job and version class); the send idempotency key plus a sent_at mark stops duplicate emails in nearly all cases, and the rare remainder is accepted and counted.

• Match per job (reverse search) vs run every alert on a schedule. Reverse matching is event-driven, costs about 0.6 job lookups a second and delivers within minutes, but needs an alert index you maintain. Scheduled polling is simpler and reuses the search engine, but it runs 5 million queries that nearly all return nothing. Pick reverse matching. Flip to polling when alerts are few (tens of thousands) or the query language is too rich to invert.
• Immediate emails vs daily digest. Immediate wins on freshness for competitive roles; a digest wins on inbox fatigue and send cost. Pick a 2-minute batching window for immediate and a digest as the default for broad alerts. Flip to digest-only if complaint rates climb toward the provider's thresholds.
• Store matches vs recompute at send time. Stored match rows give dedupe, audit and "why did I get this" answers for 15 GB a month. Recomputing is cheaper in storage but cannot prevent a repeat after the job index changes. Pick stored rows with a 30-day expiry. Flip to recompute only if match volume grows so much that storage becomes a real cost.
• Single matcher vs partitioned matchers. One in-memory matcher plus a warm standby is enough at 1.5 GB. Pick it. Flip to partitioning the alert index by facet when it passes the memory of one machine or the match rate passes a few thousand jobs a second.

• Running every saved search against the job index on a timer and not noticing that 5 million queries return almost nothing.
• Having no dedupe key, so a replayed event or a retried send emails the same job twice.
• Treating every edit as a new job, so a typo fix re-notifies thousands of people.
• Sending one email per alert instead of one per person, which multiplies complaints.
• Ignoring unsubscribe until the end, or checking it only when the alert is created and not at send time.
• Sending a closed job because the state was read once at match time.
• Sharding the database for a workload that peaks near 1,700 writes a second.