Message queues & async processing

Decoupling: what you actually buy

Synchronous calls couple two things that have no business being coupled: the latency a user experiences and the capacity of the slowest downstream service. Put a durable buffer between producer and consumer and you break that link. Four distinct wins come out of it, and naming them separately is what makes you sound like you have run this in production.

• Spike absorption. A queue turns a 20x traffic burst into a growing backlog instead of a cascade of timeouts. Your workers keep running at their sustainable rate; the queue eats the difference. Latency degrades gracefully rather than the system falling over.
• Latency shedding. Image transcoding takes 4 s; sending an email takes 800 ms; reindexing takes 300 ms. None of it needs to happen before the HTTP response. Enqueue in ~2 ms and return 202.
• Retries with durability. If the email provider is down, an in-process retry loop dies with the pod. A queued message survives the deploy, the crash and the region failover.
• Fan-out. One "order placed" event feeds billing, search indexing, the recommendation model, the fraud pipeline and the data warehouse — and adding a seventh consumer needs no change to the producer.

Queue vs log: RabbitMQ and Kafka are not the same shape

A queue broker (RabbitMQ, SQS, ActiveMQ) tracks per-message state. A message is delivered, acknowledged, and deleted. The broker owns the bookkeeping, which buys you per-message acking, arbitrary redelivery delays and routing topologies — at the cost of holding mutable state per message.

A log (Kafka, Pulsar, Kinesis, Redpanda) is an append-only file per partition. Consumers hold an offset — a number — and the broker deletes nothing until the retention window expires. Reads are sequential disk scans, which is why a single broker sustains hundreds of MB/s. Because messages aren't deleted on read, you can rewind the offset and replay a week of history into a new consumer, which is the capability queues fundamentally lack.

A cheap tell that you've thought about it: "SQS standard gives me unbounded throughput but no ordering; SQS FIFO gives me ordering per message-group at ~300 messages/second per group (3,000 batched), which is fine per user and useless as a global pipe." Named limits beat adjectives.

Delivery semantics, done properly

Every distributed queue makes the same unavoidable choice: when the consumer processes a message and then dies before acknowledging it, was the message delivered? You can only pick which side of that ambiguity you fail on.

So the real engineering question is never "which semantic do I pick?" It is "what makes my consumer idempotent?" Three answers, in descending order of how often they're the right one:

• A natural idempotency key. The message carries a stable ID (order ID, event ID). The consumer writes INSERT ... ON CONFLICT DO NOTHING into a processed-events table inside the same transaction as its business write. Second delivery hits the conflict and does nothing. Cheap, obvious, correct.
• Naturally idempotent operations. SET status = 'shipped' is safe to repeat; balance = balance + 10 is not. Prefer absolute writes over deltas when you get to choose the schema — this single habit removes most duplicate-processing bugs before they exist.
• Idempotency tokens at the boundary. When the side effect leaves your system (Stripe charge, email send), pass a client-generated key that the provider de-duplicates on. Stripe's Idempotency-Key exists for precisely this scenario.

Retries, dead letters and poison messages

At-least-once means redelivery, and redelivery means you need a policy for messages that will never succeed. A poison message — malformed payload, a referenced row that was deleted, a bug that only triggers on one record — will otherwise be retried forever, and in a Kafka partition it blocks every message behind it. That is head-of-line blocking, and it is how one bad record stops a pipeline for an entire key range.

• Bound the retries. 3-5 attempts with exponential backoff and jitter (see the rate-limiting chapter — the same backoff maths applies). Unbounded retry against a struggling downstream is a self-inflicted DDoS, and synchronised retries are worse than no retries.
• Distinguish retryable from terminal. A 503 or a connection reset deserves a retry. A 400, a schema violation or a foreign-key error will fail identically at attempt 50 — route it straight to the DLQ and skip the backoff ladder entirely.
• Dead-letter queue. After the budget is exhausted, move the message — with the original payload, the error, the attempt count and a trace ID — to a DLQ. Then alert on DLQ depth. An unmonitored DLQ is a data-loss mechanism with extra steps; the DLQ's value is entirely in someone looking at it.
• Make the DLQ replayable. Fix the bug, ship it, drain the DLQ back into the main queue. If replay requires a bespoke script written under incident pressure, you don't have a DLQ, you have a graveyard.
• In Kafka, sidestep head-of-line blocking by publishing the failure to a retry topic and committing the offset, so the partition keeps moving. You trade strict ordering for liveness, deliberately.

Backpressure: the queue is a shock absorber, not a landfill

A queue converts an overload into a backlog, which is only progress if the backlog eventually drains. The number to watch is consumer lag (messages behind, or seconds behind) and its derivative: if arrival rate exceeds processing rate at all, lag grows without bound and you have an outage on a delay timer.

• Do the arithmetic out loud. 5,000 msg/s arriving, 40 ms per message per worker → 25 msg/s per worker → 200 workers to break even, and you want ~1.5x headroom to drain a backlog rather than merely hold it. That is the sizing answer an interviewer wants.
• Autoscale on lag, not CPU. Consumer CPU is often near-idle while blocked on a downstream call; lag is the signal that actually tracks user harm. In Kafka, remember the partition ceiling — scaling to 300 pods against 50 partitions leaves 250 idle.
• Push backpressure upstream when the queue is unbounded in practice. Reject or rate-limit at the producer, or shed low-priority work. A 10-hour backlog of stale notifications is worse than dropping them: you'll deliver yesterday's alerts tomorrow.
• Separate queues by priority and by latency budget. One shared queue means a 4-hour bulk-import backlog delays password-reset emails. Different SLOs deserve different queues — this is the cheapest reliability decision in the whole design.

Ordering: per-key is almost always enough

Global total ordering costs you all parallelism — it means one partition, one consumer, one thread. Almost no product needs it. What products actually need is per-entity ordering: user 42's profile updates must apply in order relative to each other; they have no meaningful relationship to user 91's.

So partition by the entity key. hash(userId) % partitions puts every event for a user in one partition, which has one owner in the consumer group, which processes it in offset order. You get ordering where it matters and full parallelism across keys — this is the same insight as sharding by tenant, applied to a stream.

The outbox pattern: fixing the dual-write problem

Here is a bug that survives every code review because it looks correct:

await db.orders.insert(order);        // 1. commits
await kafka.send("order.created", e); // 2. broker unreachable → throws

The order exists and nobody downstream will ever hear about it. Swapping the order just moves the failure: publish first and crash before the commit, and you've announced an order that doesn't exist. There is no ordering of two independent systems that is safe, because you cannot commit to a database and a broker atomically — that is the dual-write problem, and 2PC is not a real answer at this scale (it blocks on coordinator failure and no cloud broker offers it).

The fix is to make it a single write. Insert the event into an outbox table in the same transaction as the business row. Now either both exist or neither does. A separate relay — a poller, or Debezium tailing the write-ahead log — reads that table and publishes to the broker, marking rows sent.

Cost and caveats, stated plainly: the outbox adds a write per event and a few hundred milliseconds to a few seconds of publish latency (poll interval; CDC is closer to tens of ms). You must prune sent rows or the table becomes your biggest one. And the relay gives you at-least-once, never exactly-once — the event ID in the outbox row is the consumer's de-duplication key, which is why these two patterns are always taught together. The mirror-image pattern on the consumer side is the inbox: record the processed event ID transactionally with the effect.

Recognizing it in an unseen problem

• Signals: "send a notification", "generate a report", "process the video", "update the search index", "handle Black Friday traffic", or any single request that fans out to three or more downstream systems. If work can finish after the response, it should.
• The naive design does everything inline and adds servers when it's slow — which scales the fast path and the slow path together, and still fails whenever the slowest third party does.
• Queue or log? Ask "does anyone need to read these events twice, or will someone need them a year from now?" Yes → log (replay, multiple independent consumer groups, backfilling a new service). No, it's just work to be done → queue (simpler, richer routing, no partition-count planning).
• Distinguishing it from a request/response cache: both remove latency, but a cache makes reads cheap while a queue makes writes deferrable. If the expensive thing is a read, you want the caching or CDN chapter, not this one.
• The pitfall to name unprompted: the dual-write. If your design says "save to the DB, then publish an event", say "…via an outbox table, so I'm not doing a dual write" in the same breath. It is one clause and it reliably reads as senior.
• The follow-up you will get: "what if the consumer processes the same message twice?" The answer is never "it won't." It's an idempotency key plus a processed-events table, written in the same transaction as the effect.