Message queues & async processing

Decoupling: what you actually buy

Synchronous calls couple two things that have no business being coupled: the latency a user experiences and the capacity of the slowest downstream service. Put a durable buffer between producer and consumer and you break that link. Four distinct wins come out of it, and naming them separately is what makes you sound like you have run this in production.

• Spike absorption. A queue turns a 20x traffic burst into a growing backlog instead of a cascade of timeouts. Your workers keep running at their sustainable rate; the queue eats the difference. Latency degrades gracefully rather than the system falling over.
• Latency shedding. Image transcoding takes 4 s; sending an email takes 800 ms; reindexing takes 300 ms. None of it needs to happen before the HTTP response. Enqueue in ~2 ms and return 202.
• Retries with durability. If the email provider is down, an in-process retry loop dies with the pod. A queued message survives the deploy, the crash and the region failover.
• Fan-out. One "order placed" event feeds billing, search indexing, the recommendation model, the fraud pipeline and the data warehouse — and adding a seventh consumer needs no change to the producer.

Queue vs log: RabbitMQ and Kafka are not the same shape

A queue broker (RabbitMQ, SQS, ActiveMQ) tracks per-message state. A message is delivered, acknowledged, and deleted. The broker owns the bookkeeping, which buys you per-message acking, arbitrary redelivery delays and routing topologies — at the cost of holding mutable state per message.

A log (Kafka, Pulsar, Kinesis, Redpanda) is an append-only file per partition. Consumers hold an offset — a number — and the broker deletes nothing until the retention window expires. Reads and writes are mostly sequential and lean on the operating system's page cache, which is why a single broker can sustain hundreds of MB/s. Because messages aren't deleted on read, you can rewind the offset and replay a week of history into a new consumer, which is the capability classic queues lack. The two shapes are converging: RabbitMQ Streams add a replicated, non-destructive log (with offset and timestamp replay) inside RabbitMQ, and Kafka 4.2 made share groups ("queues for Kafka", KIP-932) production-ready, giving per-record acknowledgement on top of a log. Pick by the semantics you need, not by the product name.

A cheap tell that you've thought about it: "SQS standard gives me nearly unlimited throughput but no ordering. SQS FIFO orders messages within a message group; by default each queue partition is limited to 300 API calls per second per action (3,000 messages/s with 10-message batches), and high throughput mode raises that to as much as 70,000 TPS (700,000 messages/s batched) in us-east-1, us-west-2 and eu-west-1, scaling with the number of distinct message groups. So I'd use a high-cardinality group id such as the user id, never one global group." Named limits beat adjectives, and the limits are per queue partition, not per group (check the current quota page).

Delivery semantics, done properly

Every distributed queue makes the same unavoidable choice: when the consumer processes a message and then dies before acknowledging it, was the message delivered? You can only pick which side of that ambiguity you fail on.

So the real engineering question is never "which semantic do I pick?" It is "what makes my consumer idempotent?" Three answers, in descending order of how often they're the right one:

• A natural idempotency key. The message carries a stable ID (order ID, event ID). The consumer writes INSERT ... ON CONFLICT DO NOTHING into a processed-events table inside the same transaction as its business write. Second delivery hits the conflict and does nothing. Cheap, obvious, correct.
• Naturally idempotent operations. SET status = 'shipped' is safe to repeat; balance = balance + 10 is not. Prefer absolute writes over deltas when you get to choose the schema — this single habit removes most duplicate-processing bugs before they exist.
• Idempotency tokens at the boundary. When the side effect leaves your system (Stripe charge, email send), pass a client-generated key that the provider de-duplicates on. Stripe's Idempotency-Key exists for precisely this scenario (the header is described in the IETF draft draft-ietf-httpapi-idempotency-key-header, which is an Internet-Draft, last revised in October 2025 and not an RFC).

Retries, dead letters and poison messages

At-least-once means redelivery, and redelivery means you need a policy for messages that will never succeed. A poison message — malformed payload, a referenced row that was deleted, a bug that only triggers on one record — will otherwise be retried forever, and in a Kafka partition it blocks every message behind it. That is head-of-line blocking, and it is how one bad record stops a pipeline for an entire key range.

• Bound the retries. 3-5 attempts with exponential backoff and jitter (see the rate-limiting chapter — the same backoff maths applies). Unbounded retry against a struggling downstream is a self-inflicted DDoS, and synchronised retries are worse than no retries.
• Distinguish retryable from terminal. A 503 or a connection reset deserves a retry. A 400, a schema violation or a foreign-key error will fail identically at attempt 50 — route it straight to the DLQ and skip the backoff ladder entirely.
• Dead-letter queue. After the budget is exhausted, move the message — with the original payload, the error, the attempt count and a trace ID — to a DLQ. Then alert on DLQ depth. An unmonitored DLQ is a data-loss mechanism with extra steps; the DLQ's value is entirely in someone looking at it.
• Make the DLQ replayable. Fix the bug, ship it, drain the DLQ back into the main queue. If replay requires a bespoke script written under incident pressure, you don't have a DLQ, you have a graveyard.
• In Kafka, sidestep head-of-line blocking by publishing the failure to a retry topic and committing the offset, so the partition keeps moving. You trade strict ordering for liveness, deliberately.

Durability and the other knobs worth naming

A queue is only as durable as its replication settings. For Kafka, the usual production shape is replication factor 3, producers on acks=all and min.insync.replicas=2: a write is acknowledged once the in-sync replicas have it, and the partition refuses writes rather than run on a single copy, trading availability for durability. Kafka guarantees a committed message survives as long as one in-sync replica stays alive. Other features that come up: log compaction (keep at least the latest record per key, with tombstones for deletes, which is how a topic becomes a table changelog); visibility timeout in SQS (a received message is hidden for 30 seconds by default, up to 12 hours; if the consumer does not delete it in time it reappears, which is how at-least-once redelivery works there); delay queues (SQS message timers up to 15 minutes); and consumer prefetch in RabbitMQ (cap the unacknowledged messages per consumer so one slow consumer does not hoard work). For long-running multi-step work across services, the pattern on top of all this is a saga: a sequence of local transactions with compensating actions, driven by events.

Backpressure: the queue is a shock absorber, not a landfill

A queue converts an overload into a backlog, which is only progress if the backlog eventually drains. The number to watch is consumer lag (messages behind, or seconds behind) and its derivative: if arrival rate exceeds processing rate at all, lag grows without bound and you have an outage on a delay timer.

• Do the arithmetic out loud. 5,000 msg/s arriving, 40 ms per message per worker → 25 msg/s per worker → 200 workers to break even, and you want ~1.5x headroom to drain a backlog rather than merely hold it. That is the sizing answer an interviewer wants.
• Autoscale on lag, not CPU. Consumer CPU is often near-idle while blocked on a downstream call; lag is the signal that actually tracks user harm. In Kafka, remember the partition ceiling — scaling to 300 pods against 50 partitions leaves 250 idle.
• Push backpressure upstream when the queue is unbounded in practice. Reject or rate-limit at the producer, or shed low-priority work. A 10-hour backlog of stale notifications is worse than dropping them: you'll deliver yesterday's alerts tomorrow.
• Separate queues by priority and by latency budget. One shared queue means a 4-hour bulk-import backlog delays password-reset emails. Different SLOs deserve different queues — this is the cheapest reliability decision in the whole design.

Ordering: per-key is almost always enough

Global total ordering costs you all parallelism — it means one partition, one consumer, one thread. Almost no product needs it. What products actually need is per-entity ordering: user 42's profile updates must apply in order relative to each other; they have no meaningful relationship to user 91's.

So partition by the entity key. hash(userId) % partitions puts every event for a user in one partition, which has one owner in the consumer group, which processes it in offset order. You get ordering where it matters and full parallelism across keys — this is the same insight as sharding by tenant, applied to a stream.

The outbox pattern: fixing the dual-write problem

Here is a bug that survives every code review because it looks correct:

await db.orders.insert(order);        // 1. commits
await kafka.send("order.created", e); // 2. broker unreachable → throws

The order exists and nobody downstream will ever hear about it. Swapping the order just moves the failure: publish first and crash before the commit, and you've announced an order that doesn't exist. There is no ordering of two independent systems that is safe, because you cannot commit to a database and a broker atomically — that is the dual-write problem, and 2PC is not a real answer at this scale (it blocks on coordinator failure; XA transactions exist in some classic brokers such as IBM MQ and ActiveMQ, but few teams run them at scale).

The fix is to make it a single write. Insert the event into an outbox table in the same transaction as the business row. Now either both exist or neither does. A separate relay reads that table and publishes to the broker. There are two kinds. A polling relay queries the table and marks rows sent (or remembers a high-water mark). A log-based relay such as Debezium tails the write-ahead log and sees each insert; its outbox router treats the table as insert-only (updates are not allowed), so there is no sent flag and you simply delete or expire old rows.

Cost and caveats, stated plainly: the outbox adds a write per event and a few hundred milliseconds to a few seconds of publish latency (poll interval; CDC is closer to tens of ms). You must prune old rows or the table becomes your biggest one. A polling relay that tracks the highest auto-increment id it has seen can also skip a row: id 7 may commit after id 8 because its transaction was long-running, so poll with a safety lag or use a log-based relay. And the relay gives you at-least-once, never exactly-once — the event ID in the outbox row is the consumer's de-duplication key, which is why these two patterns are always taught together. The mirror-image pattern on the consumer side is the inbox: record the processed event ID transactionally with the effect.

Dry run: one event surviving a consumer crash and a redelivery, exactly once in effect

The crash at t3 is exactly the case at-least-once delivery is built to survive, and it's also exactly the case a naive consumer gets wrong: the guarantee isn't that E77 arrives once, it's that redelivery at t4 and the duplicate publish at t7 both land on the same processed_events row and produce the same state either way — a duplicate delivery, not a duplicate effect. (A consumer-group rebalance after t6 would not redeliver E77, because the committed offset 43 is already past it; the duplicate here comes from the relay, which is why the consumer still needs its own de-duplication.)

Recognizing it in an unseen problem

• Signals: "send a notification", "generate a report", "process the video", "update the search index", "handle Black Friday traffic", or any single request that fans out to three or more downstream systems. If work can finish after the response, it should.
• The naive design does everything inline and adds servers when it's slow — which scales the fast path and the slow path together, and still fails whenever the slowest third party does.
• Queue or log? Ask "does anyone need to read these events twice, or will someone need them a year from now?" Yes → log (replay, multiple independent consumer groups, backfilling a new service). No, it's just work to be done → queue (simpler, richer routing, no partition-count planning).
• Distinguishing it from a request/response cache: both remove latency, but a cache makes reads cheap while a queue makes writes deferrable. If the expensive thing is a read, you want the caching or CDN chapter, not this one.
• The pitfall to name unprompted: the dual-write. If your design says "save to the DB, then publish an event", say "…via an outbox table, so I'm not doing a dual write" in the same breath. It is one clause and it reliably reads as senior.
• The follow-up you will get: "what if the consumer processes the same message twice?" The answer is never "it won't." It's an idempotency key plus a processed-events table, written in the same transaction as the effect.