Design a Messenger-style chat app

First questions: one-to-one only, or groups too, and how large can a group get? Are messages text only, or media too (media would go through object storage and a CDN, with only a reference in the message)? Do we need multiple devices per user, and message history sync on a new device? How strict is ordering and delivery: can a message be lost, can it be shown twice? Is end-to-end encryption required (it changes what the server can do)?

Assumed scope: text messages with attachment references, one-to-one plus groups up to a few hundred, multi-device, offline delivery, sent/delivered/read receipts, history. At-least-once delivery with client-side de-duplication, ordering within a conversation. Out of scope: voice and video calls, end-to-end key exchange, search over content, spam detection.

Planning figures for the exercise, not facts about any company.

• Messages: 20 B a day ÷ 86,400 is about 231,000 a second on average, about 700,000 at a 3x peak.
• Connections: 100 M concurrent ÷ 250,000 per gateway box is 400 boxes (200 at 500,000). Memory check: 100 M × 20 KB is about 2 TB across the fleet.
• Storage: 20 B × 200 B per message row is 4 TB a day. Times 365 is about 1.5 PB a year, and with 3 replicas about 4.4 PB.
• Writes: 700,000 a second at peak ÷ 5,000 per partition is at least 140 partitions. Plan for about 300 so each runs near 2,300 a second with headroom.
• Receipts: unbatched, each message adds up to two more events (delivered, read), about 462,000 a second at average load (231,000 × 2). Acknowledging "up to sequence number N" collapses that.
• Presence: 100 M heartbeats every 30 s is about 3.3 M a second. That cannot go through a database; piggyback it on the socket's ping and keep the routing entry in memory.

What the numbers force: a sharded, partitioned message store (one primary at 5,000 writes a second is nowhere close), a few hundred stateful gateways, and a routing registry kept in memory and treated as a cache.

Components:

• Gateways. Hold the WebSocket, terminate TLS, authenticate, translate frames to internal calls. They are the only stateful tier, and they stay thin so a restart costs little.
• Chat service. Stateless. Validates membership, then appends the message to a durable log partitioned by conversation_id.
• Sequencer / persister. One consumer owns each log partition, so it is the single writer for those conversations. It assigns the next per-conversation sequence number, writes the message, and then acks the sender. A lease epoch fences a stale owner after failover, and the new owner resumes from the last stored seq, so two owners never hand out the same number. Ordering is therefore total inside a conversation, which is the only place anyone can observe it. Never order by client timestamps.
• Message store. A wide-column or LSM-based store (Cassandra-style), partition key (conversation_id, time bucket), clustering key seq. Bucketing keeps a very busy group from becoming one unbounded partition. Reads are "the last 50 before seq N", usually one range scan within a bucket.
• Inbox table. Per user: conversations sorted by last activity, with last_read_seq and last_delivered_seq per device.
• Registry. user_id and device to gateway node, in a sharded in-memory store with a TTL slightly over twice the heartbeat. It is a cache: if lost, gateways repopulate it on their next heartbeat.

Send path: the client generates a message UUID and sends over the socket. The sequencer de-duplicates on (conversation_id, client_msg_id), so a retry after an ambiguous timeout returns the original sequence number instead of posting twice. It persists the message and acks the sender (the "sent" state). Then, for each recipient device, look up the registry, publish to that gateway's topic, and the gateway writes the frame. The device acks with the highest seq it has received, which moves the "delivered" state.

Offline path: no registry entry means no live socket. The message is already durable. Send a push notification through the platform push service as a wake-up only. On reconnect the device asks for (conversation_id, seq > last_seen_seq) for each conversation with unread activity and pulls the gap. Multi-device uses exactly the same pull, since each device tracks its own cursor.

Groups: a small group fans out on write to member inboxes. A large group stores the message once and sends only a "new seq" nudge, with members pulling. Read receipts in large groups degrade to counts.

Hard problem one is the reconnect storm: when one gateway holding 250,000 sockets restarts, all clients retry at once. Require jittered exponential backoff on the client, rate-limit new connections at the gateway, and drain gateways gradually during deploys. Hard problem two is the ack ordering: ack the sender only after the message is durable, so a crash never loses an acked message; the cost is a little added latency on every send.

Consistency: per-conversation linearizable writes through the single sequencer, eventual propagation to recipients, and read-your-writes for the sender because the sent message is appended to its local view with the returned seq.

• Server-assigned sequence vs client or hybrid timestamps. Timestamps need no coordination but clocks drift and can be set arbitrarily. Pick a per-conversation sequencer because ordering is only visible within a conversation. Flip to hybrid logical clocks if conversations must be writable in several regions at once.
• Push to inbox vs pull by cursor. Push is quick for small groups, but costs N writes per message. Pull stores once. Pick push for small groups and pull for large, with the threshold set by members times message rate, and flip it as group sizes grow.
• Ack after durable write vs ack on receipt. Acking early is faster but can lose an acked message in a crash. Pick ack after durable write, since a lost message is the worst chat bug. Flip for ephemeral features like typing indicators, which need no durability.
• Wide-column store vs a sharded relational database. Relational gives transactions and easy ad-hoc queries; a wide-column store gives cheap sequential writes and linear scaling for append-heavy history. Pick the wide-column store at this write rate and flip back if volume is small or you need joins.

• Drawing "WebSocket server and a database" with no answer to how a message reaches the specific gateway holding the recipient's socket.
• Using client timestamps or database auto-increment IDs for ordering across partitions.
• Skipping idempotency, so a retried send posts the message twice.
• Not saying which component owns ordering, so two servers can hand out the same sequence number.
• Acking the sender before the message is durable.
• Treating presence as a table update per heartbeat, which is millions of writes a second.
• Forgetting the offline path: push notifications are a wake-up, not the delivery channel.
• Not planning for a reconnect storm after a gateway restart.
• Putting one conversation in one unbounded partition.