Design a real-time collaborative document editor

Questions first: rich text, or plain text only? Is offline editing required, with merging on reconnect? How many simultaneous editors per document, and how large can a document get? Do we need version history and per-user undo? Is it fine for every client to be briefly behind the others, so long as they all converge to the same document?

Assumed scope: rich-text documents up to a few hundred pages, online-first with short offline tolerance, live cursors and presence, version history, sharing permissions. Out of scope: spreadsheets and drawings, comments threads, import and export, full-text search over documents. The key requirement is convergence: after the last edit is delivered, every client shows byte-identical content.

Planning figures for the exercise, not facts about any company.

• Sessions: 3 M concurrent users with a document open, about 3 per document, so about 1 M open documents.
• Typing: 10% are typing at any moment, so 300,000 typists. Clients batch keystrokes and send twice a second, so 600,000 operation messages a second inbound.
• Outbound: each batch goes to the other collaborators, about 2 on average, so about 1.2 M messages a second pushed.
• Log writes: 600,000 appends a second is far past 5,000 for one primary. Partition the log by document_id: 600,000 ÷ 5,000 is at least 120 partitions, plan for about 200 so each runs near 3,000 a second.
• Connections: 3 M ÷ 250,000 is 12 gateway boxes at least; plan about 24 for headroom and zone spread.
• Memory: 1 M open documents × 100 KB of snapshot and recent operations is about 100 GB. At 50,000 documents per box that is 20 boxes minimum; plan about 30.
• Storage: if the peak append rate ran all day, 600,000 × 100 B is 60 MB a second, about 5.2 TB a day. Real averages are a fraction of that, and compaction shrinks it further.

What the numbers force: the document is the unit of sharding, each document has exactly one sequencer at a time, and the operation log is partitioned and compacted.

The core decision is how concurrent edits become one document. There are two families. Operational transformation (OT) keeps a central server that orders operations and transforms each incoming operation against the ones it missed. CRDTs give every character or element a unique, ordered identity so that edits commute and merge without a central order. I would pick server-sequenced OT for an online-first product, and explain the flip below.

1. Gateway. Holds the WebSocket and routes each connection to the document's owner server, found through a lookup keyed by document_id.
2. Document owner (sequencer). Holds the document's current text and revision number in memory. It is the only writer for that document. For each incoming batch (client_id, client_seq, base_revision, ops) it transforms the operations against everything committed after base_revision, assigns new revision numbers, appends to the log, acks the author, and broadcasts the transformed operations to the other clients.
3. Client. Applies its own edits instantly. It keeps at most one batch in flight, and buffers later edits locally; when the ack or a remote operation arrives it transforms the buffer against it. This is what makes typing feel local while staying consistent.
4. Operation log. Key (document_id, revision), value is the operation batch plus author and client_seq. The primary key forbids two operations at the same revision, which is the safety net even if two servers briefly both think they own the document.
5. Snapshots. Every few hundred operations or when idle, write a snapshot (document_id, revision) to object storage and truncate older log entries after a retention period. Version history is a list of snapshots.
6. Metadata store. Relational: document, owner, title, permissions, latest snapshot pointer.
7. Presence service. Cursors and selections live only in memory and pub/sub, throttled to a few updates a second. They are never persisted.

Ownership: a lease in a small consensus-backed store (Raft-based) decides which server owns a document, with an epoch number as a fencing token. If an owner dies, the lease expires and another server loads the latest snapshot, replays the log tail, and takes over. Clients reconnect with their last known revision and resend unacknowledged batches; client_id plus client_seq makes the resend idempotent.

Consistency: each document is linearizable at the sequencer, since one process orders everything. Clients see eventual convergence, and their own edits show immediately. The log is replicated synchronously across availability zones before the author is acked, and asynchronously to another region for disaster recovery.

Permissions are checked when the connection opens and again on each batch, so revoking access takes effect immediately. Undo is per user: the inverse of the user's own operation, transformed against everything that happened since.

• OT vs CRDT. OT is compact and has lower per-character overhead, but the transformation functions are famously easy to get wrong and it wants a central server. CRDTs merge anywhere, including peer to peer and offline, but carry metadata per element and need garbage collection. Pick OT with a sequencer for an online-first editor. Flip to CRDT when long offline editing or multi-region concurrent writes are core.
• Single sequencer per document vs multi-writer. One sequencer makes ordering trivial but is a per-document bottleneck and a failover event. Pick it, because a document's write rate is tiny next to the machine. Flip only if one document needs thousands of simultaneous writers.
• Persist every operation vs periodic snapshots only. Per-operation logging is safe for crashes but costs 600,000 appends a second. Pick the operation log with group commit per document and compaction, and flip toward snapshots only if you accept losing the last seconds on a crash.
• Strong consistency vs availability during a region failure. A single owner means a document is briefly unavailable on failover. Pick that, because two owners cause divergence. Flip to CRDTs and accept eventual merge if the product must stay writable everywhere.

• Proposing last-write-wins on the whole document, which silently drops other people's edits.
• Using wall-clock timestamps to order edits.
• Saying "use OT" or "use CRDT" with no account of how a client with an in-flight edit handles a remote one.
• Not choosing one owner per document, so two servers sequence the same document and diverge.
• Persisting cursor positions or sending full document state on every keystroke.
• Having no snapshot or compaction story, so the log grows without limit and opening a document replays everything.
• Forgetting to recheck permissions on an open connection.
• Treating undo as one shared stack for the whole document instead of per user.