Design live market data delivery to many clients

First questions: where do the raw prices come from, and do I receive them as one stream I can read? Do users need every tick, or the latest price a few times a second? Is there a paid tier with deeper data, such as the full order book? Is the audience on mobile networks that drop often? Must a user see the same price as another user at the same instant, or is "within a fraction of a second" enough?

Assumed scope: one normalised price stream comes in; users hold a WebSocket, subscribe to up to 50 instruments, and get the latest price, change and volume. In scope: ingest, fan-out, subscriptions, reconnects, monitoring. Out of scope: order placement, charts and history, the order book, and the exchange connection itself.

These are planning figures for the exercise, not facts about any real broker.

• Inbound: assume 2,000 of the 5,000 instruments are actively trading, at 5 ticks a second each: 2,000 × 5 = 10,000 ticks a second. At the market open, plan a 5× burst: 50,000 a second. At 100 B a tick that is 1 MB a second steady and 5 MB a second at the burst. Tiny.
• Connections: 1,000,000 users on boxes that hold 250,000 connections at best means at least 4 boxes. With 50 KB per connection that is 50 GB of memory in total.
• Naive fan-out: 1,000,000 users × 50 instruments ÷ 5,000 = 10,000 watchers per instrument on average. Sending every tick to every watcher is 10,000 × 10,000 = 100 million messages a second. Not buildable.
• Conflated fan-out: one frame per connection every 250 ms is 4 a second. 1,000,000 × 4 = 4 million frames a second, 25 times fewer. Assume 20 changed instruments × 16 B plus 80 B of framing, about 400 B a frame: 4,000,000 × 400 B = 1.6 GB a second, about 12.8 Gbps in total.
• Spread over 20 gateway boxes: 50,000 connections, 200,000 frames and about 0.64 Gbps each.

The numbers say the input is small and the output is the problem. Fan-out must happen in the gateways, close to the sockets, and it must be conflated by time rather than by tick.

Components. A feed handler that reads the upstream stream and normalises it; a replicated log; a gateway tier holding WebSockets; a small subscription and auth service; a load balancer in front; and metrics everywhere.

Ingest path.

1. The feed handler decodes each tick into a compact record: instrument id, last price, volume, exchange timestamp, the upstream sequence number for that instrument (so both feed handlers stamp the same value), and our own ingest timestamp.
2. It appends to a replicated log (a Kafka-style topic, partitioned by instrument id so ticks for one instrument stay in order). Run two feed handlers, one active and one standby, and let the sequence number drop duplicates after a failover.
3. Every gateway reads the whole stream, each as its own consumer group. At 1 MB a second that costs a gateway almost nothing, and it removes any need to route ticks to the "right" gateway.

Gateway state. Each gateway keeps a table of the latest tick per instrument (5,000 × 100 B = 500 KB) and an inverted index from instrument to the local connections watching it. A connection has a subscription list (capped at 50), a small "pending" map holding at most one entry per instrument, and a send buffer with a hard limit.

Delivery loop. A tick updates the latest table and marks the instrument dirty. Every 250 ms a flush pass walks the dirty instruments, copies the latest value into each watcher's pending map, then writes one binary frame per connection that has changes. Frames carry absolute prices, not deltas, so a dropped or skipped frame heals itself on the next one. On subscribe, the gateway answers at once with the latest values from its table, so the screen is never blank.

Slow clients. Because pending maps hold only the newest value per instrument, a slow phone costs bounded memory. If its send buffer stays full for several seconds, close the socket and let the app reconnect. One bad connection must never stall the flush for the other 50,000.

Hard problem 1: the open and reconnect storm. At the open, a large share of users connect in the same minute, and if a gateway dies, 50,000 clients reconnect together. Clients reconnect with exponential backoff and random jitter; 1,000,000 reconnecting over 60 s is about 17,000 a second, which the load balancer and token check can absorb. Pre-scale the gateway tier before the open instead of reacting to it, and ban deploys during market hours.

Hard problem 2: hot instruments. A few instruments are on far more watchlists than the average. That is harmless here, since each gateway holds only its own share (about a twentieth) of those watchers and the work stays inside the box.

Observability. The ingest timestamp rides with every tick. When the gateway writes it to a socket it records the difference, so you get a histogram of tick-to-socket latency per gateway. Alert on p99 above about 500 ms, on send-buffer disconnects, on the connection count per box, and on log consumer lag. Run a synthetic client in each region that subscribes and checks prices against the source.

• Conflated frames vs every tick. Conflation cuts the outbound work 25-fold and gives a steady rhythm, at the cost of up to 250 ms of added delay and skipped intermediate ticks. Pick conflation for watchlists. Flip to per-tick delivery for a small set of users on a single-instrument depth screen, on a separate tier.
• Every gateway reads everything vs routing by instrument. Reading everything is simple and fails gracefully. Routing saves bandwidth but needs a map of who watches what and a rebalance whenever a gateway moves. Flip to routing when the stream grows past what a gateway can parse (full depth data at hundreds of thousands of messages a second).
• WebSocket vs polling. A socket gives push and low overhead but needs sticky state and careful draining. Polling every second from 1,000,000 clients is 1,000,000 requests a second, which is far worse. Offer a polling fallback only for networks that block sockets.
• Few big gateways vs many small ones. Big boxes mean fewer connections to manage but a larger blast radius and a heavier reconnect wave. Choose about 20 boxes so one failure is 5% of users; flip to bigger boxes if the operations cost dominates.

• Publishing each tick to each user through a general message queue. Count the messages first: it is 100 million a second.
• Putting the latest price in a shared cache and having every client poll it, which turns a push problem into a very large read problem.
• Sending deltas with no recovery path. One lost frame leaves a wrong price on screen until the next full refresh.
• Unbounded per-connection buffers, so one slow phone eats memory on the box.
• Ignoring reconnect behaviour: no jitter, so a single failure turns into a storm.
• Forgetting to measure tick-to-screen latency, so nobody knows the target is being missed.