Design a shopping system that survives a big sale event

I would ask: how much bigger is the peak than a normal day, and how sudden (a ramp, or a spike at a published start time)? Which products are scarce and how many units? Must stock be exact, or may we occasionally oversell and apologise? Which features are essential during the event: browse, cart, checkout, payment, order tracking? Do we run in one region? Is there an existing payment provider and warehouse system whose limits we inherit?

I would assume one region with several zones, a 20× average day and a 3× spike in the first minute, exact stock for limited items, and an external payment provider. In scope: browse, cart, inventory, checkout, protection of the core path. Out of scope: search ranking, recommendation models and warehouse routing.

Planning figures for the exercise, not facts about any company.

• Page load: a normal day of 50 million page views is about 50 × 12 = 600 a second. At 20× it is 12,000 a second; at a 3× spike, 36,000 a second.
• Backend reads: if a page triggers 5 backend reads (product, price, stock badge, reviews, recommendations), that is 36,000 × 5 = 180,000 a second. One database serves about 20,000 simple reads a second, so this cannot go uncached.
• Cache arithmetic: at a 99% combined CDN and cache hit rate, 1,800 reads a second reach the database. At 90% it is 18,000, right at the limit. Reason in miss rate: the gap between 99% and 90% is ten times the load.
• Orders: 1 million orders in the first hour is 1,000,000 ÷ 3,600 = about 278 orders a second. At about 2 commits per order that is roughly 556 commits a second, comfortably under 5,000.
• Hot item: 500,000 buyers go for one product in a minute: 500,000 ÷ 60 = about 8,300 decrement attempts a second on one stock row. Split into 20 buckets and each sees about 420 a second.
• Retries: 3 attempts across 3 layers is 27× the load, which is how an overload turns into an outage.

The conclusions: absorb browsing at the edge and in caches, treat the hot SKU specially, and stop retries multiplying.

Layers, outside in.

1. Edge. The CDN serves images, scripts and pre-rendered sale pages, which are pushed out before the start time, with an origin shield so a miss storm cannot reach the origin. Product pages split into a long-cached static part and a small JSON call for price and stock, cached for 1 to 5 seconds. The stock shown is coarse: in stock, low, sold out, never an exact count.
2. Gateway. Token-bucket limits per user and per IP, bot filtering, and a concurrency limit per service. Under overload it sheds low-priority traffic first and answers 503 with a retry hint.
3. Services. Catalogue (read-only, heavily cached), cart (keyed by user, TTL, fine to lose), inventory, orders and payments. Recommendations, reviews and personalisation are optional: each call has a short timeout (around 100 to 200 ms), a circuit breaker and a static fallback, and checkout never depends on them.
4. Data. One relational primary per domain with replicas for reads, a separate search cluster, and a queue between the order commit and everything slow.

Checkout write path. The client sends an idempotency key per checkout. The order service, in one transaction, reserves stock with a conditional update (available >= qty), writes the order as pending and writes an outbox row. Then it calls the payment provider with a key derived from the order id. On success it confirms the order. On failure or expiry (say 10 minutes) it releases the reservation. Invoices, warehouse messages, emails and analytics read the outbox through a queue, so the warehouse consumes at its own pace and a slow downstream never blocks a customer.

Stock rule. Reserve at order placement, not at add-to-cart. Carts are cheap and abandoned constantly; reserving there would lock stock that nobody pays for.

Hard problem 1: the hot SKU. Split its stock into buckets (20 rows of 5,000 for a 100,000-unit item) and pick a bucket per request. Put an atomic counter in an in-memory store in front as an admission gate: only requests that decrement it successfully go on to the database (if the chosen bucket is empty, try another), a released or expired reservation adds the unit back to both the counter and the database, and the counter can be rebuilt from the database after a restart. When it reaches zero the product flips to sold out at the edge. Everything after that is a cheap cached answer, not a lock queue on a dead item.

Hard problem 2: keeping checkout alive. Priorities decide what to drop: shed recommendations and suggestions first, search second, checkout last. Retries happen at one layer only (the client, with backoff and jitter), and servers enforce a retry budget; otherwise 3 attempts across 3 layers is 27 times the load on a service that is already struggling.

Before the event. Autoscaling reacts in minutes and the spike arrives in seconds, so scale up ahead of time and warm caches. Load test at 1.5× the forecast, freeze deploys, give every non-essential feature a kill switch, and tell the payment provider the expected rate in advance, since its limits apply to you too.

• Reserve at cart vs at checkout. Cart reservation is friendlier but ties up stock for people who never pay. Pick checkout; flip to cart-time holds with short timers for extremely scarce, high-value items.
• Pre-scaled capacity vs autoscaling. Autoscaling is cheap on normal days but too slow for a spike that arrives in seconds. Pick pre-scaling for the known event; keep autoscaling for the unexpected remainder.
• Exact stock display vs coarse badge. Exact counts cost a read of the hottest row on every page view. Pick coarse states with a few seconds of cache; flip to exact only on the final purchase step.
• Waiting room vs plain load shedding. A queue is fairer and protects checkout, but adds a component and delays everyone. Plain shedding is simpler but random. Pick a waiting room only for the hottest launches, and rate-limit everywhere else.
• Synchronous order confirmation vs queue-then-confirm. Confirming only after the order row commits is honest and simple; accepting the request into a queue and confirming later absorbs bigger bursts but leaves customers unsure. Pick the synchronous commit, since 278 orders a second is easy; flip to queue-then-confirm for a single item that draws tens of thousands of buyers a second.

• Saying "add autoscaling" and stopping. Spikes outrun it, and a database primary cannot be scaled up in seconds.
• Caching everything for a long time, including price and stock, or caching nothing and hitting the database with 180,000 reads a second.
• Locking stock at add-to-cart.
• Letting checkout call recommendations or reviews synchronously with no timeout or fallback.
• Retrying at every layer with no budget.
• Ignoring the single hot row; sharding the database does not help when every request wants the same SKU.
• Forgetting that the payment provider and warehouse systems have their own limits.
• Never saying how the design degrades. A sale design with no list of what to switch off first is only half finished.