Walkthrough: a payment system

What you are building, and what you are not

"Design a payment system" (it is the subject of a chapter in Alex Xu's second volume) is a question about correctness under failure, not about scale. You are almost never moving money yourself. A payment service provider (PSP; Stripe and Adyen are examples) holds the relationship with the card networks and banks. Your system is the part that decides when to ask the PSP to charge, remembers what happened, tells the order system, and keeps books that agree with the PSP's and the bank's. The scope here is a customer paying a merchant online (the "pay-in" flow); paying sellers out is the same ledger run in the other direction.

• Never charge twice, never lose a charge. Retries, timeouts and duplicate webhooks must not change the outcome.
• Every cent is explainable. For any balance, a list of entries that adds up to it, written once and never edited.
• Availability is second. It is better to say "pending" than to guess. A slow answer is acceptable; a wrong one is not.
• Smallest possible security scope. Card numbers should never touch your servers.

The happy path, using the vocabulary of Stripe's PaymentIntent (other PSPs differ in names and in detail): you create a payment for an amount and currency; the customer's card details go straight from their browser to the PSP, which returns a reference (a token); you confirm the payment; the card issuer authorizes it, which places a hold on the funds, possibly after the customer proves identity with 3-D Secure; you capture the funds, immediately or later; and the money settles to your bank days afterwards. Separating authorization from capture is real: Stripe documents that an authorization for an online card payment is typically valid for 7 days, shorter in some cases (for Visa merchant-initiated transactions the table says 5 days, with the exact window 4 days and 18 hours), and that an expired authorization is released.

Capacity estimation

Assumptions are round numbers; the conclusion is the point.

• Traffic: 1 million payments a day ÷ 86,400 = 11.6 a second; a 10× flash-sale peak is 116 a second.
• In-flight PSP calls: by Little's law, 116 × an assumed 2 s per call = 232 concurrent calls at peak.
• Ledger lines: each successful payment writes 3 lines at capture, 2 for the PSP fee and 2 when the payout lands (the example is below; real payouts are batched, so this attributes a share of one payout to each payment), so 7 lines. 1,000,000 × 7 = 7 million lines a day × 100 B = 700 MB a day, 255 GB a year. Peak capture lines alone: 116 × 3 = 348 line inserts a second, in 116 commits a second. Each payment also touches the idempotency row, the payment row, the state log, the outbox and the processed-events table, perhaps 15 to 20 write statements in all, grouped into a few local transactions per payment. At the peak that is a few thousand statements a second but only a few hundred commits a second, which fits within the course's 5,000-committed-writes-a-second planning figure for one relational primary.
• Webhooks: 4 events a payment (assumed) × 1,000,000 = 4 million a day, 46 a second on average, 463 at the peak.
• Idempotency records: 1 million keys × an assumed 1 KB with the stored response = 1 GB a day; kept for 7 days, 7 GB.
• State-change log: 1,000,000 payments × 6 transitions × 150 B = 900 MB a day.

One limit sits outside your database: the PSP's. Stripe's documented live-mode defaults are 100 API requests a second overall, 25 a second for an individual endpoint (POST /v1/payment_intents is its own endpoint) and 50 requests in progress at once, and Stripe tells you to contact support before a flash sale and to give at least 6 weeks' notice for a large increase. A 116-a-second peak, 232 concurrent calls, and a webhook handler that fetches the object on every event (up to 463 reads a second at the peak) all exceed those defaults, so you request higher limits ahead of time and throttle with a client-side token bucket.

Read the result: one well-run relational database holds all of it. Nothing here needs sharding. The difficulty is entirely in the failure modes, which is why the rest of the chapter has almost no throughput maths.

The architecture

Idempotency keys, end to end

One customer click must produce one charge however many times any step repeats. That is not one idempotency key but a chain, one link per boundary where a retry can happen. The queues chapter argues that exactly-once delivery is impossible and exactly-once effect is built from at-least-once delivery plus idempotent processing; this is that argument applied to money.

The PSP end has rules worth knowing. For API v1, Stripe compares the incoming parameters with the original request and errors if they differ. It can remove keys after at least 24 hours, so a key used later may start a new operation (API v2 differs, with a 30-day window). It does not save a result if the request failed validation or conflicted with one still executing, so those can be retried. And it recommends deriving the key from an object that identifies the operation, such as a shopping cart, or using a random V4 UUID.

Your own idempotency table has to survive a crash between two side effects, which a single "seen this key" flag cannot do. Brandur Leach's well-known design for Postgres records, per key, the request fingerprint, a locked_at time and a recovery point. Work is split into atomic phases, each a local database transaction that also advances the recovery point, and each foreign call (the charge) is its own phase. A retry reads the recovery point and resumes there instead of redoing finished phases. A second request that arrives while the key is locked gets a 409. The call to the PSP carries a key derived from the local key, so a crash during the call is safe to retry.

Dry run: a retry after a crash in the middle of a charge

The customer was charged once and every layer agrees. The crash at step 3 is the case a naive handler gets wrong: it would retry with a fresh key and charge twice.

The payment state machine

• Transitions are compare-and-set. UPDATE payments SET state = 'captured' WHERE id = ? AND state = 'authorized'; zero rows updated means someone else moved it, so re-read. Every transition also inserts an immutable row into a log (who, from, to, reason, provider id).
• Events can arrive out of order. Stripe does not guarantee that webhook events arrive in the order they were generated, and says to track event ids rather than timestamps to detect duplicates. So a webhook is a hint to look: on receipt, fetch the object from the PSP and move your state machine to match, ignoring any move backwards.
• Webhooks are the safety net, not the main path. Stripe retries live-mode events with exponential backoff for up to three days; recommends verifying the Stripe-Signature header, returning 2xx before doing slow work, and processing through a queue. Treat a missing webhook as normal and reconcile (below).
• Amounts are integers in the minor unit. 1099 for 10.99 USD, and 500 for 500 JPY because yen has no minor unit; a few currencies have special cases (Stripe notes Icelandic króna and Ugandan shilling must be sent as two-decimal values ending in 00). Never use floating point for money, and store the currency beside every amount.

Timeouts and unknown outcomes

The PSP call is the one step where you cannot tell success from failure. The principle (a timeout or 5xx is indeterminate, keep the key stable, reconcile when you cannot resolve it) holds for any PSP that supports idempotency keys; the specifics below are Stripe's API v1 behaviour, so check each provider's retention window and replay rules:

• A timeout or a 5xx is indeterminate. The operation might have produced side effects. Treat a 500 as unknown, not as failed.
• After a timeout, retry with the same key and the same parameters while the key is retained (Stripe: at least 24 hours; limit retries of an unresolved operation to that window). If the first request never arrived, the retry runs it once; if it did run, the retry gets the saved result. Do not generate a new key until the first outcome is settled, because a new key can start a second charge.
• A 500 is different: the retry only repeats it. Stripe saves the 500 response and replays it for the same key, so looping on the key returns the same error and tells you nothing new. Do not retry with a new key either. Reconcile instead: retrieve the object, inspect the request in the dashboard and wait for webhook events, because Stripe's own incident recovery may complete the operation later without changing the cached response.
• When the window ends, stop and reconcile. Retrieve the object, check the PSP's dashboard and webhooks, and only then decide. Put your own identifier in the PSP object's metadata when you create it, so a charge that appears through a webhook can be matched to your records, including after the PSP's own incident recovery.
• While a payment is unknown, do nothing irreversible. Do not fulfil the order, do not release the reserved stock, do not email "payment failed". Show "processing". A reaper job asks the PSP about every payment stuck in authorizing or unknown for more than a couple of minutes.
• Distinguish a declined card from an error. A decline is a definitive answer (402 in Stripe's table: valid parameters, request failed): the attempt is over and the payment is left retryable with another payment method (Stripe's PaymentIntent returns to requires payment method), whereas failed in the figure is a payment you have given up on. An error is not an answer.

The ledger: double entry, immutable

A double-entry ledger records every movement of money as a transaction with at least two lines, in different accounts, whose signed amounts sum to zero. Money is never created or destroyed, only moved, so an unbalanced posting shows up as a non-zero sum and a missing movement shows up as an account that should be empty and is not. Balanced but wrong postings (the wrong amount or account) pass the sum check; reconciliation catches those. Stripe describes its own ledger as an immutable, auditable log built on double-entry bookkeeping, where transactions already published cannot be deleted or modified, state is rebuilt by reprocessing events, and problems are fixed by reverting and reprocessing. The same ideas work at any size; the form used here, a correction posted as a new reversing or adjusting entry, is standard bookkeeping practice and this chapter's own design.

Schema, minimal: ledger_transactions(id, idempotency_key UNIQUE, created_at, description) and ledger_entries(transaction_id, account_id, amount_minor BIGINT, currency), with a rule, checked in code and by a deferred constraint trigger (Postgres has no declarative cross-row check), that entries sharing a transaction sum to zero per currency. A balance is the sum of an account's entries (cache it for speed, but the entries are the truth). Posting is idempotent because the transaction key is unique: the outbox relay may repeat itself harmlessly. If a rule such as "a seller's payable cannot go negative" must hold, lock the account row (SELECT ... FOR UPDATE), run the posting at SERIALIZABLE, or use a conditional update on a maintained balance row. A bare check inside the same transaction is not enough: two concurrent postings can each pass it and overdraw the account, the write-skew pattern from the transactions chapter.

Dry run: one 100.00 USD payment through its life

A marketplace takes a 10% platform fee. The PSP fee is illustrative: 2.9% plus 30 cents. Amounts in cents; positive is debit, negative is credit.

The fee is 2.9% of 10,000 = 290, plus 30, = 320. After all three transactions the clearing account is 10,000 − 320 − 9,680 = 0, which is the check: a non-zero clearing balance means money the PSP has not yet paid, or paid without a record. (Stripe's ledger post uses clearing accounts the same way, flagging non-zero balances.) A full refund later posts +9,000 to seller_payable, +1,000 to platform_revenue and −10,000 to psp_clearing, again summing to zero; whether the PSP returns its fee depends on the provider's terms.

Exactly-once effect: outbox, saga and the order flow

The payment state change and its follow-on effects (a ledger posting, an order update, an email) live in different systems, so they cannot share a transaction. The outbox from the queues chapter is the fix: the service writes the new payment state and an outbox row in one local transaction, and a relay publishes the row at least once. Every consumer is idempotent by key, as in the table above, which is how an at-least-once pipeline produces an exactly-once effect.

An order is a multi-step process spanning services that each commit separately, which is a saga (see the transactions chapter): a sequence of local transactions with a compensation for each step.

Authorizing first and capturing at the end makes the common failure cheap: if the warehouse cannot ship, you cancel a hold, and the customer's money never moved. Capturing is the step that is hard to reverse, so it goes last, inside the authorization window (typically 7 days for online cards).

Reconciliation: assume something is wrong and look

Reconciliation is a scheduled comparison of three records of the same money: your ledger, the PSP's reports of what it processed and settled, and the bank statement of what arrived. It exists because every other mechanism (retries, webhooks, state machines) can still leave a gap, and money gaps must be found by a process rather than by a customer. Stripe's description of its own data-quality work is the model: it checks that fund flows clear (non-zero clearing balances flag problems), that events arrive on time against a threshold, and that every identifier in one system has a matching event in the other; and it fixes problems by reversing and reprocessing, not by editing history.

Dry run: one day of seven payments

Check the totals. Our captured ledger for P1, P2, P3, P5 and P6 is 10,000 + 2,500 + 7,000 + 3,000 + 9,900 = 32,400 (P4 and P7 are absent); the PSP's succeeded charges for P1, P2, P4, P5, P6 and P7 are 10,000 + 2,500 + 4,200 + 3,000 + 9,090 + 1,500 = 30,290, and its net of the 3,000 refund is 27,290. The gross gap, ours minus the PSP's, is 32,400 − 30,290 = 2,110, which decomposes exactly: +7,000 (P3) − 4,200 (P4) + 810 (P6) − 1,500 (P7) = 2,110. The net gap is 2,110 + 3,000 (the P5 refund) = 5,110. If a gap does not decompose into named payments, the reconciliation itself is wrong.

PCI scope and tokenization, at interview depth

PCI DSS is the standard for anyone who stores, processes or transmits cardholder data. Three facts carry the design:

• Never store sensitive authentication data after authorization. The PCI Council's fact sheet is explicit that data on the full magnetic stripe or chip, and the printed 3 or 4 digit card validation code, must not be stored after authorization. The primary account number may be stored only if protected, and the sheet lists strong cryptography among its methods; the name, expiry and service code need protection when stored with the number. The sheet's first advice is not to store cardholder data unless necessary, and its list of data never to store after authorization also includes the PIN.
• Scope is what you touch. Stripe's integration guide says a business handling raw card numbers on its own payment page might need to meet more than 300 security controls, and that its lower-risk integrations collect card details and send them straight to Stripe without passing through the merchant's servers, reducing the obligations. That is the design here: PSP-hosted fields or a hosted checkout page.
• Tokenization replaces the number with a reference. The browser sends the card to the PSP; the PSP returns a token or payment-method id; your system stores that id plus non-sensitive details. Stripe says the card type, last four digits and expiry it returns are not subject to PCI compliance and can be stored. A stolen token table is not a stolen card table, because a token is typically usable only with your PSP credentials (check your provider's rules).

Operational consequences: scrub card-like numbers from logs and error reports, never put them in a URL or an analytics event, serve every payment page over TLS 1.2 or above, verify webhook signatures and restrict webhook sources. Do not claim that using a PSP makes you "PCI compliant" in general; it shrinks your scope, and you still have obligations. You should say so out loud.

Recognizing it in an unseen problem

• Signals: "payments", "checkout", "wallet", "transfers", "billing", "refunds", "marketplace payouts", "ledger", "balance", or any design where a duplicate or a lost write costs money.
• The structure to reach for: PSP integration with idempotency keys, a forward-only state machine, a double-entry ledger, an outbox, a saga and reconciliation.
• The numbers an interviewer wants: a modest peak payment rate that proves scale is not the problem, then the failure arithmetic: retries, a gap that decomposes into named payments, and a clearing account that returns to zero.
• The pitfalls to name unprompted: a fresh idempotency key on a retry, treating a timeout as a failure, storing money as floats, updating ledger rows in place, trusting webhook order, and capturing before the rest of the order is certain.
• The follow-up you will get: "the PSP times out: what do you tell the customer?" "Processing", retry with the same key, check the PSP, reconcile, and never charge again until the first outcome is known.