Design an end-to-end UPI-style payment flow

First, whose backend are we building: the payer's app, the central switch, or a bank? The answer changes everything. Do we hold the customer's balance (a wallet), or does the money sit only in bank accounts? What does the switch give us: a synchronous accept, a later callback, a status query by reference? How is the user authenticated, and who sees the secret PIN? What is the failure the business fears most: a double debit, a lost credit, or a wrong "failed" screen?

I would assume we are the payer's app backend, money stays in the banks, the switch accepts a request, returns a pending state and calls back, and we can query by our reference. This is a generic design; check the real scheme's rules before relying on any detail. In scope: payee lookup, payment creation, submit, async result, status, receipts, reconciliation. Out of scope: bank internals, disputes, onboarding and KYC.

These are planning figures for the exercise, not facts about any company.

• Volume: 10 million payments a day ÷ 86,400 s = about 116 a second. A salary day or festival at 10x: about 1,200 a second.
• Commits: about 4 per payment (created, submitted, result applied, receipt outbox): 1,200 × 4 = 4,800 commits a second at the peak. That is just under the 5,000 figure for one primary, with no headroom. I shard by payer from the start and plan 4 shards.
• In flight: if the switch usually answers within 5 s: 1,200 × 5 = 6,000 payments in flight at the peak.
• Status reads: about 3 polls per payment: 3,600 a second. Easy, and push can replace most of them.
• Retries: if the app, our backend and the gateway each retry 3 times, a struggling bank sees 3 × 3 × 3 = 27x: 1,200 × 27 = 32,400 requests a second. Retry at one layer only.
• Storage: 10 million × 1 KB of payment row plus state log = 10 GB a day, 3.65 TB a year. Idempotency records: 10 million × 0.5 KB = 5 GB a day; kept 7 days, 35 GB.

Components. Mobile app with a secure input component; API gateway; payment service with sharded relational storage; a switch adapter (signing, mutual TLS, retries); a callback receiver; a status checker; a fraud and limits service; a notification service; and a reconciliation job.

Data model.

• payments(payment_id, payer_id, payee_address, amount, state, switch_ref, idem_key, created_at) with a unique index on (payer_id, idem_key). The switch_ref is the end-to-end reference we generate and send to every party; it encodes the shard, so a callback finds its row without a lookup table.
• payment_events(payment_id, event_id, type, ts): append-only audit trail, unique on (payment_id, event_id).
• States: CREATED, SUBMITTED, SUCCESS, FAILED, UNCERTAIN. Transitions only move forward.

Flow.

1. Resolve the payee. The app asks the switch to resolve the payment address and shows the verified name for confirmation.
2. Authorise. The user enters the PIN in a secure component that encrypts it on the device for the bank. Our servers never see it in the clear; the device and session are bound and tokens are short-lived.
3. Create. The app sends the payment with an idem_key made once per intent. One transaction checks limits and velocity, inserts the row in CREATED, and writes an outbox row. A repeated key returns the existing payment.
4. Submit. The adapter signs and sends the request with switch_ref, a timeout, and one retry layer. The row moves to SUBMITTED just before the send, so a crash after the send leaves a row the status checker will find.
5. Result. The switch returns pending or final. Final results arrive by callback: verify the signature, dedupe on the event id, and apply with a conditional update that only moves forward. The payment event and the receipt outbox row commit together.
6. Tell the user. A push and a status endpoint; the app shows "processing" until the row is final.

Hard problem 1: the unknown outcome. A timeout means we do not know whether the bank debited. Never show "failed" and never let the user pay again as a fresh payment. Mark UNCERTAIN, show "we are checking", and let the status checker query the switch by switch_ref with backoff (it also scans SUBMITTED rows that have been quiet too long) until a final answer arrives. A payment stuck for too long goes to an operator queue.

Hard problem 2: duplicates at every hop. The chain is: client key, our unique index, the reference sent to the switch, and the callback event id. Each layer must be safe to see the same message twice. The payments walkthrough covers this chain in detail.

Hard problem 3: partial failure. If the debit happened but the credit did not, the switch and banks run a reversal. We accept the reversal event, move the payment to FAILED with a reversed note, and tell the user the money is going back, without promising a timeline we do not control.

Reconciliation. Daily, compare our payments with the switch's settlement reports and, where available, bank records, keyed by switch_ref. Differences become tickets, not silent fixes.

Security. TLS everywhere, signed messages to the switch, per-user velocity limits, a fraud score before submit, and logs with no secrets or full account numbers.

• Wait for the final result vs return pending at once. Holding the request open for the bank gives a simple client but ties up threads and hides slow banks. Pick returning pending quickly and delivering the result by push with a polling fallback. Flip to a short synchronous wait only for tiny flows where the p99 is well under a second.
• Fraud check inline vs after submit. Inline blocks bad payments before money moves but adds latency and a dependency. After-the-fact is cheaper and misses the damage. Pick inline with a tight timeout; if the fraud service is down, fail closed for large amounts and open for small ones, with a limit.
• Shard by payer vs by payment id. Payer keeps history, limits and velocity checks on one shard. Payment id spreads load evenly but makes those checks cross-shard. Pick payer, and encode the shard in the reference so callbacks route directly. Flip when one payer's volume becomes a hot key (a merchant).
• Retry in one layer vs several. One layer, the one that owns the idempotency key; several layers give 27x amplification against an already struggling bank.

• Showing "failed" after a timeout and letting the user retry as a new payment. This is how double debits happen.
• Retrying at the app, the backend and the gateway at once, which turns 1,200 a second into 32,400.
• No end-to-end reference, so there is nothing to query or reconcile on.
• Applying callbacks without dedupe or a forward-only check, so a late duplicate flips a final state.
• Designing a ledger of the money itself when the banks own the money. Keep our own record and reconcile.
• Storing the PIN, or logging it, or never saying where it is decrypted.