Design a toll fare system with Kafka streaming

Questions first. Is the road a closed system (a read at entry and a read at exit, fare by distance) or are there flat charges at single points, or both? How is a vehicle identified: a radio tag, a camera plate read, or both? Do customers prepay a balance or get invoiced? How late can a read arrive, for example after a gantry loses its uplink? Who handles disputes, and what evidence must we keep? Roughly how many gantries and reads a day?

Scope I will assume: closed system with entry and exit gantries plus a few flat-rate points; the tag is primary and a camera read is a lower-confidence fallback sent to review; prepaid wallets and postpaid invoices share one ledger. Out of scope: gateway integration with banks, roadside hardware, enforcement against vehicles with no tag.

Planning figures for the exercise, not facts about any operator.

• Reads: 10 million tag reads a day, 10 × 12 = about 120 a second on average, and a 5x rush-hour peak is about 600 a second.
• Bytes: at 200 B a read, peak is 600 × 200 B = 120 KB a second; a day is 2 GB, a year about 730 GB. Small.
• Trips: two reads make one trip, so 5 million fares a day, 5 × 12 = about 60 a second, 300 a second at peak.
• Ledger writes: 300 a second is far below the 5,000 one primary takes.
• Open trips (state): entries per second × trip time = 300 × 1,800 s (30 minutes) = 540,000 trips in flight at peak. At 200 B each that is 108 MB of state, which fits in memory.
• Outage replay: a 2-hour uplink loss leaves 10,000,000 × 2 ÷ 24 = about 833,000 reads; draining 5,000 a second takes about 167 s.

The unusual conclusion: throughput does not justify Kafka. Ordering per vehicle, replay, buffering through outages and decoupling the ledger do. A topic of 12 partitions (about 50 reads a second each at peak) is plenty.

1. Edge. Each gantry stores every read on local disk before sending (store and forward), stamping a read id made of gantry_id, an epoch and a local sequence number, and an event time from a clock synced by NTP. It uploads in batches and deletes only after an acknowledgement. This is where duplicates and lateness come from, so they are designed in.
2. Ingest. A thin API validates and produces to the topic tag-reads, keyed by tag_id (plate for camera reads, flagged low confidence). The key keeps all of a vehicle's reads in one partition, in order, in front of one consumer. Keying by gantry would instead put a busy gantry on one hot partition and scatter a vehicle's trip across consumers. Retention is 7 days, and a copy goes to object storage for reprocessing and audit.
3. Trip builder (stateful stream job, for example Kafka Streams or Flink). State per vehicle: the open entry (gantry, event time, read id). An exit read looks up the entry and emits a trip. An exit with no entry waits for a bounded time (say 6 hours) in case the entry arrives late; after that it becomes a "missing entry" trip billed by a documented rule and flagged. An entry with no exit for more than 24 hours becomes a "missing exit" case. The state store has a changelog topic, so a restart rebuilds it.
4. Rating. A pure function from (entry gantry, exit gantry, vehicle class, event times, tariff version) to a fare in minor units. Tariffs are versioned with effective times, and the trip row stores tariff_version, so a price change at midnight never reprices old trips and a replay gives the same answer.
5. Ledger. Fares go to topic fares, then a writer inserts into ledger_entries(account_id, trip_id, type, revision, amount_minor, created_at) with a unique key on (trip_id, type, revision). The trip id is a deterministic hash of tag id and entry read id (the exit read id for a missing-entry trip). Unique plus ON CONFLICT DO NOTHING makes the write idempotent, which is how a crash and replay cannot double-charge. The balance is derived from entries, and corrections are new adjustment rows, never edits.
6. Reconciliation. A nightly batch recomputes fares from the archived raw reads and compares them with the ledger. Differences go to a review queue. This is the safety net behind the stream job.

Duplicates. Three layers, cheapest first: the gantry's own sequence numbers; the stream job's dedupe store keyed by read id with a 7-day TTL; and the unique trip id in the ledger. Any layer may fail and the next still holds. Exactly-once inside Kafka (transactional produce and consume) helps the topic-to-topic hops, but the final database write is an external side effect, so it relies on the unique key.

Late and out-of-order reads. Use event time, not arrival time, for pairing and for choosing the tariff. Within one vehicle, a late entry is matched to a waiting exit. If a trip was already closed with the default rule and the real read arrives afterwards, emit a corrected fare and an adjustment entry, and tell the customer. There is no fixed window to close here, so the bounded waits above play the role of a watermark: they trade how long a fare is delayed against how often it is corrected.

• Stream processing vs a nightly batch. Streaming lets a customer see a fare within seconds and lets the wallet block on a low balance. A nightly batch is simpler and cheaper to operate. Pick streaming, with the batch kept as the reconciler; flip to batch only if billing is monthly and nobody needs the balance in real time.
• Kafka transactions vs an idempotent sink. Transactions give atomic read-process-write inside Kafka but stop at the database. A unique trip id works for every sink and every replay. Pick the idempotent sink; add transactions as well only if duplicates on the intermediate topic are expensive.
• Local state vs a database lookup per read. Local state with a changelog is fast (no network hop per read) and sized at about 108 MB. A shared database lookup is simpler to inspect. Pick local state; flip to the database if state grows past what a node can hold or you need ad hoc queries on open trips.
• Wait longer vs bill sooner. A long wait for a missing read gives fewer corrections but delays charges. A short one bills fast and corrects often. Pick a bounded wait of hours; flip shorter if disputes are rare and cash flow matters.

• Justifying Kafka by volume. 600 reads a second is tiny. Say the real reasons: ordering, replay, buffering and decoupling.
• Partitioning by gantry so a vehicle's two reads land on different consumers.
• Using arrival time for pairing and for the tariff.
• Claiming exactly-once end to end. The database write is an external effect; say what makes it idempotent.
• Editing ledger rows instead of appending adjustments.
• Forgetting missing reads. Real roads lose reads, so unmatched entries and exits need an explicit rule.
• No reconciliation job, so a stream bug is found by customers.
• Ignoring clock skew between gantries, so event times from different gantries cannot be compared.
• Hiding the tariff version, so a replay after a price change gives different fares.