A1 · Part 3 · Advanced
The CAP theorem in depth
You never "pick two" — you pick one letter during a partition, and PACELC tells you what you picked for the other 99.9% of the time.
09On this page
- The version everyone repeats is wrong, and the wrongness matters
- The three letters, defined precisely (this is where marks are lost)
- What CP actually looks like from the client's seat
- What AP actually looks like from the client's seat
- PACELC — the framing that actually predicts database behaviour
- Where real systems sit — the honest, tunable version
- The quorum dial: R + W N
- Using CAP in the room without sounding like a flashcard
- Recognizing it in an unseen problem
The version everyone repeats is wrong, and the wrongness matters
The folklore statement is "consistency, availability, partition tolerance — pick two." That framing fails the moment you take it seriously, because P is not a choice you get to make. A partition is a property of the network: a cut fibre, a failed top-of-rack switch, a bad BGP announcement, an AZ losing connectivity, or — most commonly — a node that is merely slow (a 40-second stop-the-world GC pause, a VM live-migration stall) and is therefore indistinguishable from a dead one. You cannot buy a network that never partitions, so "CA" is not a system design. It is a system with no story for partitions, which under partition quietly gives up both letters.
The precise statement is narrower and far more useful: when a partition occurs, and only then, you must choose between consistency and availability. When there is no partition — which is essentially all of the time — CAP says nothing whatsoever about your system. That last clause is the one that separates candidates: CAP is a theorem about the rare case, and the rare case is not where your latency budget lives.
The three letters, defined precisely (this is where marks are lost)
| Letter | What it formally means | What candidates think it means |
|---|---|---|
| C | Linearizability. The system behaves as if there is exactly one copy of the data: every read returns the value of the most recently completed write, and once a read sees a value, no later read sees an older one. | The C in ACID (integrity constraints). Unrelated. |
| A | Every request to a non-failing node returns a non-error response. No bound on how long it takes, and no promise the answer is fresh. | "Four nines of uptime." Also unrelated — a CP system can have a better SLA than an AP one. |
| P | The system keeps operating when arbitrary messages between nodes are dropped. | An optional feature. It is not optional. |
Two consequences fall straight out of those definitions. First, CAP-A has no latency bound, so a system that answers in 30 seconds is "available" — which is why CAP alone is useless for reasoning about production SLOs. Second, CAP-C is linearizability, the strongest single-object guarantee; weaker useful models (read-your-writes, monotonic reads, causal consistency) sit below it and are perfectly compatible with staying available during a partition. "AP" does not mean "no guarantees."
CAP's own authors and critics say the same. Brewer's 2012 retrospective calls "2 of 3" misleading, because partitions are rare and the choice can differ by operation and data. Kleppmann's critique points out that the formal definitions are ambiguous, that "available" in the proof does not match what operators mean, and proposes reasoning about how operation latency responds to network delay instead. Treat CAP as a vocabulary for partition-time behaviour, not a classification scheme for products.
What CP actually looks like from the client's seat
Take a five-node leader-based cluster — etcd, ZooKeeper, a Spanner Paxos group, a MongoDB replica set with majority write concern. The network splits it 3 | 2.
- Majority side (3 nodes): keeps or elects a leader, keeps committing writes, fully functional. If the old leader was on this side, there is no interruption at all.
- Minority side (2 nodes): cannot reach a quorum, so it cannot commit anything. Writes fail immediately with something like "no leader" or a context deadline. Reads either fail too, or are served locally and are knowingly stale — that is a per-system decision, and a good one to ask about.
- A client pinned to the minority sees hard errors for the entire duration of the partition, plus an election timeout on top (Raft's classic 150–300 ms, etcd's default 1000 ms, ZooKeeper tick-based failover typically a few seconds).
The client-visible contract is: never a wrong answer, sometimes no answer. That is the right trade whenever a wrong answer is more expensive than an error page — a ledger balance, decrementing the last unit of inventory, a cluster-membership record, a lock, a feature-flag kill switch.
Dry run: a 5-node CP cluster split 3 | 2, minute by minute
| Time | Event | Majority side (3 nodes) | Minority side (2 nodes) |
|---|---|---|---|
| t=0 | Network partition occurs; the old leader happens to be on the majority side | Continues as leader, no interruption | Isolated, cannot reach the leader |
| t=0 to t=1000 ms | A client on the minority side sends a write | — | No quorum reachable — write fails with a timeout, not a wrong answer |
| t≈1000 ms | Minority nodes hit their election timeout (etcd default 1000 ms) | — | With PreVote (etcd's --pre-vote flag defaults to true in 3.6) a node first asks "would you vote for me?", gets at most 2 of 5 yes answers, and does not bump its term. In plain Raft it becomes a candidate, increments its term, and keeps retrying with ever higher terms |
| t=1000 ms onward | Majority-side clients keep reading and writing | Fully available, fully consistent — commits proceed normally | Still returning errors or refusing to serve |
| partition heals | Minority nodes contact the majority leader | With PreVote: unchanged, the leader's heartbeats pull the minority back as followers at the current term. In plain Raft, and with CheckQuorum alone, a rejoining node with an inflated term forces the healthy leader to step down and triggers a needless election | Steps down to follower, catches up on the log |
Nothing in this timeline ever returns a stale or wrong value — that is the entire CP guarantee. The cost is visible in row two: for roughly one election timeout, any client pinned to the minority side gets hard errors and nothing else, which is exactly the "never a wrong answer, sometimes no answer" contract stated above.
What AP actually looks like from the client's seat
Now the same partition in Cassandra at consistency level ONE, or Riak. Both sides accept writes. Every client gets a 200. Nothing appears wrong — and that is precisely the danger, because the damage is deferred to reconciliation time. When the partition heals you have two divergent histories, and something has to merge them:
DynamoDB is a common wrong example here. Inside a Region, each partition has a single Multi-Paxos leader and writes need a majority of its three replicas, so it does not let both sides of a split accept writes; only the eventually consistent reads are served by any replica. Divergent writes appear in global tables in their default multi-Region eventual consistency mode, where last-writer-wins resolves conflicts. Since 30 June 2025 global tables can instead be created with multi-Region strong consistency, which replicates synchronously to at least one other Region and trades write latency for a zero recovery point.
| Reconciliation strategy | What it costs | Reach for it when… |
|---|---|---|
| Last-write-wins by timestamp | Silently discards one side's writes. With wall-clock timestamps, clock skew of a few hundred ms decides which user's data survives. | The value is a cache-like overwrite where losing an update is genuinely acceptable — a "last seen at" field, a presence flag. |
| Siblings / vector clocks returned to the app | Every read path must now handle "here are 3 conflicting values." Real complexity, pushed into product code. | The app has domain knowledge that makes merging obvious (union the shopping cart). |
| CRDTs (G-Counter, OR-Set, LWW-Register) | Restricted data types, metadata growth, and you must model the domain as a mergeable lattice. | Counters, sets, collaborative text, presence — anywhere concurrent edits are normal (Redis CRDT, Automerge, Yjs). |
| Read repair + anti-entropy (Merkle trees) | Background convergence only; does not decide semantics, just propagates whatever the winner already is. | Always — it is a complement to one of the above, never a substitute. |
⚠ "We'll go AP" is only half an answer
The interviewer's next sentence is "so how do you merge?" If you cannot
name last-write-wins, siblings, or a CRDT and say which one you'd use for
this data, you have taken AP's availability without paying AP's
price. "Eventually consistent" describes an outcome, not a mechanism.
The canonical example is the Dynamo shopping cart: under LWW, a partition
loses items a customer added. The Dynamo paper instead merges divergent
carts by union, so the worst case is a deleted item reappearing — which
Amazon judged strictly better than a lost sale. (A real OR-Set CRDT goes
further: it tags each add, so a remove deletes only the adds it has seen
and deletions are not resurrected.)
PACELC — the framing that actually predicts database behaviour
Abadi's 2010 extension is the one to bring up unprompted, because it covers the case CAP ignores: if there is a Partition, choose A or C; Else, choose Latency or Consistency. The "else" half applies over 99.9% of your system's life, and it is not a legal technicality — it is forced by physics. Keeping replicas linearizable requires a round trip to a quorum before you can answer, and round trips cost:
| Hop | Typical RTT | What that means for a quorum write |
|---|---|---|
| Same AZ | ~0.5 ms | Consistency is nearly free; take it. |
| Cross-AZ, same region | 1–2 ms | Still cheap. This is why a 3-AZ quorum inside one region is the default shape for most CP systems. |
| us-east ↔ us-west | 60–70 ms | The leader needs an ack from its nearest follower (a majority of 3 is the leader plus one), and that is a full round trip, not the one-way delay. If the nearest follower is across the continent, the write p50 starts at 60–70 ms, before any disk or queueing time. |
| us-east ↔ eu-west | 70–90 ms | Global linearizable writes cost a transatlantic round trip, permanently. No amount of engineering removes this; you can only place the quorum so the nearest follower is close (three regions, leader near the writers). |
Spanner is the clearest illustration of choosing consistency in the "else" branch and simply paying: it is PC/EC, it commits through Paxos across regions, and it adds a deliberate commit wait of roughly twice the TrueTime clock uncertainty (a few milliseconds, per the Spanner paper) so that commit timestamps are globally meaningful. Writes land in the tens of milliseconds. In exchange you get external consistency across a planet, and reads at a past timestamp that a nearby replica can serve without talking to the leader, provided that replica has caught up to the read timestamp (otherwise the read waits for it, so it can still block).
Where real systems sit — the honest, tunable version
| System | PACELC | Why, and where the knob is |
|---|---|---|
| Postgres, single primary + sync standby | PC/EC | With synchronous_commit = on and a named standby, a commit waits for the standby. Lose the standby and writes block — that is CP behaving as advertised. Switch to local/async and you become EL with replica reads stale by the replication lag (sub-ms idle, seconds-to-minutes under a heavy write burst or a long-running query on the replica). |
| DynamoDB | Writes: consensus (PC) Reads: EL by default, EC per read | Each partition has a Multi-Paxos leader; writes and strongly consistent reads go through it, so a partition that loses its leader majority cannot accept writes. Reads are eventually consistent by default and may be served by any of the three AZ replicas. Pass ConsistentRead=true and you route to the leader: 2× the read cost, a few extra ms, and unavailable if that partition has no leader. The letter is chosen per API call. Global tables add a second axis: the default multi-Region eventual consistency mode resolves conflicts by last-writer-wins, while multi-Region strong consistency (GA 30 June 2025) is synchronous. |
| Cassandra / ScyllaDB | PA/EL (default) PC/EC (QUORUM) | Consistency level is per query. ONE is fast and stale-tolerant; LOCAL_QUORUM keeps you inside one DC. QUORUM read + QUORUM write gives R + W > N, so a read sees the latest acknowledged write in the normal case, but it is not linearizable: concurrent writes are resolved by timestamp (last-write-wins) and a partially failed write can surface later. Linearizable compare-and-set needs lightweight transactions (Paxos-based, SERIAL consistency). Cassandra does not use sloppy quorums: a hint satisfies a write only at CL=ANY, so at QUORUM the acks must come from real replicas. |
| Spanner / CockroachDB | PC/EC | Paxos/Raft per range, majority commit. Consistency is not negotiable; latency is the bill. Follower reads and bounded-staleness reads are the escape hatch when you want EL for a specific query. |
| ZooKeeper | PC/EC (writes) | Zab totally orders writes through a leader with a majority. But reads are served locally by whichever server the client is connected to and may be stale — call sync() first when you must see the latest committed write. Within one session reads never go backwards, which is a session guarantee, not linearizability. Saying this unprompted is a strong signal. |
| etcd | PC/EC | Raft; linearizable reads by default via read-index, atomic multi-key transactions, and documented strict serializability for KV operations. Minority members return errors. Opt-in serializable reads trade that for latency and may be stale. This is the coordination substrate for Kubernetes precisely because it refuses to guess. |
| MongoDB | PC/EC or PA/EL | w:majority + readConcern:majority is CP-ish; w:1 plus reads from secondaries is AP-ish and can lose acknowledged writes on failover (rollback files). |
| Redis (Sentinel or Cluster) | Neither, honestly | Replication is always asynchronous, so a failover can lose writes the primary already acknowledged. It is AP-shaped without AP's conflict resolution — the losing writes just vanish. WAIT reduces the window but is not a quorum commit. Treat Redis as a cache or a lossy store, and never as your source of truth for money. |
The one line to remember
The choice is made per operation, not per database. Almost every
modern store exposes the dial as a per-request consistency level, write
concern, or read flag — so the interesting design question is never "is
this system CP or AP", it is "which of my operations can tolerate a stale
read, and which cannot".
The quorum dial: R + W > N
In a Dynamo-style system with N replicas, you require W acknowledgements to accept a write and R responses to serve a read. If R + W > N, the read set and the write set must overlap in at least one replica, so every read touches at least one node that saw the latest write.
| N=3 config | Behaviour | Reach for this when… |
|---|---|---|
| W=1, R=1 | Fastest possible both ways, no overlap, reads can be arbitrarily stale. One replica loss loses data. | Metrics, logs, view counters, anything where losing a write is a rounding error. |
| W=2, R=2 | Overlap guaranteed. Tolerates one node down for both reads and writes. | The default. Most OLTP-ish workloads on a Dynamo-style store. |
| W=3, R=1 | Reads are single-hop fast; any node down makes writes unavailable. | Read-dominated config data written rarely. |
| W=1, R=3 | Writes are single-hop fast; any node down makes reads unavailable. | Rarely correct — usually a sign the workload wants W=2, R=2. |
The overlap argument assumes the W acks come from the key's real replicas. Riak and the original Dynamo relax that with sloppy quorums and hinted handoff, which keeps writes available during a partition but can leave a read set and a write set disjoint. Cassandra only does so at CL=ANY. And even with strict quorums, R + W > N is not linearizability: concurrent writes need a conflict rule, and a failed write may still surface later.
Say it like this → "I'll run N=3 across three
AZs with quorum reads and writes for the orders table — cross-AZ RTT is
about a millisecond so consistency is nearly free there. For the product
catalogue I'll drop to R=1 and accept a few seconds of staleness, because
a stale price on a listing page is recoverable and an unavailable
listing page is not."
Using CAP in the room without sounding like a flashcard
The senior move is to refuse the question as posed. Nobody designs "a CP system"; you design a system in which different data has different requirements, and you say so explicitly:
- Payments ledger, account balances, idempotency keys: CP. Refuse the write rather than double-charge.
- Session store, presence, notification counts, feed ranking: AP. A stale unread count costs nothing.
- Inventory: split it. The last 50 units of a SKU are sold through a CP path with a real reservation; the "1,200 in stock" badge on the listing page is AP and can be minutes old. Ticketmaster-style seat selection is the same trick — browse is AP, hold is CP.
- Cluster metadata, leader election, feature-flag kill switches: CP, and delegated to etcd or ZooKeeper rather than built.
⚠ Three sentences that cost you the level
"It's a CA system" (there is no such thing across a network).
"CAP says pick two" (it says pick one, and only during a partition).
"We'll use Cassandra because it's AP" (Cassandra is whatever your
consistency level says it is, per query). Any of these tells the
interviewer you learned CAP from a blog post rather than from an outage.
Recognizing it in an unseen problem
- The prompt spans more than one datacentre, region, or AZ — the moment replicas can be separated by a network, CAP is live and the interviewer will probe it.
- Words like "globally distributed", "multi-region active-active", "must never lose a write", "must always accept a write" are direct invitations to name your partition-time choice.
- A naive design says "we'll replicate to the other region" and stops. The follow-up that exposes it is always: what does the client see while the link between them is down? Have that answer ready before it's asked.
- Distinguish from the plain replication chapter: replication is about how copies get updated; CAP is about what you do when they can't. Distinguish from PACELC's else-branch: if the network is healthy and you are still arguing about staleness, that is a latency-vs-consistency question, not a CAP question.
- If you choose AP, immediately state the merge function. If you choose CP, immediately state the blast radius: which clients get errors, for how long, and what the retry/queue story is so the user doesn't just see a 500.
- The strongest closing move is to make the choice per-operation and justify each one with the business cost of being wrong versus the business cost of being down.
Before you move on
- Can you state precisely what C, A and P mean in CAP — not the versions most candidates get wrong?
- Can you explain why PACELC matters even when there's no partition, and where it costs Spanner milliseconds?
- Given a data type, can you say whether it should be CP or AP, and if AP, name the reconciliation strategy?
- Can you explain what R + W > N guarantees, and one thing it does NOT guarantee?
- Can you classify one real system's PACELC behaviour and name where its tunable knob lives (e.g. DynamoDB's ConsistentRead)?
Sources and further reading
- Brewer, CAP twelve years later — why "2 of 3" misleads and how partition mode and recovery work
- Kleppmann, A critique of the CAP theorem — ambiguities in the formal definitions and a latency-based alternative
- Abadi, PACELC — the latency-versus-consistency trade-off when there is no partition
- Dynamo paper (SOSP 2007) — sloppy quorums, hinted handoff, and the shopping cart merge
- Brooker, DynamoDB paper review — a Multi-Paxos leader per partition and who serves which reads
- AWS: global tables with multi-Region strong consistency — general availability on 30 June 2025
- Cassandra architecture: Dynamo — hints satisfy only consistency level ANY
- etcd API guarantees — strict serializability, transactions, and stale serializable reads
- etcd configuration flags — pre-vote default, 100 ms heartbeat and 1000 ms election timeout
- ZooKeeper programmer's guide — local reads can be stale; use sync when it matters
- Spanner paper (OSDI 2012) — TrueTime, commit wait and external consistency
Claims in this chapter were checked against these sources in October 2026. Figures marked "planning figure" are rules of thumb, not guarantees.
5 questions drawn from 8, scored out of 5. This is practice: it runs in your browser, saves nothing and does not change what you have marked read.
Finished reading?
0 of 36 chapters read in this series