Scaling: vertical vs horizontal

There are only two knobs, and they cost different things

When a system runs out of capacity you can make the machine bigger (vertical, scale up) or add machines (horizontal, scale out). Everything else — caching, replicas, sharding, queues — is either a way to need less capacity or a specific tactic for making scale-out work on a tier that resists it.

The reason interviewers open here is that the choice reveals judgement. Vertical scaling costs money and has a ceiling. Horizontal scaling costs architecture: it forces you to give up in-process state, easy transactions, and the ability to reason about your system as one program. Candidates who reach for a distributed cluster before they have exhausted a single box are showing an interviewer that they will over-build in production too, and that is a genuine downgrade signal at senior level.

How big is one box, actually

Most engineers' intuition about a "single server" was formed on a laptop or a 4-vCPU cloud instance. The real top end is startling, and quoting it is one of the cheapest ways to sound like you have operated systems rather than only read about them.

What that translates to in throughput terms is more useful than the specs:

The Postgres rows are conservative planning numbers, not limits. The course rule: start planning a change when sustained reads pass ~20k/s (add a cache or replicas) or sustained writes pass ~5k/s (partition or queue), and treat ~20k+ sustained writes as "you are sharding or queueing now". How far past that a single primary can go depends on hardware and batching: OpenAI has described running ChatGPT on one Azure PostgreSQL primary with about 50 read replicas at millions of aggregate queries per second. Measure, do not assume.

Where vertical scaling actually hits the wall

The naive story is "vertical scaling gets exponentially expensive." That is only partly true, and saying it as stated invites a correction: within an instance family, cloud pricing is close to linear — twice the vCPUs is roughly twice the price. The real wall is made of five other things, and naming them precisely is the senior version of this answer.

What horizontal scaling takes away

Adding a second server silently invalidates a set of assumptions that were true and invisible when there was one. This is the part candidates skip, and it is where the interesting follow-ups live.

// Perfectly correct on one server. Quietly broken on two.
const hits = new Map();

function allowRequest(userId) {
  const n = (hits.get(userId) || 0) + 1;
  hits.set(userId, n);
  return n <= 100;            // "100 requests per user"
}
// With 10 app servers behind a round-robin LB, each user gets ~1000.
// A deploy resets every counter. Autoscaling changes the limit.
// The fix is not a bigger Map — it is moving the counter out of the process.

Stateless and stateful tiers scale differently

The reason the app tier is easy and the data tier is hard comes down to one question: does adding a node add capacity, or does it add a copy that must be kept in agreement with the others?

Linear capacity turns sizing into a division: peak load over the load one instance may safely carry, which is its tested throughput times a utilisation ceiling you choose. At 12,000 requests a second peak, an instance tested at 800 and held to 60% carries 480, so the tier needs 12,000 ÷ 480 = 25 instances.

This is why "make the app tier stateless" is not a style preference. It is the move that concentrates all the hard problems into one tier, where you can attack them with dedicated tools, instead of spreading them everywhere.

Read replicas: the first real scaling move for most systems

Nearly every consumer-facing system is read-heavy, often by 10:1 or 100:1. That asymmetry means the highest-leverage change is almost always to serve reads from somewhere other than the primary: a cache first, and replicas right behind it. A replica is a full copy of the database that applies the primary's write stream and serves read-only queries.

When not to scale out

Premature distribution is one of the most reliable senior-level red flags, precisely because it looks like sophistication. The cost is not the servers — it is that a distributed system fails in partial, non-reproducible ways, and you have taken on that tax before you had the problem it solves. Work this list before adding nodes, out loud:

• Measure. Which resource is saturated — CPU, memory, disk IOPS, connections, or a lock? "The database is slow" is not a diagnosis, and an interviewer will notice if you skip this.
• The missing index. A single index has turned a dying database into an idle one more times than every scaling technique combined.
• N+1 queries and chatty calls. 200 sequential queries per page request is an application bug, not a capacity problem.
• Connection pool configuration. Databases die from connection exhaustion far more often than from CPU.
• Caching the hot 1%. Read distributions are power laws; a small cache usually absorbs most of the traffic.
• Move work off the request path. Emails, thumbnails, analytics, and webhooks belong in a queue, not in the user's 200 ms.
• Then buy the bigger box. It is a one-line change and it buys months. Take the months.

Dry run: applying "when not to scale out" to one incident before reaching for a shard

Scenario: p99 latency on checkout climbs from 120 ms to 800 ms during a flash sale, and the team's first instinct is to shard the primary database.

Five of the seven steps never touch the database's shape at all, and the first step, measuring, is what pointed at them — the incident that looked like "we need to shard" turned out to be an index, a query pattern, a pool setting, a cache, and a queue. Sharding stays the last resort precisely because the chapter orders the checklist that way.

Recognizing it in an unseen problem

• Do the scale math first: below roughly 5,000 sustained writes/sec or a few terabytes, one primary plus replicas is a defensible design and reaching past it needs justification
• Any prompt with "high availability" is a horizontal question even at tiny scale — you add the second node for failure tolerance, and it happens to also add capacity
• Read-heavy with tolerable staleness (feeds, catalogs, profiles) → cache then replicas; write-heavy or strongly consistent (ledgers, inventory, counters) → replicas do not help and you should say so
• If the design keeps anything per-user in process memory — sessions, WebSocket maps, rate limit counters, upload buffers — that component is stateful and cannot simply be autoscaled; decide where the state goes before you add the second instance
• Distinguish scaling out from sharding: adding stateless app servers is nearly free, and partitioning a database is a one-way door involving hot keys, cross-shard queries, and resharding pain
• The trap is symmetric — under-designing for a stated billion-user scale reads as naive, and over-designing for a stated thousand-user scale reads as undisciplined; the number they gave you is the tiebreaker