Design a GitHub-like code hosting service (load balancing and observability)

Questions: how many users, repositories and daily git operations, and how big is a typical repository? Which protocols: HTTPS, SSH, or both? What is the ratio of fetches to pushes? Is there a web UI and API (pull requests, issues) in scope, or only git hosting? What are the availability and latency targets? Are there regions, or one? How quickly must a push be visible to the next fetch?

Assumed scope: git over HTTPS and SSH, repository storage, web and API traffic, load balancing, health, monitoring and safe releases. Out of scope: code search, CI, and the internals of pull request review.

Planning figures for the exercise, not facts about any real service.

• Fetch and clone: 20 million a day / 86,400 = about 231 a second; at a 5x peak, about 1,160 a second.
• Push: 2 million a day / 86,400 = about 23 a second; at 5x, about 116 a second.
• Web and API: 100 million a day / 86,400 = about 1,160 a second; at 5x, about 5,800 a second.
• Concurrent git streams: by Little's law, 1,160 starts a second x an assumed 20 seconds each = about 23,000 open streams at peak.
• Bandwidth: assume an average of 5 MB per fetch (most are small incremental fetches). 20 million x 5 MB = 100 TB a day. 100 TB / 86,400 = about 1.16 GB a second, or 9.3 Gbit a second on average; 5x peak is about 46 Gbit a second.
• Repository data: 10 million repositories x 50 MB = 500 TB; three copies is 1.5 PB. On 4 TB machines filled to 70% (2.8 TB each), 1,500 / 2.8 = about 540 storage machines.

What the numbers force: the request rate is small (about 7,000 a second at peak across everything), but bandwidth and storage are not. The load balancers are limited by bytes (46 Gbit a second needs at least five 10-Gbit machines, so run nine, three in each of three AZs, and losing one AZ still leaves six), and repository data must be spread over hundreds of storage machines. The metadata (users, permissions, issues) stays on a relational primary with replicas, because 5,800 reads a second is under what a cached primary handles.

1. Edge. DNS to a layer-4 balancer fleet for SSH and for git over HTTPS when you want to pass TLS through and keep per-connection cost low, and a layer-7 fleet for the web UI and API where routing by path, authentication and rate limits are needed. Both fleets span three AZs. Layer 7 adds latency and CPU per request but understands requests; layer 4 is nearly free but sees only connections.
2. Routing to repositories. A repository is not stateless: its bytes are on specific machines. A placement table in the metadata store maps repo_id to a replica set of three storage nodes and names a primary. It is a lookup, not a hash, so one repository can move without renumbering the rest. Gateways cache the table and refresh on a change event.
3. Fetch and clone. Route to any healthy replica, choosing by least outstanding requests, not round robin, because one clone of a huge repository weighs a thousand small fetches. A gateway process on the storage node runs the git protocol against the local repository. Cache generated packfiles and archive downloads for popular repositories, and put a CDN in front of archive and release downloads.
4. Push. Route to the repository's primary. It writes the objects, updates the refs, and must replicate to at least one other replica before it acknowledges (two of three). Ref updates are compare-and-swap on the old value, so two simultaneous pushes cannot both win. A push to a repository on a failed primary waits for a replica to be promoted, and promotion picks the survivor with the newest refs: an acknowledged push exists on two of the three copies, so at least one survivor has it. Fetches continue from the others.
5. Connection handling. Clones last minutes, so deploys must drain: stop sending new requests, let existing streams finish up to a deadline, then close. Use a shallow health check (process alive) for the balancer and a deep one (can read a canary repository, replication lag under threshold) for traffic-shifting decisions. A deep check that every node runs against a shared dependency can take everything out at once, so shed rather than eject when too many fail.
6. Hot repositories. A popular repository concentrates reads on three nodes. Add read-only replicas for that repository, raise the packfile cache hit rate, and rate-limit anonymous clones per IP.
7. Observability. Define SLIs per user journey: fetch success rate, push success rate, time to first byte, push latency p99, web p99. Alert on multi-window burn rate against the SLO, not on raw CPU. Metrics are labelled by service, route class and status, never by repository or user: 10 million repositories would be 10 million series. Per-repository detail lives in structured logs and sampled traces, with a request id that flows from the balancer through the gateway to the storage node.
8. Signals per tier. Balancer: connections, bytes, 5xx, backend ejections. Gateway: request rate, errors, duration, open streams. Storage node: disk utilization and queue depth, CPU in pack generation, free space, replication lag. Add a synthetic probe from each region that clones, pushes and fetches a canary repository once a minute.
9. Release safety. Roll balancer configuration and storage software in waves: one AZ, then one more, with a bake time and automatic stop on error-rate or latency regressions versus the old version. Treat config pushes as releases.

Hard problem one: stateful routing with uneven requests. Placement lookup plus least-outstanding balancing across replicas.

Hard problem two: knowing about problems early without drowning in series. Journey SLIs, burn-rate alerts, canary probes, and high-cardinality detail in logs and traces instead of metrics.

• Layer 4 vs layer 7 for git traffic. Layer 4 is cheap and passes through any protocol but cannot route by repository or read a token. Layer 7 can route and rate-limit per repository and user but costs more per request. Pick layer 4 for SSH and a layer-7 tier for web and API. Flip git-over-HTTPS to layer 7 when you need per-repository routing or per-user limits at the edge.
• Round robin vs least outstanding requests. Round robin is simple and fine for uniform requests. Git requests vary by orders of magnitude, so a few heavy clones can pile up on one node. Pick least outstanding. Flip to consistent hashing by repository id when cache locality on the storage node matters more than even spread.
• Synchronous vs asynchronous replication on push. Waiting for a second copy adds a round trip (0.5 ms in the same AZ, 1 to 2 ms across AZs) and avoids losing an acknowledged push if the primary dies. Asynchronous is faster but can lose recent pushes. Pick two-of-three synchronous. Flip to asynchronous only for mirrors or forks where the source can be recopied.
• Deep vs shallow health checks. Deep checks catch more but can flap or cascade; shallow checks miss a node that is up but useless. Pick shallow for ejection and deep for alerts and gradual traffic shifting. Flip to deeper ejection checks when failures are mostly node-local.

• Treating the service as stateless web servers behind one balancer and missing that a repository lives on specific machines.
• Round-robin balancing for requests whose cost varies by a factor of a thousand.
• Sizing the balancers by request rate (about 7,000 a second) when bandwidth is the constraint.
• Putting repository or user id into metric labels and exploding the series count.
• Alerting on CPU instead of on user-facing success and latency.
• Restarting gateways with no connection draining, killing every long clone.
• Acknowledging a push before a second copy exists.
• A health check that depends on a shared service, so one dependency outage ejects every node.