Scale and edge cases · Git

Fresher Mid Senior

Git was built for the Linux kernel: many small text files and a long history. It stays fast well past that, but three things push it off the happy path. Large binaries, millions of files, and a team spread across Windows, macOS and Linux. This chapter is the toolkit for each.

In one minute

If a clone takes forever, you probably do not need all of it: git clone --filter=blob:none downloads history but fetches old file contents only when asked. If git status is slow, run git maintenance start once in that repository. Big images or videos belong in Git LFS, set up before the first one is committed. And on a mixed Windows and Mac team, commit a .gitattributes with * text=auto so line endings stop showing up as changes.

Problem	Reach for
Large binaries: design files, videos, models	Git LFS: pointers in Git, contents on a separate store
CI only needs the latest commit	Shallow clone, `--depth 1`, or `--revision=<sha> --depth 1` for one exact commit
Clone is slow because history is big	Partial clone, `--filter=blob:none`
Huge monorepo, you work in one folder	`git sparse-checkout set apps/web`
`git status` and `git log` are slow	`git maintenance start`, commit-graph, fsmonitor
All of the above at once	`scalar clone`
A formatting commit clutters `blame`	`.git-blame-ignore-revs`

Git LFS

Git stores every version of every file forever, and binary files barely compress or delta against each other. Commit a 200 MB video ten times and every clone downloads about 2 GB, even people who never open it. Git LFS (Large File Storage) keeps a tiny text pointer in the repository and puts the real bytes on a separate server. You only download the versions you check out.

LFS works through Git's filter mechanism. The clean filter runs when a file enters the index, the smudge filter runs when it is written to your working tree, and a pre-push hook uploads new objects. Git itself only ever sees the pointer.

git lfs install
git lfs track "*.psd" "*.mp4"
git add .gitattributes
git add design.psd
git commit -m "Add hero artwork"
git lfs ls-files

git lfs install is once per machine and registers the filters in your global config. track writes a line like *.psd filter=lfs diff=lfs merge=lfs -text into .gitattributes. Commit that file first, or the next person's clone won't know which files are LFS.

What goes wrong

Pointers instead of files. Someone cloned without LFS installed and sees a three-line text file where a PNG should be. Fix: git lfs install, then git lfs pull.
Binaries already in history. track only affects new commits. To move old ones, rewrite history with git lfs migrate import --include="*.psd" --everything, then force-push and have everyone re-clone.
Quotas. Hosts meter LFS storage and bandwidth separately from Git. A CI job that downloads every LFS file on every run can burn through a monthly quota. Cache the LFS directory, or skip what you don't need with GIT_LFS_SKIP_SMUDGE=1 and git lfs pull --include="path/*".
Binary merges. Git can't merge two edits to the same PSD. git lfs lock design.psd gives one person the file at a time, if the host supports locking.

Cloning less: shallow, partial and sparse

These three are easy to confuse because they all make a clone smaller. They cut along different axes. Shallow cuts history. Partial clone cuts which objects you download up front. Sparse-checkout cuts which files appear in your working tree.

Shallow drops old commits. Blobless keeps every commit but leaves file contents on the server until something asks for them. Sparse-checkout decides which folders appear on disk, and with a blobless clone that also decides which blobs are ever downloaded.

Shallow clone

git clone --depth 1 https://github.com/org/app.git
git fetch --deepen 50
git fetch --unshallow
git clone --revision=v2.4.0 --depth 1 https://github.com/org/app.git

Only the last N commits arrive, and --depth implies --single-branch. It's the right call for a CI job that builds once and exits, and it's why actions/checkout defaults to fetch-depth: 1. The costs show up when you do anything with history. The last line (Git 2.49 and later) fetches exactly one tag or commit and creates no branches, leaving you on a detached HEAD, which suits a reproducible build. The costs of shallow history:

git log and git blame stop at the cut-off.
git merge-base fails when the common ancestor is beyond the boundary, so "diff against main" in CI breaks. Fetch more depth or the base branch.
git describe fails if no tag is inside the window.
Later fetches into a shallow repository are more expensive for the server to compute. For day-to-day development a blobless clone is almost always better.

Partial clone

git clone --filter=blob:none https://github.com/org/app.git
git clone --filter=tree:0 https://github.com/org/app.git
git clone --filter=blob:limit=1m https://github.com/org/app.git

A blobless clone downloads every commit and tree, so history, log and branch switching all work normally. File contents are fetched from the remote the first time something needs them. The remote is recorded as a promisor: Git trusts that it can supply missing objects later. A treeless clone (tree:0) goes further and is good for throwaway builds, but commands that walk history fetch trees one batch at a time and crawl. The trade-off is that git blame or git log -p on old history now needs the network, and fetches missing blobs one request at a time. git backfill (Git 2.49, still marked experimental) downloads the missing blobs in large batches grouped by path, so a developer can start working from a quick blobless clone and fill in history in the background. With --sparse it only fetches blobs inside your sparse-checkout.

Sparse-checkout

git clone --filter=blob:none --sparse https://github.com/org/monorepo.git
cd monorepo
git sparse-checkout set apps/web libs/ui
git sparse-checkout add libs/auth
git sparse-checkout list
git sparse-checkout disable

Sparse-checkout limits the working tree to the directories you name. Cone mode, the default since Git 2.37, takes directories rather than arbitrary patterns. You get everything under each named directory, plus the files directly inside each parent directory, including the repository root. That restriction is what lets Git check paths quickly. On its own sparse-checkout saves disk and speeds up status. Combined with a blobless clone, it also saves the download, because blobs outside your cone are never requested. For very large repositories, add git config index.sparse true so the index itself shrinks too.

Keeping a big repository fast

git maintenance start
git config core.fsmonitor true
git config core.untrackedCache true
git config feature.manyFiles true
git commit-graph write --reachable --changed-paths

Feature	What it speeds up
`git maintenance start`	Registers the repo and schedules background tasks through launchd, cron, systemd or Task Scheduler: hourly prefetch and commit-graph updates, daily loose-object cleanup and incremental repack, weekly packing of refs. It sets `maintenance.strategy=incremental` and turns off the automatic foreground gc for that repository. Prefetch downloads into hidden `refs/prefetch/` refs, so your `origin/*` branches do not move behind your back, but your next `git fetch` has little left to download.
commit-graph	A precomputed file of commit parents and generation numbers. `log --graph`, `merge-base` and reachability checks stop parsing every commit. `--changed-paths` adds Bloom filters, which make `git log -- some/file` much faster.
`core.fsmonitor`	The built-in file system monitor on macOS and Windows. `git status` asks a daemon what changed instead of checking every file.
`core.untrackedCache`	Remembers directory scans, so finding untracked files is cheaper.
`feature.manyFiles`	Turns on a group of settings for repositories with many files, including a smaller index format and the untracked cache.

For seniors: repacking at scale

A plain git gc rewrites every object into one pack. That is fine at 100 MB and painful at 50 GB: it is slow, can need about twice the disk while it runs, and competes for disk and CPU with everything else on the machine. The incremental strategy instead keeps a multi-pack-index, one index over many packs, and repacks a few small packs at a time. Current versions also offer maintenance.strategy=geometric, which keeps pack sizes in a geometric progression so most runs only merge the newest, smallest packs, and puts unreachable objects into a cruft pack when a full repack is needed. Large hosts run the same ideas server-side. If your repository has hundreds of thousands of refs, as CI systems that tag every build tend to create, the reftable backend is the other half of the fix, because the files backend rewrites packed-refs on every deletion.

Scalar

scalar clone https://github.com/org/monorepo.git
scalar register

Scalar ships with Git since 2.38 and came from Microsoft's work on the Windows repository. It applies everything in this section in one step: a blobless partial clone, cone-mode sparse-checkout starting from the root files only, background maintenance, fsmonitor and tuned config. scalar clone puts the working tree in a src/ folder inside the one it creates. scalar register applies the same settings to a repository you already have.

Finding out why it's slow

git count-objects -vH
GIT_TRACE2_PERF=1 git status
git rev-list --objects --all \
  | git cat-file --batch-check="%(objecttype) %(objectname) %(objectsize) %(rest)" \
  | awk '$1 == "blob"' | sort -k3 -n | tail -20
git diagnose

count-objects shows loose versus packed size. GIT_TRACE2_PERF prints where each phase of a command spent its time, so you can tell a slow untracked-file scan from slow hooks or a slow filter. The rev-list pipeline lists the twenty biggest blobs anywhere in history, which is usually the answer to "why is the clone 3 GB". The open-source git-sizer tool gives a fuller report, and git diagnose bundles repository statistics into a zip to attach to a bug report.

Line endings

Windows tools often write CRLF line endings, while Linux and macOS use LF. Left alone, a mixed team produces diffs where every line of a file looks changed, and shell scripts that fail with /bin/bash^M: bad interpreter. There are two controls, and one of them should win.

	`core.autocrlf`	`.gitattributes`
Where it lives	Each person's config	Committed in the repository
Applies to	Whoever set it	Everyone who clones
Values	`true`: CRLF on checkout, LF on commit. `input`: LF on commit only. `false`: no conversion.	`text=auto`, `text eol=lf`, `binary` and more, per path pattern
Priority	Lower	Wins when both are set

* text=auto
*.sh    text eol=lf
*.bat   text eol=crlf
*.png   binary
*.lock  -diff

text=auto lets Git detect text files and store them with LF. eol=lf forces LF in the working tree even on Windows, which scripts need. binary turns off conversion and textual diffs. -diff keeps a file as text but hides its diff, which is handy for lockfiles in review. After adding the file to an existing repository, run git add --renormalize . and commit, so the stored files match the new rules. git ls-files --eol shows what each file has in the index and on disk.

Case sensitivity

Linux file systems are case-sensitive. The default file systems on macOS (APFS) and Windows (NTFS) are case-insensitive but case-preserving. Git records the setting as core.ignorecase when it creates the repository. Leave it alone: flipping it by hand makes things worse. Three problems come up again and again.

A rename that changes only case. Renaming button.tsx to Button.tsx in Finder may not register as a change. Use git mv button.tsx Button.tsx. If that complains, go through a temporary name: git mv button.tsx tmp.tsx, then git mv tmp.tsx Button.tsx.
Imports that only work on a Mac. import "./button" resolves to Button.tsx on macOS and fails on Linux CI. Enable the case check in your bundler or TypeScript (forceConsistentCasingInFileNames).
Two paths that differ only by case. Someone on Linux commits both Readme.md and README.md. On a Mac only one can exist, so one is always shown as modified. Git warns about colliding paths on clone. The fix is to delete one of them in a commit made on a case-sensitive system, or with git rm --cached.

Windows adds its own rules: names like CON, AUX and NUL are reserved, and deep paths can hit the 260-character limit. git config core.longpaths true helps with the second.

Signing commits and tags

Anyone can set user.email to anything, so the author field is a claim, not proof. A signature proves the commit was made by whoever holds a particular private key. The host shows "Verified" when that key is registered to the account the email belongs to.

SSH signing, simpler than GPG

git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true
git config --global tag.gpgsign true

GPG signing

gpg --full-generate-key
gpg --list-secret-keys --keyid-format=long
git config --global user.signingkey 3AA5C34371567BD2
git config --global commit.gpgsign true
gpg --armor --export 3AA5C34371567BD2

Either way, upload the public key to your host as a signing key. On GitHub that is a separate entry from the SSH key you use to authenticate, even if it is the same key.

Verifying

git log --show-signature -1
git verify-commit HEAD
git verify-tag v2.4.0
git log --format="%h %G? %GS %s"
git config --global gpg.ssh.allowedSignersFile ~/.config/git/allowed_signers

%G? prints G for a good signature, N for none, B for bad, and a few other letters for untrusted or expired keys. For SSH signatures, local verification needs an allowed-signers file, one line per person: ana@example.com ssh-ed25519 AAAAC3Nza…. Without it Git can't tell whose key signed.

What a signature does not prove

It says who created the commit object, not that the code is good or reviewed. Rebasing or amending creates new commits, so the person who rewrites signs them. When a host squash-merges or rebases a pull request, it signs the result with its own key. If you need proof of review, enforce it with branch protection and required reviews, not with signatures.

Blame through the noise

git config blame.ignoreRevsFile .git-blame-ignore-revs
git blame -w -C -C -C src/cart.ts

List the full hashes of formatting-only commits in .git-blame-ignore-revs and commit that file. blame then skips them and credits each line to the commit that really changed it. GitHub and GitLab read the same file in their web blame views. -w ignores whitespace, and each -C looks further for code moved or copied from other files.