Design a Dropbox-style file storage and sync service

Questions first: is this one folder per user, or shared folders with permissions? Do we keep version history, and for how long? What is the largest file, and is the average closer to 100 KB or 100 MB? Must two devices see a change within seconds, or is a minute fine? Do we need end-to-end encryption, which rules out server-side dedup and previews? How should we handle two edits to the same file?

I would assume per-user folders, 30 days of version history, an average file of 2 MB with a long tail into gigabytes, a few seconds of sync delay, and encryption at rest handled by the platform. In scope: upload, download, sync, versions, conflicts, dedup. Out of scope: sharing, search, previews and billing.

These are planning figures for the exercise, not facts about any company.

• Users: 10 million, 20% active a day = 2 million. Each changes 5 files a day: 10 million file changes ÷ 86,400 s = about 120 a second, 600 at a 5x peak.
• Bytes: 10 million × 2 MB = 20 TB a day, which is 20,000,000 MB ÷ 86,400 = about 230 MB a second, or roughly 1.9 Gbit a second; the peak is about 9 Gbit a second. If dedup and sending only changed chunks save half, new storage is 10 TB a day, which is 3.65 PB a year before replication.
• Chunks: take 2 per change on average (most files fit in one 4 MB chunk, the long tail adds many), which is 20 million references a day; half are new, so 10 million chunk rows at about 100 B: 1 GB a day.
• Versions: 10 million rows a day × 300 B = 3 GB a day, about 1.1 TB a year.
• Sync reads: each active client asks for changes 50 times a day: 100 million ÷ 86,400 = 1,160, call it 1,200 a second; 6,000 at peak.
• Notification connections: 2 million at 10 KB is 20 GB, or 4 to 8 gateway boxes.

Two conclusions. Bytes must flow straight to object storage, never through the API servers. Metadata writes (600 a second at peak) and reads (6,000) fit one relational primary at first, but I shard by user from the start: it is the key every query uses, it is cheap to do early, and metadata grows by over a terabyte a year.

Components. A desktop and mobile client with a local database; an API and metadata service; a metadata database; an object store for chunks; a notification service; a queue for async work (thumbnails, garbage collection, audit); and a background verifier.

Data model.

• files(user_id, file_id, path, current_version, deleted), sharded by user_id.
• versions(file_id, version, base_version, chunk_list, size, device_id, mtime).
• chunks(user_id, hash, size, storage_key, ref_count), sharded by user_id like the rest, so a commit stays on one shard. Within a user the chunk key is the content hash, so uploads are idempotent by construction.
• journal(user_id, seq, file_id, version): a per-user append-only change log with an increasing seq. This is what devices sync from: "give me everything after cursor N".

Upload path.

1. The client splits the file into chunks (4 MB here), hashes each, and asks the metadata service which hashes are missing.
2. The service returns presigned upload URLs for the missing chunks only. The client uploads them in parallel; a failed chunk is retried alone, which gives resume for free.
3. The client calls commit with the file id, the base_version it started from, and the chunk list.
4. The service checks that every listed chunk exists in the object store and has a chunks row. Then, in one transaction: check current_version = base_version, raise the ref counts, insert the new version, bump the pointer, append to the journal, write an outbox row. If the base does not match, the commit is a conflict (below).

Download and sync. The outbox feeds the notification service, which pushes "cursor moved" to the user's other devices. They call the change feed with their last cursor, then fetch only chunks they lack through presigned download URLs. Chunks already on disk are reused, so a small edit costs one chunk.

Hard problem 1: conflicts. Never overwrite silently and never trust timestamps (device clocks drift). With optimistic concurrency on base_version, the second writer's commit is rejected; the client saves its copy as a conflicted copy named with the device, and both versions remain. Users resolve them; auto-merge is only for formats we understand.

Hard problem 2: garbage collection. Dedup means a chunk can be shared. Keep ref_count from committed versions, but delete only after a grace period and never a chunk uploaded in the last day, since its commit may be in flight. A weekly mark-and-sweep reconciles counts, because counters drift.

Fault tolerance. Object storage replicates across zones; the metadata primary has a synchronous standby in a second zone. Ask the object store for a checksum on upload (check the current docs for what it supports); the verifier re-hashes a sample of chunks and flags mismatches, so corruption is caught before it is the only copy.

• Fixed-size vs content-defined chunking. Fixed 4 MB chunks are simple, but inserting a byte near the start shifts every later chunk and kills dedup. Content-defined boundaries keep unchanged regions stable at the cost of more client CPU. Pick fixed for the first version; flip to content-defined when users edit large files in the middle.
• Global vs per-user dedup. Global saves the most space, but answering "this hash exists" can reveal that someone else holds that file. Pick per-user dedup by default. Flip to global with a proof-of-possession step when storage cost dominates.
• Conflicted copy vs auto-merge. A copy never loses data and needs no format knowledge. Merging is nicer for text but dangerous when wrong. Pick the copy; flip to merge for formats with a safe merge rule.
• Push vs poll for sync. Push needs 4 to 8 gateway boxes for 2 million connections but gives seconds of delay. Polling every 30 s from 2 million clients is about 67,000 requests a second: stateless, but not cheaper, and slower. Pick push for desktop; on mobile in the background use the phone's own push service and poll when the app opens.

• Sending file bytes through the API servers. They would carry roughly 1.9 Gbit a second on average and 9 at peak for no benefit.
• Whole-file uploads, so one dropped connection restarts a 2 GB transfer.
• Last-write-wins by timestamp, which loses edits when clocks differ.
• Deleting a chunk the moment its count hits zero, racing with an in-flight upload.
• Ignoring the change feed: asking every device to rescan the whole tree.
• Hand-waving dedup without saying what the key is.
• Putting the file tree in one table with no per-user cursor, so sync means a full comparison every time.
• Forgetting that deletes, renames and moves are also changes that must reach every device.