Storage systems

Three storage abstractions, and the one you should default to

"Where do the files go?" appears in nearly every design that touches photos, video, documents or backups, and it is a question with a boring correct answer that a surprising number of candidates miss. The three abstractions differ in what the storage layer knows about your data: block storage knows nothing but offsets, file storage knows a directory tree, object storage knows an immutable blob plus metadata and nothing else.

Notice the cost column spans two orders of magnitude for storing the same bytes. That gap is not a discount for being clever; it is what you pay for random-write latency you probably do not need. A 500 KB profile photo does not need sub-millisecond random access at any offset. It needs to be fetched whole, over HTTP, from a CDN.

Why object storage is the default, and what the durability number means

S3-class object stores advertise something like eleven nines of durability (99.999999999%). That is not marketing noise, but it also does not mean what people assume. It is a design target for annual expected object loss, not a contractual guarantee: store ten million objects and you would statistically expect to lose one roughly every ten thousand years. AWS describes how it gets there as redundant storage on multiple devices across at least three Availability Zones, continuous checksum verification, and fast detection and repair of lost redundancy, so that the loss of a whole AZ loses nothing. The generic technique behind object stores of this kind is erasure coding — split the object into k data fragments plus m parity fragments spread across independent failure domains, where any k of the k+m fragments reconstruct it — but treat that as the textbook mechanism, not a statement of S3's internals.

The other property to name explicitly is consistency. Modern S3 is strongly read-after-write consistent for PUT, DELETE and LIST, in every Region, since December 2020 — a listing issued after a successful write shows it. That removed a class of bug that used to bite everyone. The caveats that survive are different ones: bucket-level configuration (for example, enabling versioning, or deleting a bucket then listing buckets) is still eventually consistent; there is no atomic update across keys; two concurrent writers to one key are last-writer-wins; and any CDN or client cache in front of the bucket reintroduces staleness that the storage layer's guarantee says nothing about. For the concurrent-writer case S3 now supports conditional writes (If-None-Match: * to create only if absent, If-Match with an ETag to overwrite only if unchanged), which gives you a compare-and-set on a single key. Cache-busting by putting a content hash in the key — writing new objects instead of overwriting them — sidesteps the cache problem and the overwrite race.

Presigned URLs: get your app servers out of the byte path

This is the single most reliably-asked storage detail in an interview, and the naive design fails on it hard. If uploads are POSTed to your API, then every byte of every file traverses your app servers: they buffer it, hold a worker thread or event-loop turn for the whole transfer, consume bandwidth twice (in from the client, out to storage), and now your autoscaling is driven by upload volume rather than request volume. A single user on a slow connection uploading a 2 GB video occupies a server slot for minutes.

A presigned URL is a normal storage URL with a signature, an expiry and a set of constraints baked into the query string. Your app server holds the storage credentials; the client never does. Signing is a local cryptographic operation — no network call to the storage provider — so issuing them is essentially free.

The constraints matter, and interviewers probe them. A presigned PUT should pin: expiry (5-15 minutes, not 7 days), the exact object key which your server generates so clients cannot overwrite each other's objects, and content type. Size needs care: you can sign the exact Content-Length you expect, but a presigned PUT cannot express "any size up to N". For a size ceiling use a presigned POST with a content-length-range condition, or re-check the size from the object-created event and delete oversize objects. The key should be server-chosen and unguessable — a UUID or content hash, never a user-supplied filename, which is both a collision problem and a path traversal problem.

Downloads get the same treatment in reverse. Serving private media through your API means proxying gigabytes; a presigned GET (or a signed CDN URL with a short TTL) lets the CDN serve it while still enforcing authorization, because the signature is the authorization and it expires.

Large media: chunked, resumable, and parallel

A single PUT is fine up to about 100 MB (AWS suggests multipart from there) and is capped at 5 GB. Beyond that, and on any mobile network, one long-lived request is the wrong shape: a dropped connection at 95% costs you the whole transfer.

Multipart upload solves this. The client initiates an upload and gets an upload ID, then uploads independent parts (5 MiB to 5 GiB each, up to 10,000 parts), each returning an ETag. The part-count cap sets the part size: a 5 TB object over 10,000 parts needs parts of at least ~524 MiB, and the maximum 48.8 TiB object needs parts near 5 GiB. When all parts are in, the client sends a complete request listing the parts and their ETags, and the storage service assembles the object. Three properties fall out of that design and each is worth stating:

• Resumability. A failed part is retried alone. The client can query which parts already landed and resume after an app restart or a change of network.
• Parallelism. Parts are independent, so a client can run 4-8 concurrent uploads and saturate the link rather than being limited by a single TCP stream's throughput.
• Integrity. Per-part checksums catch corruption at part granularity instead of after the whole 5 GB transfer.

Each part can have its own presigned URL, so the bytes still bypass your app entirely. The one thing you must remember operationally: incomplete multipart uploads consume storage you are billed for and are invisible in a normal object listing. A lifecycle rule to abort uploads older than seven days is not optional, and mentioning it reads as operational experience.

Dry run: one 2.3 GB upload through the presigned-URL flow above

Track which actor touches the file's bytes: the app server never does — it only ever handles the few small JSON calls in steps 1, 3 and 5 (a multipart upload adds the part-signing and complete calls), independent of whether the upload is 200 KB or 200 GB. And the pending → ready transition is driven entirely by the storage layer's own event in steps 5-6, never by the client's word that it finished — exactly the trust boundary the warning box above is about.

Transcoding: the pipeline behind every video product

Raw uploads are unservable. A 4K phone recording is the wrong codec, bitrate and container for most viewers, so you generate a ladder of renditions and let the player adapt to the network.

Three design points to raise. First, transcoding is asynchronous by definition — the upload response returns immediately with a processing status and the client polls or gets pushed a notification, which is the message-queue chapter applied directly. Second, jobs must be idempotent and keyed on the source object plus rendition, so a worker crashing halfway just gets retried without producing duplicates. Third, the output is not one file per rendition but hundreds of small HLS/DASH segments plus a manifest, because that is what lets a player switch bitrate mid-stream — and it means the CDN serves a huge number of small cacheable objects rather than a few enormous ones.

Where not to put blobs: your relational database

Storing a file as a BYTEA or BLOB column is tempting because it gives you transactions and one backup story. It is almost always wrong at scale, for reasons worth being able to list quickly:

• It destroys your buffer pool. The database caches pages of hot rows in RAM. A 2 MB blob evicts hundreds of useful index and row pages to serve one request. Your unrelated queries get slower.
• Backups and restores become impossible to schedule. A 200 GB database backs up in minutes; the same database with 8 TB of images does not, and your recovery time objective quietly becomes hours.
• Replication amplifies it. Every blob write ships to every replica over the replication stream, which is exactly the traffic you least want in a channel whose lag governs your read consistency.
• You cannot put a CDN in front of a SQL query. Every byte comes out through your connection pool, competing with real queries for connections.
• Cost. Database storage is priced like block storage, roughly 3-10× object storage, and you are paying it for bytes you never query on.

The correct pattern: blob in object storage, row in the database holding the key, size, content type, checksum and status. You lose atomicity across the two — which is why the pending-then-confirmed state machine in the upload diagram exists, and why an orphan-sweeping job (objects with no row, rows with no object) belongs in the design.

Hot, warm, cold: tiering and what it actually saves

Access to stored data is extraordinarily skewed. A photo is viewed heavily in its first week and then approximately never — but must still be there in ten years. Tiering prices that reality.

Prices are approximate US East list prices at the time of checking; they vary by region and change, so confirm on the provider's pricing page.

Make the saving concrete. Take 5 PB of user media where 5% is read in any given month. All-hot costs about 5,000,000 GB × $0.023 = $115,000 a month. Move the 95% cold portion to deep archive and it becomes 250,000 × $0.023 + 4,750,000 × $0.00099 ≈ 5,750 + 4,700 = about $10,500 a month — a 90% cut, with the tradeoff that a rare access takes hours and costs a retrieval fee. That is the steady-state number; getting there has a bill of its own. Lifecycle transitions are charged per object (about $0.05 per 1,000 objects into Deep Archive), and Glacier-class storage adds roughly 40 KB of metadata overhead per object (8 KB billed at Standard rates, 32 KB at archive rates). If the 4.75 PB being archived is 500 KB photos, that is about 9.5 billion objects and a one-time transition charge near $475,000, against savings of roughly $104,000 a month: about four and a half months to pay back, plus the 180-day minimum storage charge if you delete early. With 100 MB videos the same transition is a rounding error, so object size decides whether tiering is worth it.

One more line item people forget entirely: egress. Pulling data out of a cloud region to the internet lists at up to about $0.09 per GB for the first tier (the first 100 GB a month is free, and the rate falls with volume), which is on the order of four times the monthly price of storing the same gigabyte. For a media-heavy product, bandwidth out of the CDN is usually a larger bill than the storage itself, which is the real financial argument for high cache-hit ratios.

Operational details worth one sentence each

• Request rates. S3 scales per prefix: at least 3,500 write and 5,500 read requests per second per prefix, with no limit on the number of prefixes. Spreading keys across prefixes parallelises hot workloads, and a 503 "Slow Down" means you are scaling faster than the prefix has partitioned.
• Small files. Billions of tiny objects make per-request cost and per-object overhead dominate. Systems built for photos (Facebook's Haystack and f4) pack many small blobs into large files with an in-memory index to avoid a metadata lookup per read; the lesson for a design is to batch small items into larger objects when you control the write path.
• Recovery targets. Say the recovery point objective (how much data you can lose) and recovery time objective (how long you can be down) for the blob store, and back them with a 3-2-1 plan (three copies, two media, one off-site or cross-account) and versioning plus object lock against deletion.
• Origin protection. Put the CDN in front and restrict the bucket to the CDN's identity, so a cache miss storm or a scraper cannot hit the bucket at full price.

Recognizing it in an unseen problem

• The prompt mentions photos, video, documents, avatars, attachments, backups or "user uploads" — object storage is the answer, and presigned direct upload is the detail that earns the point.
• A naive design POSTs the file to the API and stores it in the database, then wonders why the app tier autoscales on upload traffic and backups take six hours.
• Distinguish it from a caching question: caching is about latency for data you already have; storage tiering is about cost for data you rarely touch. They use the same vocabulary (hot/cold) and mean different things.
• Distinguish durability from availability the moment anyone says "we can't lose the data" — and add that neither one protects against a bad deploy, which is what versioning and object lock are for.
• Files over ~100 MB, or mobile clients, means multipart and resumable uploads; anything with video means an async transcoding pipeline with idempotent, segment-level jobs.
• Pitfall: designing the write path beautifully and forgetting the read path. Say how the bytes get back out — signed CDN URLs, cache TTLs, and who pays the egress.