Storage systems

Three storage abstractions, and the one you should default to

"Where do the files go?" appears in nearly every design that touches photos, video, documents or backups, and it is a question with a boring correct answer that a surprising number of candidates miss. The three abstractions differ in what the storage layer knows about your data: block storage knows nothing but offsets, file storage knows a directory tree, object storage knows an immutable blob plus metadata and nothing else.

Notice the cost column spans two orders of magnitude for storing the same bytes. That gap is not a discount for being clever; it is what you pay for random-write latency you probably do not need. A 500 KB profile photo does not need sub-millisecond random access at any offset. It needs to be fetched whole, over HTTP, from a CDN.

Why object storage is the default, and what the durability number means

S3-class object stores advertise something like eleven nines of durability (99.999999999%). That is not marketing noise, but it also does not mean what people assume. It is an annual expected-loss figure: store ten million objects and you would statistically expect to lose one roughly every ten thousand years. It is achieved by erasure coding — the object is split into k data fragments plus m parity fragments spread across independent failure domains, and any k of the k+m fragments reconstruct it. Losing several disks, or an entire facility, loses nothing.

The other property to name explicitly is consistency. Modern S3 is strongly read-after-write consistent for new objects and overwrites, which removed a class of bug that used to bite everyone. But two caveats survive and are worth knowing: listing a bucket can still lag behind individual object writes at scale, and any CDN or client cache in front of the bucket reintroduces staleness that the storage layer's guarantee says nothing about. Cache-busting by putting a content hash in the key — writing new objects instead of overwriting them — sidesteps both.

Presigned URLs: get your app servers out of the byte path

This is the single most reliably-asked storage detail in an interview, and the naive design fails on it hard. If uploads are POSTed to your API, then every byte of every file traverses your app servers: they buffer it, hold a worker thread or event-loop turn for the whole transfer, consume bandwidth twice (in from the client, out to storage), and now your autoscaling is driven by upload volume rather than request volume. A single user on a slow connection uploading a 2 GB video occupies a server slot for minutes.

A presigned URL is a normal storage URL with a signature, an expiry and a set of constraints baked into the query string. Your app server holds the storage credentials; the client never does. Signing is a local cryptographic operation — no network call to the storage provider — so issuing them is essentially free.

The constraints matter, and interviewers probe them. A presigned PUT should pin: expiry (5-15 minutes, not 7 days), a maximum content length so a client cannot upload 500 GB, the exact object key which your server generates so clients cannot overwrite each other's objects, and ideally content type. The key should be server-chosen and unguessable — a UUID or content hash, never a user-supplied filename, which is both a collision problem and a path traversal problem.

Downloads get the same treatment in reverse. Serving private media through your API means proxying gigabytes; a presigned GET (or a signed CDN URL with a short TTL) lets the CDN serve it while still enforcing authorization, because the signature is the authorization and it expires.

Large media: chunked, resumable, and parallel

A single PUT is fine to about 100 MB and is capped around 5 GB. Beyond that, and on any mobile network, one long-lived request is the wrong shape: a dropped connection at 95% costs you the whole transfer.

Multipart upload solves this. The client initiates an upload and gets an upload ID, then uploads independent parts (minimum 5 MB each, up to 10,000 parts, which is what gets you to a 5 TB object), each returning an ETag. When all parts are in, the client sends a complete request listing the parts and their ETags, and the storage service assembles the object. Three properties fall out of that design and each is worth stating:

• Resumability. A failed part is retried alone. The client can query which parts already landed and resume after an app restart or a change of network.
• Parallelism. Parts are independent, so a client can run 4-8 concurrent uploads and saturate the link rather than being limited by a single TCP stream's throughput.
• Integrity. Per-part checksums catch corruption at part granularity instead of after the whole 5 GB transfer.

Each part can have its own presigned URL, so the bytes still bypass your app entirely. The one thing you must remember operationally: incomplete multipart uploads consume storage you are billed for and are invisible in a normal object listing. A lifecycle rule to abort uploads older than seven days is not optional, and mentioning it reads as operational experience.

Transcoding: the pipeline behind every video product

Raw uploads are unservable. A 4K phone recording is the wrong codec, bitrate and container for most viewers, so you generate a ladder of renditions and let the player adapt to the network.

Three design points to raise. First, transcoding is asynchronous by definition — the upload response returns immediately with a processing status and the client polls or gets pushed a notification, which is the message-queue chapter applied directly. Second, jobs must be idempotent and keyed on the source object plus rendition, so a worker crashing halfway just gets retried without producing duplicates. Third, the output is not one file per rendition but hundreds of small HLS/DASH segments plus a manifest, because that is what lets a player switch bitrate mid-stream — and it means the CDN serves a huge number of small cacheable objects rather than a few enormous ones.

Where not to put blobs: your relational database

Storing a file as a BYTEA or BLOB column is tempting because it gives you transactions and one backup story. It is almost always wrong at scale, for reasons worth being able to list quickly:

• It destroys your buffer pool. The database caches pages of hot rows in RAM. A 2 MB blob evicts hundreds of useful index and row pages to serve one request. Your unrelated queries get slower.
• Backups and restores become impossible to schedule. A 200 GB database backs up in minutes; the same database with 8 TB of images does not, and your recovery time objective quietly becomes hours.
• Replication amplifies it. Every blob write ships to every replica over the replication stream, which is exactly the traffic you least want in a channel whose lag governs your read consistency.
• You cannot put a CDN in front of a SQL query. Every byte comes out through your connection pool, competing with real queries for connections.
• Cost. Database storage is priced like block storage, roughly 3-10× object storage, and you are paying it for bytes you never query on.

The correct pattern: blob in object storage, row in the database holding the key, size, content type, checksum and status. You lose atomicity across the two — which is why the pending-then-confirmed state machine in the upload diagram exists, and why an orphan-sweeping job (objects with no row, rows with no object) belongs in the design.

Hot, warm, cold: tiering and what it actually saves

Access to stored data is extraordinarily skewed. A photo is viewed heavily in its first week and then approximately never — but must still be there in ten years. Tiering prices that reality.

Make the saving concrete. Take 5 PB of user media where 5% is read in any given month. All-hot costs about 5,000,000 GB × $0.023 = $115,000 a month. Move the 95% cold portion to deep archive and it becomes 250,000 × $0.023 + 4,750,000 × $0.00099 ≈ 5,750 + 4,700 = about $10,500 a month — a 90% cut, with the tradeoff that a rare access takes hours and costs a retrieval fee.

One more line item people forget entirely: egress. Pulling data out of a cloud region runs around $0.05-0.09 per GB, which is roughly four times the monthly cost of storing it. For a media-heavy product, bandwidth out of the CDN is usually a larger bill than the storage itself, which is the real financial argument for high cache-hit ratios.

Recognizing it in an unseen problem

• The prompt mentions photos, video, documents, avatars, attachments, backups or "user uploads" — object storage is the answer, and presigned direct upload is the detail that earns the point.
• A naive design POSTs the file to the API and stores it in the database, then wonders why the app tier autoscales on upload traffic and backups take six hours.
• Distinguish it from a caching question: caching is about latency for data you already have; storage tiering is about cost for data you rarely touch. They use the same vocabulary (hot/cold) and mean different things.
• Distinguish durability from availability the moment anyone says "we can't lose the data" — and add that neither one protects against a bad deploy, which is what versioning and object lock are for.
• Files over ~100 MB, or mobile clients, means multipart and resumable uploads; anything with video means an async transcoding pipeline with idempotent, segment-level jobs.
• Pitfall: designing the write path beautifully and forgetting the read path. Say how the bytes get back out — signed CDN URLs, cache TTLs, and who pays the egress.