Design a firmware-update delivery system

Questions: how many devices, how large is an image, and how often do releases happen? Are devices always online or do they check in on a schedule? Do they have a spare partition so a failed update can roll back? Can we target subsets by model, region or version? How fast must a critical security fix reach everyone? Are some devices behind slow or metered links? Who may publish an update?

Assumed scope: publish a signed update, decide which devices get it and when, deliver the bytes, install with rollback, report status, halt a bad rollout. Out of scope: building the firmware itself, hardware design, and a customer-facing dashboard.

Planning figures for the exercise, not facts about any real fleet.

• Fleet and image: 50 million devices; a 40 MB full image.
• Total bytes for one release: 50,000,000 x 40 MB = 2,000,000,000 MB, which is 2 PB. If a delta update averages 5 MB, the total is 250 TB, one eighth of that.
• Time window: roll out over 7 days. 2 PB / (7 x 86,400 s) = about 3.3 GB a second, or about 26 Gbit a second sustained for full images; the delta case is about 3.3 Gbit a second. A critical fix pushed in 24 hours is about 23 GB a second, 185 Gbit a second. Either way this is CDN work, not origin work.
• Check-ins: each device asks "anything for me?" every 6 hours: 50,000,000 / 21,600 s = about 2,300 a second. With a 5x spike after a power outage, about 11,600 a second.
• Origin load with a CDN: if 98% of image bytes come from the edge, the origin serves 2% of 2 PB = 40 TB across the week.
• Device state: 50 million rows x 200 bytes = 10 GB.

What the numbers force: the check-in service is small (about 2,300 reads a second is well under what one primary does; 11,600 at the spike is still within its reads, but if every check-in wrote last_seen it would pass the 5,000 writes a second one primary commits, so batch or sample those updates), while the bytes must come from a CDN or other edge. The image must be an immutable object, and the manifest that points at it must be tiny and cacheable.

1. Build and signing. The release pipeline produces the image, computes a SHA-256 digest, and creates a manifest: version, hardware models allowed, minimum current version, image URL, size, digest, optional delta URLs, and a rollout id. The manifest is signed with a release key held in an HSM or managed key service; the image is signed too. Devices ship with the public verification key (and a way to rotate it, with a key id) and verify before installing. Publishing requires two-person approval.
2. Storage. Images and deltas are immutable objects in an object store, named by content digest, served through a CDN. Never overwrite an object; a new release is a new name. This makes caching safe and rollbacks trivial.
3. Update service. A device checks in with its id, model, current version and a signed device token. The service reads the rollout rules and returns either "nothing" or a manifest with a short-lived download URL. Rules are small and cached; device state (device_id, model, version, state, last_seen, rollout_id) lives in a key-value or relational store keyed by device id.
4. Rollout control. A rollout has target filters (model, region, version range) and a wave plan: for example 0.1%, 1%, 5%, 25%, 100%. A device is in a wave by a stable hash of device_id + rollout_id, so the share is close to the target and the same devices stay in. Between waves there is a bake time and automatic gates: install success rate, boot-health reports, crash rate, and rollback rate compared with the control group still on the old version.
5. Halt. A kill switch flips the rollout state; the update service stops returning the manifest on the next check-in (at most one check-in interval, plus a cache TTL of a minute). Devices that already downloaded but not yet installed re-check before the install step.
6. Download. Devices use HTTP range requests and resume after a drop, verify the digest of the finished file, and only then start the install. Stagger starts by a random delay to avoid a herd, honour a metered-link or battery policy, and prefer delta updates when the device's current version has one.
7. Install and rollback. A/B partitions: write to the inactive slot, verify the signature, set the boot flag to try the new slot once, reboot, and mark it good only after a health check passes within a time limit; otherwise the bootloader falls back to the old slot. The device reports downloaded, installed, healthy or rolled_back with the version.
8. Anti-downgrade. The manifest carries a monotonic security version; a device refuses lower ones, so an attacker cannot force an old vulnerable image. Pair that with a way to recall a bad release: publish a higher version rather than reuse an old one.
9. Observability. Funnel counts per wave (offered, downloaded, installed, healthy, rolled back), time to install, error codes by model, and download throughput. Alert on rollback rate against the control group.

Hard problem one: a bad image must not reach the whole fleet. Hash-based waves, bake time, health gates against a control group, and an A/B slot that restores the old firmware on its own.

Hard problem two: trust. Signed manifest and image, digest check, key rotation, and a security version against downgrade. Transport security alone is not enough, because the signature must hold even if a cache or mirror is compromised.

• Device pull vs server push. Pull (devices poll) works through firewalls and with sleeping devices and keeps the server simple, at the cost of up to one interval of delay. Push is faster for an urgent fix but needs a persistent connection per device. Pick pull. Flip to a push hint (a small wake-up message that triggers a check-in) for critical security fixes.
• Full image vs delta updates. Deltas cut bytes by about eight times in our example, but only work from known base versions and add build and test cost. Full images always work. Pick full images first with deltas for the most common source versions. Flip to deltas everywhere if the links are metered or bandwidth cost dominates.
• Fast rollout vs staged waves. A staged rollout limits the blast radius and takes days; going to everyone at once is quick and risks bricking the fleet. Pick waves. Flip to a fast schedule only for an actively exploited vulnerability, and keep at least a small first wave.
• A/B partitions vs single slot with recovery mode. A/B safely handles bad images and costs flash space (a second slot). A single slot saves space but a failed write can leave the device unable to boot without physical service. Pick A/B where flash allows. Flip to a minimal recovery image only on very constrained hardware.

• Serving 2 PB from the origin servers instead of an edge cache with immutable, content-addressed objects.
• Relying on HTTPS alone and never signing the image, so a compromised mirror can ship malware.
• Rolling out to everyone at once, or choosing waves at random on each check-in so the share drifts.
• Having no rollback path: a failed write leaves the device bricked.
• Letting the device mark the new slot good before a health check, so a boot loop becomes permanent.
• Allowing downgrades, which lets an attacker reinstall a known vulnerable version.
• Overwriting an object in place, so caches serve a mix of old and new bytes.
• Ignoring the herd after a power outage, when millions of devices check in at the same moment.