Encrypting backups with a key we never hold: the design, and what it doesn't protect
"Encrypted at rest" usually means the provider encrypts your data with a key the provider holds. That protects against a stolen disk. It doesn't answer the question a customer actually asks a backup service: can you read
"Encrypted at rest" usually means the provider encrypts your data with a key the provider holds. That protects against a stolen disk. It doesn't answer the question a customer actually asks a backup service: can you read my backups?
I build BackupLatte, a backup service for Linear and ClickUp. This month we shipped an option to seal stored backups with a key only the customer holds. Here is how it's designed, what we had to give up, a bug it caused that had nothing to do with cryptography, and the list of things it does not protect.
The constraint: backups run while nobody is watching
Backups run on a schedule, at 2 a.m., with no one at a keyboard. So the server must be able to encrypt without the customer, but must not be able to decrypt later.
That points to envelope encryption with an asymmetric key:
- The customer adds an RSA public key (3072 bits or more, exponent 65537). The private key stays with them.
- Every backup run gets a fresh random 256-bit data key. The data is encrypted with it.
- The data key is stored only wrapped: encrypted with RSA-OAEP (SHA-256, MGF1-SHA-256, empty label) for each of the account's active public keys.
An account can hold up to three active keys, and every data key is wrapped for all of them. That makes a spare key cheap: generate a second pair, put the private half somewhere offline, and either key opens every backup.
We stuck to boring, interoperable parameters on purpose. A wrapped data key opens with plain OpenSSL:
openssl pkeyutl -decrypt -inkey private-key.pem \
-pkeyopt rsa_padding_mode:oaep \
-pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256 \
-in wrapped.bin -out data-key.bin
Next to the wraps, the server stores a key check value: the first 16 bytes of HMAC-SHA-256 with the data key as the key and a fixed label as the message. It reveals nothing useful about the key. Its only job comes later, when a browser sends a data key back and the server needs to know it's the right one.
The object format: chunked AES-GCM
A snapshot can be large, and the worker streams it from the platform API to object storage. A single AES-GCM stream is authenticated only at its very end, so a reader would have to either buffer the whole object or act on data it hasn't verified yet. So every stored object is a sequence of separately authenticated chunks:
offset size field
0 4 magic "BLE1"
4 1 version = 1
5 16 data key id
21 32 random salt
53 4 chunk size (65536)
57 … chunks: AES-256-GCM ciphertext || 16-byte tag
- Each object gets its own key: HKDF-SHA-256 of the data key with that object's random salt.
- Because the key is unique per object, chunk nonces can be plain counters. The nonce is an 11-byte big-endian chunk counter followed by one byte that is
1for the last chunk and0otherwise. This is the STREAM construction that tools like age use. - The 57-byte header is the associated data of every chunk.
function chunkNonce(counter: number, last: boolean): Buffer {
const nonce = Buffer.alloc(12);
nonce.writeUIntBE(counter, 5, 6); // counter in the first 11 bytes, big-endian
nonce.writeUInt8(last ? 1 : 0, 11);
return nonce;
}
Swapping, reordering, repeating or dropping chunks fails authentication, and so does editing the header. The last-chunk flag catches truncation: cut the object at a chunk boundary and the new final chunk was sealed with a "not last" nonce. A damaged object is an error, never silent garbage. Memory use is one chunk whatever the object size, and the overhead is a 57-byte header plus 16 bytes per 64 KiB.
Two details that are easy to get wrong:
-
Compress before you encrypt. Datasets are gzipped first, then sealed. The sealed object is stored as
application/octet-streamwith noContent-Encoding, otherwise an S3 client may helpfully try to gunzip ciphertext. -
Don't sniff the magic bytes. Whether an object is sealed is recorded in the database, not guessed from its first four bytes. Customers' copied files are arbitrary, and one of them can start with
BLE1.
Restoring without sending the private key to the server
A restore has to write plain data back into Linear or ClickUp, so the server needs the plaintext at some point. What it never needs is the private key. The unlock goes like this:
- The customer picks their private key file in the browser. Unencrypted PEM goes straight into WebCrypto. A passphrase-protected key, or the legacy OpenSSL 1.x format, needs MD5 and 3DES or AES-CBC, which WebCrypto doesn't have, so
node-forgeis loaded only for that case. - The key is imported as a non-extractable
CryptoKeythat can only decrypt. - The browser asks for the wraps of the data keys this one snapshot needs: the snapshot's own key, plus the keys of earlier runs whose copied files it references. It unwraps them locally.
- It sends the unwrapped data keys, never the private key. The server checks each against its key check value and keeps them, encrypted under a separate server-derived key, with an expiry: 72 hours for a restore, 24 hours for an export.
The expiry is a usability decision. A restore into Linear can stop halfway for reasons outside our control: Linear is unavailable for a while, or access is revoked mid-run. Asking for the key again on every resume would be miserable, while keeping the keys forever would quietly undo the point of the feature.
Two guard rails on the way in:
- Proof of possession. When a public key is added, the server wraps a random 32-byte challenge for it and the browser has to open it. Without this step, a pasted public key with a typo, or someone else's key, would make every future backup unreadable, and nobody would notice until a restore.
- "I saved it" is checked, not trusted. If the browser generates the key pair (RSA-OAEP 4096 through WebCrypto), it downloads the private key and then asks the customer to select that file again before encryption can be switched on.
Changing keys works the same way. A new key seals new backups immediately. To let it open older snapshots, the browser unwraps their data keys with an old private key and wraps them again for the new public key. Only the small wraps change, and the snapshots themselves aren't rewritten.
What we gave up
Incremental collection. Some Linear datasets, like issue history and document versions, used to carry unchanged records over from the previous snapshot instead of fetching them again. With encryption on, the server can't read the previous snapshot. Any server-side cache that would bring the optimization back is, by definition, data the server can read. So for encrypted accounts those datasets are collected in full every day, and large workspaces back up more slowly. We documented that rather than working around it.
Content features. Anything that reads snapshot content on the server, such as content tools for AI clients, can't work on sealed snapshots.
A short window of server-held keys during a run. A backup can span several segments and retries, and a retry has to read the shards already written. So while a run is in flight its data key is also stored under a server-derived key. The copy is deleted when the snapshot commits. After a failed run it stays for at least 24 hours, until a cleanup job finds no retry that still needs it, so a retry can continue from its checkpoint instead of starting over.
Getting data out: a ZIP inside a ZIP
Customers need their data in a form they can open without us. An encrypted snapshot leaves only as a ZIP protected with AES-256 (WinZip AE-2), with a password generated in the browser:
const ALPHABET = 'ABCDEFGHJKLMNPQRSTUVWXYZ23456789'; // 32 symbols: no 0/O, no 1/I
const bytes = crypto.getRandomValues(new Uint8Array(28));
const password = [...bytes].map((b) => ALPHABET[b & 31]).join('');
That's 28 × 5 = 140 random bits. b & 31 doesn't bias the result, because 256 is a multiple of 32.
There's a catch in the ZIP format: it encrypts the contents of entries but not their names. A plain export lists names like data/issues.json and one file per attachment. So the protected ZIP holds a single entry, backuplatte-snapshot.zip, which is the normal export stored whole. Without the password, that one name is all you see.
The ZIP format itself has a usability cost, too. The archive tools built into Windows and macOS can list an AES-encrypted entry but can't open it. The app tells people to use 7-Zip, Keka or 7z before the download starts. Otherwise the first support question would be "your ZIP is broken".
The bug that had nothing to do with cryptography
On staging, the first real encrypted export failed three times in a row with ETIMEDOUT while talking to object storage.
The cause was Node's Happy Eyeballs. When a hostname resolves to several addresses, Node tries them in turn, giving each attempt 250 ms by default. That timer runs on the event loop. Our worker container is limited to half a CPU, and sealing the ZIP blocked the event loop for one to two seconds at a time. When the loop came back, the 250 ms timer had already fired and the connection was dropped, even though it had actually succeeded. The IPv6 fallback then failed immediately, because the host has no IPv6. The request ended with an AggregateError whose message was an empty string, so our failure log said nothing at all.
The fix was one line at startup in the API and the worker:
import { setDefaultAutoSelectFamilyAttemptTimeout } from 'node:net';
setDefaultAutoSelectFamilyAttemptTimeout(10_000);
A real connection failure still errors immediately, and an address that doesn't answer still falls through to the next one. We also started naming errors with empty messages by their code instead of storing an empty string. The general lesson: CPU-heavy work on the event loop doesn't only slow your own code down. It can break timers in code you never think about, like the socket connect inside an HTTP client.
What this doesn't protect
This is the part I'd want to read first if I were evaluating someone else's service.
- The live workspace. To make backups, the service holds OAuth tokens that can read and write the live Linear or ClickUp workspace. Someone who fully compromises the server gets those, whatever happens to the backups. Every backup service has the same problem, and encrypting the backups doesn't change it.
- Data in flight. During a backup, the server sees the data, because it reads it from the platform API itself. During a restore or export the customer unlocked, it sees the snapshot until the operation ends.
- The page that reads the key. The private key is parsed by app.backuplatte.com's own JavaScript. The app's Content Security Policy allows scripts only from its own origin and from Paddle, the payment provider, but a compromised frontend could still steal the key. That holds for any design where a web page handles the key.
- Metadata. The account email, workspace names and settings, snapshot dates, sizes and record counts, file checksums, types and sizes, error messages from the platform, and the record of what a restore created all stay readable. The service needs them to run, to show usage, and to resume restores.
- Lost keys. If every private key on an account is lost, the encrypted backups are gone for good. We can't help, because we never had the key. A spare key kept offline is the only protection.
So this isn't "zero-knowledge", and we don't call it that. What it does give you: the stored copies of your Linear and ClickUp data, the thing that sits in a bucket for a year, open only with a key you hold.
The customer-facing side, including setup, key rotation, and the full list of what stays unencrypted, is in the encryption guide.
If you've built customer-held-key encryption for a SaaS product, I'd like to hear how you handled the parts we found hardest: incremental work that needs the previous copy, and resuming long jobs without keeping keys around for too long.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.