A green boot does not prove your key manager is being used

Five production apps, three encryption keys each, all of them sitting in a Kubernetes secret as base64 plaintext. The plan was to move them into Scaleway's Key Manager so the cluster only ever holds ciphertext and the pod unwraps it at boot. Straightforward work, one afternoon, maybe two.

It took three days, and every single failure mode was the same shape: something booted fine and was quietly wrong.

The slot that resolves twice

Our framework has a env schema per app. A field can be marked as a key manager slot:

KUMIKO_SECRETS_MASTER_KEY_V1: base64Key32
  .describe("AES-256 master key (KEK) for tenant-secrets encryption.")
  .meta({ kumiko: { kms: true } }),

At boot, the runtime walks those slots, finds the _CIPHERTEXT twin for each one, calls the key manager, and hands the app an env with the plaintext filled in. One round trip, everything resolved, done.

Except two of our five apps also build their own crypto provider before the framework's boot path runs, because they need it to construct a context object the framework then receives. So the same slot gets resolved twice, from two different env objects, in two different places. Our boot log made this visible only because both passes log with a prefix:

[publicstatus] KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par
[runProdApp]   KUMIKO_SECRETS_MASTER_KEY_V1 source=key-manager keyId=441a92a0 region=fr-par

I had been reading only the second line for a week. The first one is the one that would have crashed.

Plaintext next to ciphertext wins, silently

The first version of the Pulumi code set both values, on the theory that having the plaintext around was a safe fallback during the migration. It is the opposite of safe. The resolver checks for a plaintext value first, and if it finds one it never calls the key manager at all. You get a pod that boots, serves traffic, passes every health check, and has never once talked to the KMS you just spent two days wiring up.

So the deployment code is either/or, with a comment that says why:

// Either/or, never both: a plaintext slot left beside its ciphertext
// wins silently, so the pod would boot green while still bypassing the
// Key Manager.
...(secretsMasterKeyCiphertext
  ? { KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: secretsMasterKeyCiphertext }
  : { KUMIKO_SECRETS_MASTER_KEY_V1: masterKey.base64 }),

If you are doing this migration, make the two states mutually exclusive in code before you touch any environment. A fallback that can win without telling you is worse than no fallback.

The schema ate the key before the resolver saw it

The second pass resolves against the parsed env, and Zod drops unknown keys. The _CIPHERTEXT twin is generated by the framework for the composed schema, but each app also hand builds a typed schema on top. If the twin is not declared there, the parsed env simply does not contain it, and the resolver gets an env where the ciphertext does not exist and the plaintext is empty. It throws, the pod crashloops, and the error message talks about a missing key rather than a stripped one.

The fix is one line per app, and it needs a comment, because the next person will read it as redundant with the framework's generated twin and delete it:

KUMIKO_SECRETS_MASTER_KEY_V1_CIPHERTEXT: z.string().min(1).optional()
  .describe("Key-Manager ciphertext, wrapped by the same key as the KEK. Must stay in this schema: bin/main.ts hands the PARSED env to resolvePlatformKeks, which never sees an undeclared key."),

I actually deleted it myself, mid migration, after convincing myself it was dead code. It was not.

Zod 4 throws on extend for refined schemas

When a slot arrives as ciphertext only, the plaintext field has to be relaxed to optional before parsing, otherwise the required check rejects exactly the deployment the feature exists to enable. The obvious implementation is schema.extend({ [name]: field.optional() }).

That works for four of our apps. The fifth one has an object wide superRefine for a cross field rule, and Zod 4 throws on .extend() for any schema carrying a refinement. It failed at boot, in production, on a Friday. The version that works uses safeExtend:

return Object.keys(relaxed).length === 0 ? schema : schema.safeExtend(relaxed);

A framework helper that manipulates a user supplied schema has to be tested against a refined schema. That is the case that behaves differently, and it is never the case in your fixtures.

The app that did not mount the feature but needed the key

One app never mounts the tenant secrets feature, so its schema had no reason to know about the master key slot. It does use multi factor auth, and the MFA code envelope encrypts TOTP secrets with the same provider, pulling the key straight out of process.env.

Name based greps did not find it. The provider resolves its keys by regex over the environment:

const match = /^KUMIKO_SECRETS_MASTER_KEY_V(\d+)$/.exec(name);

So no consumer ever types the key name, and a grep for the name finds nothing in the one app that most needed the change. What found it was grepping for the provider's constructor instead. If you are auditing who consumes a secret, grep for the thing that reads it, not for the name of the thing being read.

Verifying that the key did not change

The whole migration is only safe if the wrapped value decrypts back to the exact same bytes. Existing rows in the database were encrypted with that key, and a key that is merely valid will decrypt nothing.

The check that proves this is narrow. Encrypt the live value, then decrypt the fresh ciphertext and compare the raw plaintext field from the API response against the source string. Do not compare your own encode of the decode, which passes at any nesting depth and proves only that your codec is symmetric. We had already been burned by exactly that once, which is why the wrap script aborts unless the raw fields match:

if [ "$roundtrip" != "$plaintext" ]; then
  echo "!! $app: decrypt of the fresh ciphertext does NOT match the source value" >&2
  exit 1
fi

A monitor that reported green on an unchecked commit

Small one, but it nearly cost me a bad merge. I had a poll loop watching four pull requests, filtering the GitHub check rollup for anything failing and anything pending. No failures and nothing pending reads as green.

Right after a force push the rollup is momentarily empty, so for about thirty seconds a branch with zero checks run against it satisfies both conditions. My monitor announced green on a commit that CI had not looked at yet. The rewrite uses the exit code of gh pr checks instead, where 0 is passed and 8 is pending, so an empty rollup cannot masquerade as success.

Whenever you derive a positive state from the absence of negatives, ask what an empty input looks like.

What actually went wrong on rollout day

Two things, neither of them about keys.

Four apps rolled at once and hit remaining connection slots are reserved for roles with the SUPERUSER attribute, because the old pods still held their Postgres connections while the new ones started. One app restarted and came back. Worth staging your rollouts if your connection ceiling is anywhere near your replica count times your pool size.

And the rollback path was not what I assumed. The Pulumi code reads the ciphertext with config.require(...), so removing the config value does not revert anything, it makes the next pulumi up throw before it can do any work. The real rollback is deploying the previous revision of the infrastructure code. That is a thing to work out before you need it rather than during.

Where it landed

All five apps now log source=key-manager for all three key slots, in both resolution passes, and the plaintext slot is gone from every Kubernetes secret. The first app went out alone with a targeted apply, the other four followed once its boot log confirmed the path.

The part I would repeat is the log line that names its own pass. Almost every wrong turn above was visible in that log before it was visible anywhere else, and one of them was only visible there.