When a Renewed cert-manager CA Runs on the Same Key but a New Key Identifier
On cert-manager v1.21.2, a v1.18.6-issued rotationPolicy: Never CA renewed with the same key got a new SKI (SHA-1 → truncated SHA-256); OpenSSL accepted a pre-renewal leaf only with the old CA too. When It Runs — Run R
On cert-manager v1.21.2, a v1.18.6-issued rotationPolicy: Never CA renewed with the same key got a new SKI (SHA-1 → truncated SHA-256); OpenSSL accepted a pre-renewal leaf only with the old CA too.
When It Runs — Run Report #RR06 · Testing what infrastructure actually does.
Provider · Component: cert-manager · Subject Key Identifier generation on the CA
Certificaterenewal path (the controller's Gocrypto/x509CreateCertificate, withrotationPolicy: Never)
Versions tested: measured — cert-manager v1.18.6 issuing (the latest 1.18 patch; vendor-listed EOL 2026-03-10) → in-place upgrade through v1.19.6 → v1.20.4 (each minor's latest patch, following the vendor upgrade guidance; no behavioral assertions on the transit versions) → v1.21.2 renewing (current supported release; latest as of 2026-09-11). source-read — the controllercmd/controller/go.modgoline: v1.18.6go 1.24.0· v1.19.0go 1.25.0(the first affected version; the 1.19 line is EOL 2026-07-08) · v1.21.2go 1.26.0.
Config profile: a single-nodekindcluster (nodekindest/node:v1.33.4, digest-pinned) on a dedicated test environment; a SelfSignedIssuer→ a CACertificate(isCA: true,rotationPolicy: Never, RSA-2048) → a CAIssuer→ a leafCertificate; the CA renewed in place with the same key.
Verified on: 2026-10-13 (desk sources re-checked; thego.modlines, PR #8480 open and unmerged, and v1.21.2 the current supported release) · lab measured 2026-09-24 (v1.18.6 → v1.21.2).
Affects: cert-manager CACertificates that were issued withrotationPolicy: Neveron ≤1.18 and are then renewed with the same key on ≥1.19. The controller does not set the certificate's Subject Key Identifier itself, and from v1.19.0 the controller is built withgo 1.25.0, whosecrypto/x509fills a missing SKI with a truncated SHA-256 (earlier: SHA-1). A leaf issued before the renewal therefore carries an Authority Key Identifier derived from the old SHA-1 CA SKI, which no longer matches the renewed CA's SHA-256 SKI. Whether this breaks path building/verification is client-dependent — see §5 and §6; measured: with OpenSSL 3.0.13'sopenssl verify, the pre-renewal leaf failed against a trust store holding only the renewed CA and verified against the old and renewed CA bundled together.
TL;DR: On cert-manager v1.21.2, renewing a v1.18.6-issuedrotationPolicy: NeverCA with the same key left the public key unchanged but changed the CA's Subject Key Identifier from SHA-1 to a truncated SHA-256, so a pre-renewal leaf's Authority Key Identifier no longer matched the renewed CA — andopenssl verify(OpenSSL 3.0.13) rejected that leaf against the renewed CA alone ("unable to get local issuer certificate") while accepting it against the old and renewed CA bundled.
1. The Claim
This report starts from a promise operators lean on, a change recorded only one layer down, and a report that connects the two.
What the docs promise about rotationPolicy: Never. cert-manager's Certificate usage documentation, describing the rotationPolicy field, states (checked 2026-09-14) that with this setting "cert-manager reuses the existing private key on each issuance", and:
"With
rotationPolicy: Never, a private key is only generated if one does not already exist in the target Secret resource (using thetls.keykey). All further issuances will re-use this private key."
Never was the default until cert-manager 1.18: the 1.18 release notes announce "We have changed the default value of Certificate.Spec.PrivateKey.RotationPolicy from Never to Always" and advise operators to "Explicitly set rotationPolicy: Never on your Certificate resources" to keep the old behavior (checked 2026-09-14). So a long-lived CA issued on an older release, and kept on the old behavior, re-uses one private key across renewals. The surface an operator reads promises private-key continuity. It does not say anything about the continuity of the certificate's identifiers.
Where the identifier change is actually recorded. The Subject Key Identifier is generated by the Go standard library, and its generation changed. The Go 1.25 release notes state (checked 2026-09-14):
"
CreateCertificatenow uses truncated SHA-256 to populate theSubjectKeyIdif it is missing. TheGODEBUGsettingx509sha256skid=0reverts to SHA-1."
Whether a given build generates SKIs the new way (truncated SHA-256) or the old way (SHA-1) is decided by the go line of the main module, per Go's godebug documentation (checked 2026-09-14): a GODEBUG setting's default "is derived from … the defaults for the Go toolchain used to build the program, amended to match the Go version listed in go.mod, and then overridden by explicit //go:debug lines in the program." cert-manager's controller module declares go 1.25.0 from v1.19.0 onward (v1.18.6 declares go 1.24.0; v1.21.2 declares go 1.26.0 — read from the tagged go.mod), its packet-of-interest source does not set SubjectKeyId itself, and there is no //go:debug x509sha256skid=… or GODEBUG override in the module. A cert-manager ≥1.19 build therefore generates SHA-256 CA SKIs by default, and a ≤1.18 build generated SHA-1 ones. cert-manager's own release notes for 1.18 through 1.21 and its 1.17→1.18 and 1.18→1.19 upgrade guides do not mention this change (checked 2026-09-14).
The report this test is built on. In issue #8289 (checked 2026-09-14) a user described issuing a rotationPolicy: Never CA Certificate on v1.10.0, then — after upgrading to v1.19.1 — renewing it and finding that the private key was unchanged but the CA's SKI was different. The reporter traced the cause to the Go 1.25 change and closed the issue themselves as "not a bug." Two later comments recorded operational impact (for example, disruption to Galera clusters) — a report layer this report does not itself reproduce.
What this report tests. Not whether the docs are wrong or whether Go's default is wrong — both are documented where they live. The gap is that a rotationPolicy: Never CA renewed across the go 1.24 → 1.25 boundary keeps its key but changes its SKI, so a leaf signed before that renewal carries an Authority Key Identifier that points at an SKI the renewed CA no longer has. §3 measures the identifiers before and after the renewal; whether any given client then fails to build a path is a separate, client-dependent question handled in §5 and §6.
2. The Setup
-
Environment: a single-node
kindcluster on a dedicated Linux test VM (GNU coreutils), node imagekindest/node:v1.33.4(digest-pinned). k8s v1.33 is inside the vendor-listed supported range of every cert-manager version exercised here (1.18: k8s 1.29–1.33; 1.19: 1.31–1.35; 1.20: 1.32–1.35; 1.21: 1.33–1.36), so one node version covers the whole issue → upgrade → renew sequence;kubectlwithin ±1 minor of the node. The SKI mechanism is independent of the Kubernetes version. -
Version: cert-manager v1.18.6 to issue (installed from the official Helm chart
cert-manager-v1.18.6.tgzfrom charts.jetstack.io with one non-default value,crds.enabled=true, so the chart installs the CRDs (chart defaultfalse); the in-place upgrades below use each minor's official chart with the same value; image digests and chart hashes recorded in our lab record). The 1.18 line is EOL (vendor-listed 2026-03-10); v1.18.6 is its latest patch. The controller is then upgraded in place to v1.19.6 → v1.20.4 → v1.21.2 — one minor at a time, each the minor's latest patch, following the vendor guidance "We recommend that you upgrade cert-manager one minor version at a time, always choosing the latest patch version for the minor version." No behavior is asserted on the transit versions; the renewal is measured on v1.21.2 (the current supported release). The first affected version, v1.19.0 (go 1.25.0), is read from source only. -
Configuration: a SelfSigned
Issuer; a CACertificatewithisCA: true,rotationPolicy: Never, RSA-2048; a CAIssuerreferencing that CA's Secret; and a leafCertificateissued by the CAIssuerbefore the upgrade. After upgrading to v1.21.2 the CACertificateis renewed in place, which underrotationPolicy: Neverkeeps the same private key.
3. Expected vs. Observed — does a same-key CA renewal keep the CA's Subject Key Identifier stable?
Expected (what the rotationPolicy: Never promise, read by itself, suggests): renewing the CA with the same key keeps the CA's identity stable, so a leaf issued before the renewal still chains to the renewed CA.
Observed (measured 2026-09-24; raw in our lab record). The measurements taken, before the renewal (on v1.18.6) and after (on v1.21.2):
- CA public-key fingerprint before vs. after the renewal: unchanged — SPKI SHA-256
84b323b6927d5dfe25b242dce19f4287ed28c1f2b456d3f4f603588eb2bf1a57both times. The renewal kept the key; only the certificate was re-issued (serial25034c5688dd96891327b113cfc02eb4→19b489e55575e31270275311b74a2454832fb881). - CA Subject Key Identifier before (v1.18.6):
42b062a23924df0a7492534af9e3d64925be0106(20 bytes) — equal to the SHA-1 of the CA's public key (the subjectPublicKey bit string), which we computed from the certificate itself. - CA Subject Key Identifier after the renewal (v1.21.2):
9de807cf05452aca93f19f6f1ca515f1ceba4809(20 bytes) — equal to the first 20 bytes of the SHA-256 of that same public key (the truncated SHA-256 form), and different from the value above. The in-place upgrades to v1.19.6, v1.20.4 and v1.21.2 did not re-issue the CA or the leaf on their own (both certificates were byte-identical after each step); the renewal was triggered on v1.21.2 withcmctl renew(cmctl v2.6.1). - the pre-renewal leaf's Authority Key Identifier vs. the renewed CA's Subject Key Identifier: the leaf issued on v1.18.6 carries AKI
42b062a23924df0a7492534af9e3d64925be0106— the old CA SKI — which does not match the renewed CA's SKI9de807cf05452aca93f19f6f1ca515f1ceba4809. The leaf itself carries no SKI. - client verification of the pre-renewal leaf against each measured trust store (the client-verification step): one client, OpenSSL 3.0.13 (
openssl verify -show_chain -CAfile <store> <leaf>). (a) Renewed CA alone:error 20 at 0 depth lookup: unable to get local issuer certificate, exit code 2. (b) Old CA + renewed CA bundled:OK(chain depth 1CN = rr06-ca), exit code 0. No optional client (GnuTLS, Java, Go) was run in this round; the trust-store file hashes are in our lab record.
Raw evidence: the CA certificate before and after the renewal and the pre-renewal leaf, the openssl x509 output of their Subject Key Identifier and Authority Key Identifier extensions, and the public-key fingerprints (hex) are kept in our lab record.
4. Reproduction
The minimal reproduction (raw output is kept in our lab record; the manifest and the commands below are in the reproduction repo linked below):
# 0. Bring up a kind cluster (node k8s v1.33.4) on the test VM and install cert-manager v1.18.6
# from the official Helm chart, with the CRDs (chart hash + image digests recorded; rollout
# confirmed before any measurement):
# kind create cluster --name rr06 --image kindest/node:v1.33.4@sha256:25a6018e48dfcaee478f4a59af81157a437f15e6e140bf103f85a2e7cd0cbbf2
# helm install cert-manager https://charts.jetstack.io/charts/cert-manager-v1.18.6.tgz \
# -n cert-manager --create-namespace --set crds.enabled=true --wait
# for d in cert-manager cert-manager-webhook cert-manager-cainjector; do kubectl -n cert-manager rollout status "deploy/$d" --timeout=300s; done
# 1. Create a SelfSigned Issuer, then a CA Certificate that re-uses one key across issuances
# (Issuer: spec.selfSigned: {}; Certificate: spec.isCA: true, spec.privateKey.rotationPolicy: Never,
# algorithm RSA size 2048, issuerRef -> the SelfSigned Issuer, secretName -> the CA secret).
# Then a CA Issuer referencing that CA secret, and a leaf Certificate issued by the CA Issuer.
# kubectl apply -f step1-manifest.yaml # the manifest below, saved to this file
# kubectl -n rr06 wait --for=condition=Ready certificate/rr06-ca --timeout=120s
# kubectl -n rr06 wait --for=condition=Ready issuer/rr06-ca-issuer --timeout=120s
# kubectl -n rr06 wait --for=condition=Ready certificate/leaf1 --timeout=120s
# 2. RECORD #1 (still on v1.18.6): the CA's Subject Key Identifier (SHA-1, 20 bytes) and public key,
# the leaf's Authority Key Identifier (== the CA SKI above), and the CA public-key fingerprint (SPKI SHA-256).
# kubectl -n rr06 get secret rr06-ca-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ca-1_v1.18.6.pem
# kubectl -n rr06 get secret leaf1-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > leaf1-1_v1.18.6.pem
# openssl x509 -in ca-1_v1.18.6.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier
# openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl pkey -pubin -outform DER | sha256sum # CA public-key fingerprint (SPKI SHA-256)
# openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha1sum # SHA-1 of the public key (the subjectPublicKey bit string)
# openssl x509 -in ca-1_v1.18.6.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha256sum | cut -c1-40 # the first 20 bytes of its SHA-256
# openssl x509 -in leaf1-1_v1.18.6.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier
# 3. Upgrade cert-manager IN PLACE, one minor at a time, each the latest patch, per the vendor guidance:
# v1.18.6 -> v1.19.6 -> v1.20.4 -> v1.21.2
# helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.19.6.tgz \
# -n cert-manager --set crds.enabled=true --wait
# helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.20.4.tgz \
# -n cert-manager --set crds.enabled=true --wait
# helm upgrade cert-manager https://charts.jetstack.io/charts/cert-manager-v1.21.2.tgz \
# -n cert-manager --set crds.enabled=true --wait
# (confirm rollout + version at each step; no renewal, no assertions on the transit versions.)
# 4. On v1.21.2, RENEW the CA Certificate with the same key (rotationPolicy: Never keeps the key):
# trigger an in-place renewal of the CA Certificate and wait for it to complete: the revision has
# changed and Ready is True (at most 120 one-second reads).
# rev0=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.revision}'); echo "CA revision before: $rev0"
# cmctl renew -n rr06 rr06-ca
# for i in $(seq 1 120); do rev=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.revision}'); rdy=$(kubectl -n rr06 get certificate rr06-ca -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'); [ "$rev" != "$rev0" ] && [ "$rdy" = True ] && break; sleep 1; done
# echo "CA revision after: $rev (Ready=$rdy)"
# 5. RECORD #2 (on v1.21.2, after the renewal): the CA's Subject Key Identifier now, compared to RECORD #1;
# the CA public-key fingerprint, compared to RECORD #1 (expected unchanged); and the pre-renewal leaf's
# Authority Key Identifier, compared to the renewed CA's Subject Key Identifier.
# kubectl -n rr06 get secret rr06-ca-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ca-2_v1.21.2_renewed.pem
# openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -subject -issuer -serial -ext subjectKeyIdentifier,authorityKeyIdentifier
# openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl pkey -pubin -outform DER | sha256sum # CA public-key fingerprint (SPKI SHA-256)
# openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha1sum # SHA-1 of the public key (the subjectPublicKey bit string)
# openssl x509 -in ca-2_v1.21.2_renewed.pem -noout -pubkey | openssl rsa -pubin -RSAPublicKey_out -outform DER 2>/dev/null | sha256sum | cut -c1-40 # the first 20 bytes of its SHA-256
# 6. CLIENT VERIFICATION of the pre-renewal leaf, in two trust stores:
# (a) the renewed CA alone; (b) the old CA + the renewed CA, bundled.
# Required client (path that reads AKI/SKI): openssl verify. OpenSSL's issuer test, in addition to
# matching the issuer DN, compares the sub-fields of the subject's authorityKeyIdentifier (keyid,
# serial, issuer) against the issuer's subjectKeyIdentifier, serial number, and issuer when present
# (X509_check_issued(3)). Optional, recorded without a path classification: GnuTLS certtool --verify,
# Java keytool/PKIX, Go crypto/x509 Verify. Client names, versions, and trust-store file hashes: our lab record.
# cp ca-2_v1.21.2_renewed.pem store-a_renewed-CA-only.pem
# cat ca-1_v1.18.6.pem ca-2_v1.21.2_renewed.pem > store-b_old-CA+renewed-CA.pem
# openssl verify -show_chain -CAfile store-a_renewed-CA-only.pem leaf1-1_v1.18.6.pem # (a)
# openssl verify -show_chain -CAfile store-b_old-CA+renewed-CA.pem leaf1-1_v1.18.6.pem # (b)
# 7. Tear down (the test cluster only).
# kind delete cluster --name rr06
The manifest for step 1 — the SelfSigned Issuer, the CA Certificate, the CA Issuer, and the leaf Certificate, in their own namespace — exactly as we applied it:
apiVersion: v1
kind: Namespace
metadata:
name: rr06
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned
namespace: rr06
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: rr06-ca
namespace: rr06
spec:
isCA: true
commonName: rr06-ca
secretName: rr06-ca-tls
privateKey:
algorithm: RSA
size: 2048
rotationPolicy: Never
issuerRef:
name: selfsigned
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: rr06-ca-issuer
namespace: rr06
spec:
ca:
secretName: rr06-ca-tls
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: leaf1
namespace: rr06
spec:
commonName: leaf1.rr06.example.test
dnsNames:
- leaf1.rr06.example.test
secretName: leaf1-tls
privateKey:
algorithm: RSA
size: 2048
issuerRef:
name: rr06-ca-issuer
kind: Issuer
Outputs for each RECORD step land in §3 (above) and our lab record.
A recorded run of this reproduction: https://youtu.be/GMwCRX-3BhU
Reproduction repo: https://github.com/whenitruns/when-a-renewed-cert-manager-ca-runs-on-the-same-key-but-a-new-key-identifier
5. Impact
A client that builds a chain by matching a leaf's Authority Key Identifier to a CA's Subject Key Identifier can, against a trust store that holds only the renewed CA, fail to build a path for a leaf that was issued before the renewal — even though the CA's key never changed and the leaf has not expired (the exact clients and trust stores are measured in the client-verification step; in this run, openssl verify failed with the renewed CA alone and succeeded with the old and renewed CA bundled). The belief this quietly breaks is "the key didn't change, so certificates already issued by this CA keep working." Renewing a rotationPolicy: Never CA in place is the kind of routine action taken for exactly that reason. This is an availability and interoperability property, not a security weakness: nothing here lets a certificate be forged or a check be bypassed. Reported production incidents (e.g. Galera clusters) sit in the report layer of #8289 and are not reproduced here.
6. What We Couldn't Conclude
- Client behavior is not exhaustive. We report only the clients and trust-store configurations we measured (the client-verification step). Whether a given library, language runtime, or middlebox builds the path by DN, by key identifier, or by trying alternates is a per-client question; "TLS verification breaks" is not a general claim of this report. The reporter's own note in #8289 — that "openssl verify does succeed" for the bundled certificate while some applications do not — is consistent with that client-dependence.
- The private key. This report's claim is about the identifier, not the key. If a renewal ever changed the public-key fingerprint, that would be key rotation — a different mechanism, outside this report — and we record the before/after fingerprint precisely so that case is visible rather than assumed. The #8289 reporter confirmed the key was unchanged.
- "Before the renewal," not "before the upgrade." The affected leaf is one issued before the CA is renewed. We do not measure every issue-time/upgrade-time ordering.
- The reported production impact (Galera and similar) is a report-layer account by other users, not our reproduction; we do not attribute those incidents.
-
Out of measured scope: other trust-store shapes, other key algorithms, and intermediate-CA topologies are not covered. The transit versions (v1.19.6, v1.20.4) and the first affected version (v1.19.0) are not measured for renewal behavior — v1.19.0 is read from
go.modonly; the renewal is measured on v1.21.2. -
Clients that tolerate the mismatch. A client whose path logic tolerates the AKI/SKI mismatch could still verify the pre-renewal leaf, and that outcome would belong to this finding too: the mechanism (key unchanged, SKI SHA-1 → SHA-256, AKI/SKI link broken) holds whether or not a given client tolerates it. In this run,
openssl verifydid not tolerate it — it failed against the renewed CA alone (§3, §5).
7. Conclusion
On the versions and in the configuration recorded above, renewing a rotationPolicy: Never CA Certificate across cert-manager's go 1.24 → 1.25 build boundary kept the CA's private key while changing its Subject Key Identifier from SHA-1 to a truncated SHA-256, leaving a pre-renewal leaf's Authority Key Identifier pointed at an identifier the renewed CA no longer carries — and openssl verify then rejected that leaf against the renewed CA alone while accepting it with the old CA bundled in; the measured identifiers and the client-verification results are recorded as of the Verified-on date and within this scope only. The change is documented where it originates — the Go 1.25 release notes and the godebug reference — and cert-manager does not set the identifier itself; the point of this report is that the surface an operator reads for rotationPolicy: Never, which promises key re-use, is not where the identifier change is written down.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.