🔑 The Leak That Wasn't Really About the Leak

Every few months there's another story: a private key sitting in a public GitHub repo, a .pem file attached to a support ticket, a wildcard cert's key copied into three different Slack channels "just in case." The headline is always about the leak, but the actual root cause is almost never "someone was careless." It's that the organization had no system that made carelessness hard.

If a key can be emailed, it will eventually be emailed. If a cert can live for three years because nobody wants to touch the renewal process, it will live for three years, unmonitored, with its key copied onto four servers and one engineer's laptop. The fix isn't a better reminder in the calendar. It's removing humans from the path entirely.

This post walks through what that actually looks like: a cert lifecycle built on Vault's PKI engine and ACME, where keys are generated close to where they're used, certs are short-lived by default, and renewal is a cron job instead of a ticket.

🧨 Why Manual Key Handling Keeps Failing

A few failure modes show up over and over in postmortems:

• Key reuse across environments. The same key/cert pair gets copied from staging to prod because "it already works." Now a staging compromise is a prod compromise.
• Long validity windows as a crutch. Nobody wants to touch the renewal process, so certs get issued for 1–3 years. That's 1–3 years of exposure if the key leaks, with no forcing function to rotate.
• Keys generated outside the server that uses them. Someone generates a CSR and key on their laptop, emails the key over, and now there's a copy sitting in a Sent folder forever.
• No revocation path. Even when a leak is discovered, half these orgs don't have CRL/OCSP infrastructure actually wired up, so "revoke" just means "we meant to reissue eventually."
• No audit trail. Nobody can answer "who issued this cert, when, and for what identity" without digging through someone's terminal history.

Notice that none of these are cryptography problems. They're process and tooling problems. That's good news, because it means they're automatable.

🏗️ The Shape of an Automated Pipeline

The target architecture is boring on purpose:

1. Vault as the private CA (or root of trust for intermediate issuance), issuing short-lived leaf certs.
2. ACME as the protocol clients use to request/renew certs — either via Vault's native ACME support or a bridge like step-ca fronting Vault.
3. cert-manager (for Kubernetes workloads) or acme.sh (for VMs/bare servers) as the client that actually talks ACME and handles renewal scheduling.
4. No human ever touches a private key. Keys are generated on the node that needs them and never leave it.

The key property we're optimizing for: if a cert leaks today, the blast radius is measured in hours, not years, because it was going to expire and rotate anyway.

🔧 Standing Up Vault as a Private CA

Start with Vault's PKI secrets engine as an intermediate CA:

bash
vault secrets enable -path=pki_int pki
vault secrets tune -max-lease-ttl=43800h pki_int

vault write -format=json pki_int/intermediate/generate/internal \
common_name="example.com Intermediate Authority" \
| jq -r '.data.csr' > pki_intermediate.csr

Sign with your root CA (also in Vault, ideally offline/sealed except when signing)

vault write -format=json pki_root/root/sign-intermediate \
csr=@pki_intermediate.csr format=pem_bundle ttl="43800h" \
| jq -r '.data.certificate' > intermediate.cert.pem

vault write pki_int/intermediate/set-signed [email protected]

Then define a role that constrains what can actually be issued — this is where you encode policy instead of trusting a human to remember it:

bash
vault write pki_int/roles/web-server \
allowed_domains="example.com" \
allow_subdomains=true \
max_ttl="72h" \
key_type="ec" \
key_bits=256

max_ttl=72h is doing a lot of work here. It means even if a requester asks for something longer, Vault caps it. No more three-year certs because nobody circled back.

Lock issuance down with a policy so only the right identities can request certs for the web-server role:

hcl
path "pki_int/issue/web-server" {
capabilities = ["create", "update"]
}

🤖 ACME: Let the Protocol Do the Boring Part

Vault added native ACME support, so clients that already speak ACME (which is most of them) can request certs without any custom integration code:

bash
vault write pki_int/config/cluster \
path="https://vault.example.com/v1/pki_int" \
aia_path="https://vault.example.com/v1/pki_int"

vault write pki_int/config/acme enabled=true

For Kubernetes workloads, wire that up as a cert-manager ClusterIssuer:

yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: vault-acme
spec:
acme:
server: https://vault.example.com/v1/pki_int/acme/directory
privateKeySecretRef:
name: vault-acme-account-key
solvers:
- http01:
ingress:
class: nginx

And a Certificate resource that a deployment actually consumes:

yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: web-server-cert
namespace: prod
spec:
secretName: web-server-tls
duration: 72h
renewBefore: 24h
dnsNames:
- api.example.com
issuerRef:
name: vault-acme
kind: ClusterIssuer

cert-manager watches the expiry, requests a new cert 24 hours before expiry, and replaces the Kubernetes secret in place. The private key is generated inside the cluster and never leaves it.

For non-Kubernetes hosts, acme.sh does the same job with a cron entry:

bash
acme.sh --issue -d api.example.com \
--server https://vault.example.com/v1/pki_int/acme/directory \
--webroot /var/www/html

acme.sh --install-cert -d api.example.com \
--key-file /etc/nginx/ssl/api.key \
--fullchain-file /etc/nginx/ssl/api.fullchain.pem \
--reloadcmd "systemctl reload nginx"

acme.sh already registers a twice-daily cron check that only actually renews when the cert is within its renewal window — no separate scheduler needed.

🔄 Short-Lived Certs Change the Threat Model

This is the part that's easy to undersell: moving from 1-year to 72-hour certs isn't just "more rotation." It fundamentally changes what a leaked key is worth. A key with a 72-hour lifespan that leaks has a tiny, bounded exposure window and the attacker is racing an expiry clock that's already ticking regardless of whether anyone notices the leak.

It also forces you to actually solve automated renewal, because manual renewal every three days is untenable — which means the "we'll rotate it properly eventually" failure mode structurally can't happen. You've made the secure path the only convenient path.

📋 Closing the Loop: Monitoring and Revocation

Automation doesn't mean "set and forget." You still want:

• Expiry/issuance metrics scraped from Vault's audit log or cert-manager's certmanager_certificate_expiration_timestamp_seconds metric, alerting if a cert gets within a renewal window without a successful reissue.
• CRL/OCSP actually tested, not just configured — run a quarterly game day where you revoke a test cert and confirm clients reject it.
• Audit logging enabled on Vault so every issuance is attributable to a workload identity, not a person.

🙋 So, Where Are Your Keys Right Now?

If you had to answer "which private keys in our infrastructure are more than 90 days old, and who has copies of them," could you? For a lot of teams the honest answer is "no idea," and that uncertainty is the actual vulnerability — not any specific leaked key. What's the oldest cert in your environment right now, and do you know why it's still manually managed?