ACME Automation Failures: Renewal Outages, Rate Limits & DNS-01 Pitfalls

ACME made certificates free and automatic, which is exactly why expiry outages still happen: nobody watches a cron job that has worked for two years. Then a firewall rule changes, a DNS API token rotates, or a deploy script wipes the client's config directory, and the renewal fails silently for 30 days until browsers start throwing NET::ERR_CERT_DATE_INVALID. With certificate lifetimes shrinking, the gap between "renewal broke" and "site down" keeps getting shorter.

I wrote this with help from AI tools and checked every command and claim myself against current Let's Encrypt documentation. I build PixelValut, a set of free browser-based developer and network tools, and this is Part 5 of Web PKI & Certificate Forensics in Practice. Part 3 covered revocation and short-lived certificates. Part 4 covered Certificate Transparency monitoring. Here we debug the automation that issues all those certificates.

📚 Series Roadmap — Web PKI & Certificate Forensics in Practice

• Part 1: Leaked TLS Private Keys: Proving Compromise, Finding Every Affected Cert & Rotating Safely
• Part 2: Reading X.509 Certificates by Hand: SANs, Key Usage, Chains & Intermediates
• Part 3: Certificate Revocation in 2026: CRLs, the Decline of OCSP & Short-Lived Certificates
• Part 4: Certificate Transparency Monitoring: Catching Unexpected Certificates for Your Domains
• Part 5 (This Post): ACME Automation Failures: Renewal Outages, Rate Limits & DNS-01 Pitfalls
• Part 6: Mutual TLS for Internal APIs: Private CAs, Rotation & Pinning Trade-offs

Related: Subdomain Takeover & Dangling DNS Reconnaissance · Zero-Trust Reverse Proxy Hardening in Nginx

Why do ACME certificate renewals fail in production?

Direct answer: Most ACME renewal outages come from three things: validation paths that changed (HTTP-01 blocked by redirects, firewalls or CDNs; DNS-01 broken by expired API tokens or slow propagation), clients that lose state and burn rate limits by re-issuing, and nobody alerting on certificate age. Fix it with ARI-aware clients, staging dry runs, and expiry monitoring independent of the client.

The failure taxonomy

The last row is the sneakiest: Certbot renews, the file on disk is fresh, and Nginx keeps serving the old certificate from memory.

Let's Encrypt rate limits, stated correctly

Rate limits attract the most folklore. These are the current production limits from Let's Encrypt's documentation (token buckets that refill gradually):

Three details matter during an incident:

• Renewals via ACME Renewal Info (ARI) are exempt from all rate limits. Non-ARI renewals (same exact identifier set) skip the per-domain and per-account order limits but still count against the exact-set and authorization-failure limits.
• Revoking certificates does not reset rate limits. Revoke-and-retry makes things worse.
• Staging has much higher limits. Point every test, rebuild and CI job at https://acme-staging-v02.api.letsencrypt.org/directory.

The classic self-inflicted outage: an entrypoint runs certbot certonly on every start with no persistent /etc/letsencrypt. A few crash-loop restarts exhaust "5 per exact set", then you wait ~34 hours per token.

HTTP-01 pitfalls and a correct Nginx config

Let's Encrypt follows redirects for HTTP-01, but app-level redirects, auth middleware and CDN rules often break the path. Serve the challenge directory explicitly before anything else:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;

    # Must come before the catch-all redirect
    location ^~ /.well-known/acme-challenge/ {
        root /var/www/acme;
        default_type "text/plain";
        try_files $uri =404;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

Validation runs from multiple network perspectives, so geo-blocking or IP allowlists on port 80 can fail it even when your own curl works. Check the path from outside: request a test file and confirm the status code with the PixelValut HTTP status checker, or:

echo ok | sudo tee /var/www/acme/.well-known/acme-challenge/probe
curl -sI http://example.com/.well-known/acme-challenge/probe
# Expect: HTTP/1.1 200 OK, not 301 to an app route or 403 from a WAF

DNS-01 pitfalls: propagation, delegation and wildcards

Wildcards require DNS-01, and it is the fallback when port 80 cannot be exposed. Its failures look different:

1. Propagation. Your provider's API returns success before all authoritative nameservers serve the TXT record. Query the authoritative servers directly, not your resolver cache.
2. Wrong zone. _acme-challenge.app.example.com must exist in whichever zone is authoritative for that name. Delegated subzones catch people out.
3. Split-horizon DNS. Internal views hide what the public sees.
4. Over-privileged tokens. A zone-wide DNS token on every web server is a takeover risk.

# Check what the CA will actually see: query each authoritative NS
for ns in $(dig +short NS example.com); do
  echo "== $ns"; dig +short TXT _acme-challenge.example.com @"$ns"
done

# CNAME delegation: point the challenge name at a dedicated validation zone
# _acme-challenge.example.com.  CNAME  _acme-challenge.example.com.acme-validation.example.net.
dig +short CNAME _acme-challenge.example.com

With CNAME delegation, the client only needs write access to a small validation zone. Verify the CNAME, TXT and CAA records from a public vantage point with the PixelValut DNS lookup tool.

CAA records that help instead of hurt

CAA lets you restrict which CAs may issue for your domain. RFC 8657 adds accounturi and validationmethods, which pin issuance to your ACME account and allowed challenge types:

example.com.  CAA 0 issue "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/123456789; validationmethods=dns-01"
example.com.  CAA 0 issuewild "letsencrypt.org; validationmethods=dns-01"
example.com.  CAA 0 iodef "mailto:[email protected]"

This pairs with Part 4: CT tells you after the fact, CAA prevents other accounts from issuing at all. The trade-off: a new ACME account fails issuance until you update the record, so put it in the runbook.

Monitoring that does not trust the client

Never trust the client to report failure. Monitor the served certificate:

#!/usr/bin/env bash
# cert-age-check.sh: alert if the served cert expires within N days
host="$1"; days="${2:-14}"
cert=$(echo | openssl s_client -connect "$host:443" -servername "$host" 2>/dev/null)
end=$(openssl x509 -noout -enddate <<<"$cert" | cut -d= -f2)
if ! openssl x509 -noout -checkend $((days*86400)) <<<"$cert" >/dev/null; then
  echo "ALERT $host expires $end (< ${days}d)"; exit 1
fi
echo "OK $host expires $end"

# Certbot: test renewal against staging and reload on success
sudo certbot renew --dry-run
sudo certbot renew --deploy-hook "systemctl reload nginx"
systemctl list-timers | grep -i certbot

Set the threshold relative to lifetime. The default classic profile still issues 90-day certificates, but Let's Encrypt's published plan moves it to 64 days on February 10, 2027 and 45 days on February 16, 2028; the opt-in tlsserver profile is already 45 days and shortlived is about six days. Hardcoded "renew at 60 days" schedules will break, so prefer ARI windows or renewing around two-thirds of lifetime.

Diagnostic checklist

1. Does an external monitor check the served certificate's expiry on every hostname?
2. Is the ACME client state (/etc/letsencrypt, acme.sh home, cert-manager secrets) on persistent storage?
3. Do all test and CI runs use the staging directory?
4. Does your client support ARI so renewals are rate-limit exempt?
5. Is /.well-known/acme-challenge/ served before redirects, auth and WAF rules?
6. Is port 80 reachable from all regions, not just your office?
7. For DNS-01, do you verify TXT records on every authoritative nameserver?
8. Is the DNS API token scoped (or CNAME-delegated) and its expiry tracked?
9. Do CAA records match your current CA and ACME account?
10. Does a deploy hook reload the server, and do you verify the new serial is served?

FAQ

Does revoking a broken certificate let me issue again sooner?

No. Let's Encrypt states revocation does not reset rate limits. Fix the cause, then wait for the bucket to refill.

I hit "5 per exact set of identifiers". What now?

Wait for refill (1 every 34 hours), or add another hostname to change the set. That new set is not a renewal and counts against the per-domain and per-account limits.

Is DNS-01 more secure than HTTP-01?

Neither is inherently safer. DNS-01 avoids exposing port 80 but puts DNS write credentials somewhere. CNAME delegation and scoped tokens reduce that risk.

Why does validation fail when curl works from my laptop?

Validation runs from multiple vantage points; geo-blocking, allowlists or regional DNS can fail some.

Should Kubernetes clusters use cert-manager or a cron'd client?

cert-manager, usually: state lives in Secrets and renewals reconcile. Still monitor externally.

What do shorter certificate lifetimes change?

Less tolerance for silent failures. Automate fully, enable ARI, and alert on overdue renewals rather than days remaining.

Next in this series, Part 6 covers mutual TLS for internal APIs: running a private CA, rotating client certificates without downtime, and when pinning helps or hurts.

Follow & Connect

• Website: PixelValut Tools
• YouTube: PixelValut (@pixel-valut)
• TikTok: PixelValut (@pixelvalut007)
• GitHub: Muhammad Ali Mughal (@ali111887)
• LinkedIn: Muhammad Ali Mughal