The migration that passed every check and still broke 140 websites

At 22:41 UTC on October 20, 2026, the team at Buttondown flipped a switch. They had moved their custom domains off Heroku and onto a CloudFront distribution, and they had checked their work. Every domain they had marked as valid, all 212 of them, loaded fine through the new setup. So they switched the DNS and went home.

Then the reports started coming in. Archives on about 140 custom domains would not load at all. Browsers showed a TLS error: ERR_SSL_VERSION_OR_CIPHER_MISMATCH. The sites were not slow. They were not showing the wrong page. They were refusing to open, as if the site had no identity at all.

The first report arrived at 02:14 UTC, in the middle of the night. It took until 14:28 UTC, more than twelve hours later, for an engineer to start investigating. The first report about the biggest group of broken domains arrived at 19:32 UTC, and when the team tried the domain themselves, it loaded fine for them. That delayed things further.

By the time the dust settled, the last two domains were not fixed until October 27, a full week after the switch. And the strangest part of the whole story is this: every check the team ran before the migration had passed. The checks were not wrong. They were checking the wrong list.

This is a detective story about a list, and about what happens when the thing you forgot to check is the thing that was never on the list in the first place.

Clue one: the error was not where anyone looked

The first thing to get straight is what the error actually meant. When you visit a website over HTTPS, your browser does two separate jobs before it shows you anything. First it asks DNS, the internet's address book, for the IP address of the domain. Then it asks the server at that address to prove it is who it claims to be, by showing a TLS certificate, a small signed document that ties the domain name to a cryptographic key. If the certificate is missing, or does not match the domain, the browser stops. It does not show you a broken page. It shows you an error and refuses to continue.

That is what was happening here. DNS was fine. The domains resolved. The servers answered. But there was no certificate to show, so the browser would not proceed. ERR_SSL_VERSION_OR_CIPHER_MISMATCH is the browser's way of saying: I found the house, but nobody could prove they owned it.

So the question was not "is the site up?" The question was "why do these domains have no certificate on the new system when they worked perfectly on the old one?"

Clue two: the list was smaller than the truth

Buttondown lets newsletter authors publish an archive of their issues on their own domain, like archive.example.com. To make that work, each custom domain needs its own setup on the hosting side. Buttondown calls each one a tenant: a small configuration that says "this domain belongs to this author, serve their archive here, and here is its TLS certificate." Authors point their domain at Buttondown with a CNAME record, which is a DNS entry that says "this name is really that other name over there." The CNAME target was custom-domains.buttondown.com.

Before the switch, the team created a tenant for every domain they had marked as valid, and tested all 212 of them through CloudFront. All 212 worked. The problem was the domains that were not on that list, and there were three kinds of them.

The first kind was the biggest: 146 domains that Buttondown had never verified. Here is the mechanism, and it is worth slowing down for, because it is the kind of thing that hides in plain sight in a lot of systems.

Buttondown only checked a custom domain's DNS when its settings page was loaded. If an author added a domain and nobody ever opened its settings page again, that domain sat in the database in an unverified state forever. On Heroku, that never mattered. Heroku served those domains anyway, no questions asked. The unverified flag was just a label in a database. It had no effect on anything.

But the migration only created tenants for verified domains. So 146 domains with perfectly good DNS, domains that had been working for months or years, ended up pointing at CloudFront with no tenant and no certificate. The database said "unverified." The real world said "working fine." The migration trusted the database.

The second kind was root domains using CNAME flattening. DNS has a rule: you cannot put a CNAME record on a root domain, the bare example.com with nothing in front of it. Some DNS providers, like Porkbun, get around this with a trick called flattening. They look up the target's IP addresses and publish those addresses directly, as if the root domain had them all along. It works fine for serving traffic. But CloudFront proves you own a domain by looking for the CNAME record, and flattening hides it. No visible CNAME, no proof of ownership, no certificate.

The third kind was domains proxied through Cloudflare. Same shape of problem, different cause. When Cloudflare proxies a domain, it sits in front of your site and hides your real DNS records from the outside world, including from CloudFront. CloudFront looked for the CNAME, could not see it, and never issued a certificate.

Three different causes, one shared shape: the new system needed proof that the old system had never asked for.

Clue three: the check that passed was checking itself

This is the part of the story I keep coming back to, because it is the easiest trap to fall into and the hardest to see from inside.

The team checked all 212 migrated domains, and all 212 worked. That check was real. It was careful. It was also useless for finding the problem, because it only covered domains that had been migrated. A check that only looks at the things you already handled will always pass. It cannot see what you left out, because the left-out things are not in its list.

The same trap showed up in detection. The first user report came in at 02:14 UTC, outside support hours, and reached an engineer at 14:28. When the first report about the never-verified domains arrived at 19:32, the team tried the domain and it loaded fine for them. Buttondown's postmortem does not say why it loaded for them and not for the author, and I will not guess. But the effect is clear: a check that passes on your machine proves nothing about the machine where it fails, and it cost them hours.

The fix, once they understood it, was straightforward. For the never-verified domains, they verified them by hand and created their tenants one by one. The certificates were issued and 140 of those domains were working again by 00:53 UTC on October 22. For the root domains, they asked authors to add a _cf-challenge DNS record, a special entry CloudFront can use to verify ownership even when the CNAME is flattened away. The permanent fix was attaching static IP addresses to the CloudFront distribution, which lets CloudFront verify those domains without any extra record. That took until October 27, because Buttondown hit its AWS quota for static IPs and had to wait for AWS to approve an increase. The last two root domains got their certificates at 21:26 UTC that day. For the Cloudflare-proxied domains, they reached out to the authors with the DNS changes needed on their end.

And then they changed the system so the trap could not happen again. They now verify never-checked domains every hour, instead of waiting for someone to load a settings page. Root domains work out of the box thanks to the static IPs. Future migrations will build the list of affected domains from DNS, the real source of truth about which domains exist, instead of from the database. And they added monitoring for TLS errors on custom domains, so they find out about this kind of failure before their users do.

The guard you can add this week

The lesson here is not "check your certificates." The lesson is: never let a migration's checklist come from the same place as the migration itself. Build the expected list from an independent source, then compare it against what you actually moved. Anything in the expected list but missing from the moved list is a domain, a record, a customer, or a file that your migration silently left behind.

Here is a small Python script that does exactly that check for custom domains. It takes a list of domains from DNS, or from your source system, and verifies each one over HTTPS on the new setup: does it answer, does it present a certificate, and is that certificate still valid?

import socket
import ssl
from datetime import datetime, timezone

def check_domain(domain, timeout=10):
    """Return (ok, reason) for one domain on the new setup."""
    ctx = ssl.create_default_context()
    try:
        with socket.create_connection((domain, 443), timeout=timeout) as sock:
            with ctx.wrap_socket(sock, server_hostname=domain) as tls:
                cert = tls.getpeercert()
                expires = datetime.strptime(
                    cert["notAfter"], "%b %d %H:%M:%S %Y %Z"
                ).replace(tzinfo=timezone.utc)
                days_left = (expires - datetime.now(timezone.utc)).days
                if days_left < 0:
                    return False, "certificate has expired"
                return True, f"certificate valid, {days_left} days left"
    except ssl.SSLCertVerificationError as exc:
        return False, f"no valid certificate: {exc.verify_message}"
    except ssl.SSLError:
        return False, "TLS handshake failed, probably no certificate at all"
    except (socket.timeout, ConnectionError, OSError) as exc:
        return False, f"could not connect: {exc}"

# The expected list comes from DNS, not from the migration's own records.
expected_domains = [
    "archive.example.com",
    "news.example.org",
    "blog.example.net",
]

missing_or_broken = []
for domain in expected_domains:
    ok, reason = check_domain(domain)
    print(("OK  " if ok else "FAIL") + f" {domain}: {reason}")
    if not ok:
        missing_or_broken.append(domain)

if missing_or_broken:
    print(f"\n{len(missing_or_broken)} domain(s) need attention before cutover.")
    raise SystemExit(1)
print("\nEvery expected domain answers with a valid certificate.")

Two details in that script are doing the real work. First, the list of domains comes from outside the migration. If you build the list from the same database the migration read, you will miss exactly the domains the migration missed. Second, the script treats anything it cannot verify as a failure. A domain whose status is "unknown" is not a domain that is fine. It is a domain you have not checked, and the script says so out loud.

You can run a version of this before any cutover, not just a domain migration. The shape is always the same: take the list from the source of truth, check every item on the destination, and refuse to proceed while anything is unaccounted for. The check costs minutes. The week Buttondown spent fixing this cost considerably more.

The quiet failure mode

Most outages are loud. Something crashes, an alarm fires, a graph falls off a cliff. This one was quiet. Nothing crashed. The old system kept serving the forgotten domains right up until the DNS switch, and then the new system served nothing at all for them, with no error on the server side to notice. The failure was not in the code that moved the domains. It was in the assumption that the database knew about all the domains.

That assumption had been true enough for years, right up until the day it was not. The flag that said "unverified" had never meant "broken," so nobody treated it as a risk. It meant "nobody has looked at this since it was added." And nobody had looked, because nothing had ever gone wrong.

Buttondown's own postmortem puts it plainly: when planning future migration work, they will build the list of affected domains from DNS instead of from their database, so they do not miss domains their database does not know about. It is a one-sentence fix for a week-long incident, and it is the kind of sentence that only gets written after the incident.

So here is my question for you. Where is the "unverified" flag in your system, the status field that is only ever set when a human happens to open a page, the list that everyone assumes is complete because it has always been complete enough? And what would you find if you built that list from the source of truth instead, just once, before your next migration?

All facts in this article come from Buttondown's own postmortem, "Archives on custom domains unreachable," published November 2, 2026: buttondown.com/blog/incident-0036