Heartbleed: One Unchecked Length, Two Years, Half a Million Websites (and the C# Version)

In April 2014, about half a million trusted websites had the same bug. Ask them the right way, and they'd hand you up to 64 KB of their memory: passwords, session cookies, private messages, sometimes the private key that protects the whole site. The attacker didn't break in. They sent one small message and lied about one number.

This post walks through Heartbleed using the primary sources: the OpenSSL advisory, the commit that added the bug, the commit that fixed it, and the RFC it was supposed to implement. At the end there's the part that matters for .NET developers, because the same mistake can still happen in safe C#, just more quietly.

The heartbeat

When you see the lock in your browser, your connection is protected by TLS. In 2014, a huge part of the web did that locking with OpenSSL, the free, open-source library behind Apache and nginx.

In early 2012, OpenSSL 1.0.1 added the heartbeat extension, from a new standard, RFC 6520. The idea is simple. A secure connection can sit idle for a while, so one side asks "are you still there?", and the other side answers. The request carries a small payload and says how long it is. The response must echo that payload back.

Say the payload is the word PING, four bytes. The server copies those four bytes into its answer and sends them back. That's a heartbeat.

One number nobody checked

Now, what if the request lies? It sends PING, four bytes, but claims the length is 64,000.

Here's the vulnerable function from OpenSSL 1.0.1f (ssl/t1_lib.c, tls1_process_heartbeat), trimmed:

hbtype = *p++;
n2s(p, payload);   /* read the length the client claims */
pl = p;            /* pointer to the payload */
/* ... */
buffer = OPENSSL_malloc(1 + 2 + payload + padding);
bp = buffer;
*bp++ = TLS1_HB_RESPONSE;
s2n(payload, bp);
memcpy(bp, pl, payload);   /* copy `payload` bytes, trusting the client */

payload comes straight from the attacker's message. Nothing compares it with the number of bytes that actually arrived. So memcpy starts at the four real bytes and keeps going, into whatever sits next in the server's heap: other people's requests, their passwords and cookies, sometimes key material.

The length field is two bytes, so the largest value is 65,535: about 64 KB per heartbeat. And nothing limited the number of heartbeats. As heartbleed.com put it, the 64 KB limit applies to a single heartbeat, not to the attack.

Worst of all, it was silent. Exploiting it left nothing abnormal in the logs.

The irony is that the standard already said what to do. RFC 6520: if the payload_length of a received message is too large, the message must be discarded silently. The code just never did that.

Two years in plain sight

The heartbeat code was written by one of the authors of the heartbeat RFC. A core OpenSSL developer reviewed it and committed it on December 31, 2011, New Year's Eve. The author later said publicly that he missed a length check, the reviewer missed it too, and it wasn't deliberate.

OpenSSL 1.0.1 shipped on March 14, 2012, with the heartbeat on by default. Then it sat there through 2012 and 2013, on servers all over the world. "Given enough eyeballs, all bugs are shallow" is a nice line, but plenty of people used this code and very few read it.

Two teams, one week

In early 2014, Neel Mehta of Google Security was auditing OpenSSL. He found the missing check, and it was reported privately to the OpenSSL team on April 1, 2014. On April 3, the Finnish security company Codenomicon found the same bug independently. They gave it a name, a bleeding-heart logo and its own website.

On April 7, OpenSSL published the advisory and version 1.0.1g. The fix is exactly what the RFC asked for:

if (1 + 2 + payload + 16 > s->s3->rrec.length)
    return 0; /* silently discard per RFC 6520 sec. 4 */

Netcraft checked that week: about 17.5% of SSL sites with trusted certificates had the heartbeat extension enabled, around half a million certificates.

What leaked

Patching stopped new leaks, but nobody could know what had leaked before, because there was no trace in the logs.

The scariest question was the private key. Cloudflare first concluded that stealing it was "at least extremely hard and, likely, impossible", and put up a vulnerable server as a public challenge. About nine hours later, two people had extracted its key; one of them sent at least 2.5 million requests. Cloudflare updated its post: they had been wrong. So patching wasn't enough. Affected sites had to revoke their certificates and get new keys.

In Canada, the tax agency took its online services down in the middle of tax season. It still wasn't fast enough: the agency reported that about 900 social insurance numbers were taken through Heartbleed, over a window of about six hours.

Who was holding it up

Then came the second question: who pays for OpenSSL?

Almost nobody. In a post from April 12, 2014, Steve Marquess of the OpenSSL Software Foundation wrote that it typically received about $2,000 a year in donations. Most of its money came from support contracts and consulting, and only one team member worked on it without another full-time job. In the week after the news, about two hundred people donated, mostly $5 or $10, around $9,000 in total.

On April 24, 2014, the Linux Foundation launched the Core Infrastructure Initiative, with companies including Google, Microsoft, Amazon and Facebook each pledging at least $100,000 a year for three years. OpenSSL was the first project funded.

Could this happen in C#?

The classic version, mostly no.

• A C# array knows its own length. Read past the end and you get an IndexOutOfRangeException, not your neighbor's data.
• A new array starts out zero-initialized, so there's no old secret sitting in it.
• Windows itself wasn't affected out of the box: it uses its own TLS stack, SChannel, not OpenSSL, as Microsoft said at the time.

But modern .NET has one place where the old danger comes back: ArrayPool<T>. To avoid allocations, you rent a buffer and give it back when you're done, so the next caller gets the same array. The docs for Rent say two things:

1. The returned array is at least minimumLength long. It can be bigger.
2. The returned array may not be zero-initialized. It can still hold whatever the previous renter left in it.

So this is a small Heartbleed, written entirely in safe code:

byte[] buffer = ArrayPool<byte>.Shared.Rent(256);
try
{
    int written = WriteReply(buffer);      // writes, say, 40 bytes
    await stream.WriteAsync(buffer);       // BUG: sends buffer.Length bytes,
                                           // including old data from another request
}
finally
{
    ArrayPool<byte>.Shared.Return(buffer);
}

The fix is the same idea as OpenSSL's: only send what you actually wrote.

await stream.WriteAsync(buffer.AsMemory(0, written));

And the closer cousin: trusting a length the client sent you, and slicing that much out of a pooled buffer. If the claimed length is smaller than the rented array, nothing throws. You just copy out bytes that were never part of this request. Check the claimed length against the bytes you actually received, and if it doesn't match, drop the message.

If a pooled buffer ever holds secrets, Return also takes a clearArray flag:

ArrayPool<byte>.Shared.Return(buffer, clearArray: true);

What to take from it

1. Every number from outside is a claim, not a fact. Lengths, counts, offsets: check them against what you actually received.
2. No trace in the logs doesn't mean nothing happened. With Heartbleed, everyone had to assume the worst and rotate keys.
3. The code you depend on is maintained by real people. Sometimes very few of them.

What's the one open-source library your app can't live without, and do you know who maintains it? Tell me in the comments.

I make Vlad's Stack: short explainers and stories about how the tools you use every day actually work, for .NET and full-stack developers. The channel is here.