One webhook in ten failed its signature check because of a trailing zero

A marketplace that sells our products sends us a webhook for every order, signed with an HMAC SHA-256 of the request body, which we check before accepting anything. In April we moved the receiver out of an old service and into our Node gateway. Nine days later sellers asked why some orders had never reached their dashboards.

About one webhook in ten had been rejected with a 401 since the move. The marketplace retried each one for three days and then gave up on it, as their documentation says they will.

The gateway applies a JSON body parser to every route, so our handler received an object rather than bytes. To check the signature the new code turned the object back into text with JSON.stringify and computed the HMAC over that. For most payloads this gives back exactly what was sent: the marketplace writes compact JSON, and JavaScript keeps the keys in the order they arrived. Numbers are different. The marketplace writes amounts with two decimal places, so a total arrives as 24.50. Parsing turns that into the number 24.5, and stringify writes 24.5. One byte shorter, a different HMAC, and a perfectly genuine order refused as a forgery. Any amount ending in a zero failed, whole pounds included, since 30.00 comes back as 30. That is roughly one order in ten.

Our tests passed because their fixtures had been written with JSON.stringify. Text that our own code produced always survives a trip through our own code.

We replayed nine days of events through the marketplace's events API, eleven hundred and forty orders, and sellers had them by the evening. The signature is now computed over the raw bytes of the body, captured on that route before any parsing, and compared in constant time. The body is parsed only after it has been verified. Test fixtures are real payloads captured byte for byte from the marketplace's sandbox, and they include amounts ending in zero, escaped characters and a long description. And 401s on webhook routes are a metric with an alert above one percent, because a signature failure is either an attack or a bug of ours, and either one is worth knowing about the same hour.

A signature is computed over bytes, not over meaning. Our code had checked that the order meant what it said, and then hashed a different spelling of it.

– Sergey Shinder