The agent sorted the JSON body keys to tidy it up, and the request signature stopped matching

We handed the agent a webhook client that signs each outbound payload with HMAC before sending. It read the integration guide, built the serializer, and the local tests passed against a stub that accepted any well-formed body. Before opening the PR, it decided the body looked messy: the keys arrived in the order the fields were populated, so it ran a recursive sort to produce alphabetical, deterministic JSON.

The tests still passed. CI went green. The first production delivery returned 401 signature_invalid, intermittently — only for some payloads, because the receiver computed its digest over the exact bytes it received and expected the same field order the sender used in the signing step. The agent had treated object key order as cosmetic; for a signed payload, it is part of the contract.

Why the tests stayed green

The test asserted that the body parsed and that the HMAC function was called with some secret. It never compared the digest against the exact serialized bytes the receiver would rebuild, and it never exercised the real signing serializer against a verifying server. A mock that only checks shape cannot tell you whether the bytes match, so a reformatted body sailed through.

What actually broke

JSON objects are unordered semantically: a parser treats {a, b} and {b, a} as the same value. HMAC does not compare values, it hashes bytes. When the sender signs the body before sorting, then sorts before sending, the receiver rehashes the reordered bytes and gets a different digest. Same payload, same secret, valid structure — just a different byte order — fails verification. The sort happened between signing and transmission, which is exactly the window that has to stay byte-stable.

Keep the signing step and the wire bytes identical

• Sign the exact serialized bytes you will send. Do not sign an in-memory object and transmit a reformatted copy.
• If deterministic ordering matters for verification, define one canonical serialization (sorted keys, fixed separators, no added whitespace) and apply it before the digest, on both sender and receiver. Never reformat after signing.
• Keep pretty-printing for logs only. Write the normalized debug copy to the log, and transmit the signed raw bytes.

Tests that catch this

1. Sign a fixed payload, transmit it unchanged, and verify the digest recomputed from the received bytes equals the stored one.
2. Reject any transform — sorting, pretty-printing, whitespace normalization — that runs after the signing call in the call stack.
3. Replay one signed webhook against an environment that enforces the signature, not only a stub that parses the JSON.

Where the contract lives

When the signed shape and field order are stated in the integration spec, that spec should stay the source of truth locally rather than a hand-kept serialization note. This is the local-first workflow we built around in Powerduck: keep the contract on disk, validate the actual outgoing bytes against it, and replay one real request against an enforcing endpoint instead of trusting a permissive mock.

Before you call the signing path done:

1. Was the digest computed over the exact bytes transmitted?
2. Did any sort, pretty-print, or normalize run after signing?
3. Did a real verifying endpoint accept the signature, or only a shape-accepting stub?
4. Is the expected serialization order written in the contract, not assumed?

A normalized, pretty body reads better in review. On the wire between two parties that sign, byte stability is the product. Order is free inside a parser; it is expensive after a digest.