Your Green CI Is Lying to You: How I Built a Release-Evidence Gate

Disclosure: I'm the maintainer of CBP. This post is a technical write-up of a problem I kept running into, not a product announcement.

The Problem I Couldn't Unsee

CI passes. Tests are green. Coverage is above threshold. You publish the artifact.

Then someone asks: "Is the thing you just published actually the thing your CI built and tested?"

And you realize you don't have a verifiable answer. The CI log says one thing. The release page says another. The artifact that actually gets installed might be a third thing entirely.

This is the "execution object ≠ proven object" problem. It's invisible until it's a supply-chain incident.

Why Existing Tools Weren't Enough

Semgrep and SonarQube analyze code. Scorecard scores repository health. cosign signs artifacts. All valuable—but none of them answer the specific question I needed: "Does the artifact I'm about to publish match the artifact my CI proved?"

That's a reconciliation problem, not a scanning or signing problem. It needs a gate that checks evidence against reality, and fails closed when anything is unverifiable.

What I Built

CBP is a release-evidence gate. Its core mechanism is simple: the action never installs a wheel from the repository.

Instead, it:

1. Parses a version tag (vX.Y.Z) — if it can't, exit 2 (unverifiable).
2. Downloads the wheel from the GitHub Release.
3. Compares the local digest against the release API's digest — mismatch → exit 3.
4. Runs gh attestation verify — failure → exit 4.

Exit codes are classified by why verification failed, not just "something went wrong." This makes the gate debuggable and the failure modes explicit.

yaml
# Exit code classification example
# 2 = Unverifiable (no version / invalid version / missing gh / download failed / no digest)
# 3 = Digest mismatch (the downloaded artifact is not the released artifact)
# 4 = Attestation verification failed (untrusted source)