My Release Pipeline Refused to Ship My Release. It Was Right.

I pushed the v0.2.0 tag. The release workflow built the wheel, generated the SBOM, wrote the checksums, signed everything with keyless Sigstore — and then failed at its own last step:

X Verify the release before publishing
ModuleNotFoundError: No module named 'cryptography'

The workflow never published. GitHub release: absent. That is the system working exactly as designed, and this is the story of the bug it caught — one my test suite structurally could not have caught, in a place I'd stopped thinking about years ago.

This is from the agentwatch v0.2.1 release — the agent-security tool in agentsec-ecosystem. Repo: agentsec-ecosystem/agentwatch.

The design that saved the release

The release workflow has a step before publishing: install the package and run agentwatch verify-release — re-verify the built artifacts from inside a clean environment. The workflow file documents its own contract in one line:

The workflow refuses to publish an artifact that agentwatch verify-release cannot confirm.

It's not a smoke test. It's a publication precondition. If the CLI can't run, nothing ships. So when the CLI couldn't run, the only thing that shipped that day was... nothing.

That single design decision converted what would have been "v0.2.0 is live, oops" into a failed run and a patch number.

The traceback worth reading slowly

The failure wasn't in verify logic. It was in import:

File ".../bin/agentwatch", line 3, in <module>
    from agentwatch.cli import main
  File ".../agentwatch/cli/main.py", line 71, in <module>
    from agentwatch.demo import purge_demo, render_demo, run_demo
  File ".../agentwatch/demo.py", line 22, in <module>
    from agentwatch.adapters import claude_code
  File ".../agentwatch/adapters/__init__.py", line 5, in <module>
    from agentwatch.adapters import (...
  File ".../agentwatch/adapters/a2a_proxy.py", line 36, in <module>
    from agentwatch import agent_card
  File ".../agentwatch/agent_card.py", line 26, in <module>
    from cryptography.exceptions import InvalidSignature
ModuleNotFoundError: No module named 'cryptography'

The release environment runs pip install -e packages/python-sdk — no extras. And cryptography in this project is an optional extra ([signing]): a hard dependency would be wrong, since most installs never sign anything.

But agent_card.py — the A2A agent-card signature verifier — imported InvalidSignature at module top. And agent_card is imported by the adapters package. Which is imported by demo. Which is imported by the CLI. So:

A plain pip install agentsec-agentwatch produced a CLI that died on --help. No signing involved. No A2A involved. Just an import chain doing what import chains do — eagerly dragging everything into memory at startup.

Every environment where this ran had cryptography present — dev setups install extras, CI quality jobs install extras. Only the release job installs bare. The one environment guaranteed to represent a real user — a clean, no-extras install — was the one that caught it.

What "optional" has to mean

The fix is small and I now consider it a rule:

def _verify_signature(public: Any, alg: str, signing_input: bytes, signature: bytes) -> bool:
    from cryptography.exceptions import InvalidSignature          # lazy
    from cryptography.hazmat.primitives import hashes
    from cryptography.hazmat.primitives.asymmetric import ec, ed25519, padding, rsa
    ...

The imports moved inside the one function that actually uses them — the verification path, which is also the path that requires the extra to be present anyway. Everything above it — module load, CLI startup, adapter imports, the entire non-signing surface — no longer cares whether cryptography exists.

Then the proof, which is the part I'd never done before for an import: a test that blocks the module, then imports the CLI:

class Blocker:
    def find_spec(self, name, path=None, target=None):
        if name == "cryptography" or name.startswith("cryptography."):
            raise ModuleNotFoundError("cryptography blocked for test")
        return None

sys.meta_path.insert(0, Blocker())
import agentwatch.cli.main  # must succeed

A meta-path hook that makes cryptography unimportable, then imports the CLI against it. If anyone ever moves that import back to module top, this test goes red in CI instead of at the next release. The A2A card tests (26 of them) still pass with cryptography present, so the feature didn't move an inch.

tip: An extra is only optional if the package works without it — and "works" is checked where it's truest: a clean no-extras install importing the entry point.

Why my test suite couldn't catch this

This is the part I sat with afterward. The suite that guards packaging is solid — it enforces one version across surfaces, validates the SBOM, checks the wheel contents. But every test environment installs extras. A missing-optional-dependency bug is invisible to any environment that has the dependency — which is all of them, except the one a user gets.

The only two places this class of bug can surface:

1. A clean install in the release pipeline — which is what verify-release is, and why it's a precondition and not a post-publish nicety.
2. A fresh-venv install-from-the-registry check after publish — which is the human-run check that, one release later, caught the version-banner bug the pipeline couldn't catch (the CLI imported fine; it just printed the wrong number).

Between those two checks, this class of bug has nowhere left to live. That's the actual architecture I'm taking out of this: gates at the boundaries where the environment stops lying to you.

The ending that could have been

Play it back without the verify step: tag pushes, workflow builds, signs, publishes. v0.2.0 goes to PyPI with a CLI that cannot start. Every clean install — the majority case — gets a broken tool with a signed, SBOM'd, provenance-attested wheel. All the supply-chain rigor intact, delivering a package that dies on import.

Instead: a failed run, a 2-line fix (PR #495), a re-tag, and a release with 8 assets — wheel, sdist, SBOM, checksums, each with its Sigstore bundle — that all actually work. v0.2.1 shipped the same way, first try, because the class of bug was already dead.

The question worth asking of your release pipeline: what's the last thing that runs before publish, and would it catch a broken import? If the answer is "nothing, we trust the build" — your next release is already scheduled, you just don't know what it's about.

References

• Release: agentwatch v0.2.1 on GitHub · on PyPI
• The fix: PR #495 — lazy cryptography import · the release workflow ("refuses to publish an artifact that verify-release cannot confirm")
• The no-extras import proof: a sys.meta_path hook that blocks cryptography and then imports the CLI — run at release time; the pattern is worth committing as a test
• v0.2.0 field test report
• Suite: agentsec-ecosystem — the full set of agent-security tools