Artifact Immutability as a Release Primitive: Content Addressing, Digest Pinning, and the Substitution Attack Surface

Most teams treat image tags as release identifiers. They are not. A tag is a mutable pointer sitting on top of a content-addressed blob store. The digest—sha256:<64-hex>—is the identifier. Conflating the two produces a class of failure that looks like a deployment problem but is actually an artifact provenance problem: the rollback you thought was safe pulled a different image than the one you certified, the canary that passed load testing ran a different binary than production, or an attacker overwrote a tag between your CI scan and your Kubernetes pull.

This article examines what artifact immutability actually means in mechanical terms, where the guarantees break down, and how to enforce them across the CI/CD boundary without slowing rollouts.

What a Digest Guarantees and What It Does Not

An OCI image digest is the SHA-256 hash of the image manifest—not the layers, not the config blob, the manifest. The manifest references layers by their own digests. So the digest transitively covers the entire content graph, which means:

• Two images with identical digests are identical.
• Any mutation to any layer or config produces a different manifest digest.
• A registry that serves a manifest under a given digest must serve that exact content or the client can detect tampering by recomputing the hash.

What the digest does not guarantee: it says nothing about what the image contains beyond content identity. A digest-pinned image can still carry a known CVE, a misconfigured binary, or a supply-chain-compromised base layer. Digest pinning is not a security policy; it is a consistency and reproducibility primitive on top of which you build policy.

Also worth stating plainly: client-side digest verification only works if the client checks. docker pull nginx:1.27 does not verify anything. docker pull nginx@sha256:abc123... instructs the registry to return a specific manifest, but the client must rehash the received manifest and compare. Container runtimes such as containerd do this by default for digest-referenced pulls; they do not do it for tag-referenced pulls where the registry resolves the tag server-side.

The Mutable-Tag Failure Mode in Detail

Consider a standard pipeline:

CI: build → tag as :v1.4.2 → push → scan → pass gate → deploy :v1.4.2

The scan and the deploy reference the same string, but if any of the following occur between push and deploy, the deployed image differs from the scanned one:

1. A developer force-pushes a hotfix to the same tag.
2. The base image rebuild triggers an automated retag in the registry.
3. A registry mirror sync races with the pull.

Scenario three is particularly subtle on Kubernetes. When a node's image is evicted and a pod reschedules to a new node, imagePullPolicy: IfNotPresent causes the new node to pull the tag fresh. If the tag has moved since original deployment, the new node runs a different image than the original nodes—silently.

The fix is mechanical: record the digest at push time and use only digests downstream.

// Pseudocode: resolve and record digest after push
type PushedArtifact struct {
    Repository string
    Tag        string
    Digest     string // sha256:...
    PushedAt   time.Time
}

func pushAndResolve(ctx context.Context, ref string) (PushedArtifact, error) {
    pusher := registry.NewClient(registryHost)
    if err := pusher.Push(ctx, ref); err != nil {
        return PushedArtifact{}, fmt.Errorf("push: %w", err)
    }
    desc, err := pusher.Resolve(ctx, ref)
    if err != nil {
        return PushedArtifact{}, fmt.Errorf("resolve post-push: %w", err)
    }
    return PushedArtifact{
        Repository: extractRepo(ref),
        Tag:        extractTag(ref),
        Digest:     desc.Digest.String(),
        PushedAt:   time.Now().UTC(),
    }, nil
}

The returned PushedArtifact.Digest becomes the canonical release identifier. Every downstream system—Helm values, Kubernetes manifests, rollback scripts—receives the digest, not the tag.

Enforcing Digests at the Kubernetes Admission Boundary

Recording digests in CI is necessary but insufficient if the deployment process can still reference tags. An admission webhook enforces the invariant at the cluster boundary.

The webhook inspects every Pod creation or update. If any container's image field resolves to a tag reference rather than a digest reference, the webhook rejects the pod.

func validateImageRef(image string) error {
    ref, err := name.ParseReference(image)
    if err != nil {
        return fmt.Errorf("unparseable image ref %q: %w", image, err)
    }
    if _, ok := ref.(name.Digest); !ok {
        return fmt.Errorf("image %q must be digest-pinned (got tag ref)", image)
    }
    return nil
}

This single check eliminates the entire class of tag-drift failures at admission time, before the kubelet ever contacts the registry. The tradeoff: your CD system must be capable of injecting digest references before submitting manifests. Kustomize image transformers and Helm post-render hooks both support this. If they do not fit, a mutating webhook that resolves tags to digests at admission is the fallback—but mutation is harder to audit than rejection.

Supply Chain: Cosign Signatures and the Verification Gate

Digest pinning proves content identity. It does not prove provenance—that the image was built by your CI system, with your Dockerfile, from your source commit. For that you need a signature.

Cosign's keyless signing mode, backed by Sigstore's Rekor transparency log and OIDC identity, attaches a signature to the image manifest without a separate key management burden. The signature is itself stored as an OCI artifact in the same registry, referenced by the image digest.

The verification gate belongs in two places: the CD pipeline before manifest generation, and the admission webhook before pod scheduling.

// Admission webhook: verify cosign signature before allowing digest-pinned pod
func verifyCosignSignature(ctx context.Context, imageRef string) error {
    verifier, err := cosign.NewCmdVerify(
        cosign.VerifyOpts{
            Rekor:     rekorURL,
            OIDCIssuer: "https://token.actions.githubusercontent.com",
        },
    )
    if err != nil {
        return err
    }
    _, err = verifier.Verify(ctx, imageRef)
    return err
}

The latency cost of a Rekor lookup at admission time is real—typically 150–400 ms depending on the transparency log endpoint and your network topology. Options to manage it: cache verification results keyed by digest (digests are immutable, so a verified digest stays verified), use an in-cluster Rekor mirror, or move signature verification entirely to CD and make the webhook trust a verified-digest allowlist updated by CD. The allowlist approach introduces a synchronization hazard: if the allowlist update fails, no pods can schedule. Make the webhook fail-open for allowlist fetch errors with a compensating audit log, or accept fail-closed for high-security workloads.

Artifact Promotion and the Multi-Registry Problem

Production registries are typically separate from CI build registries for security isolation. Artifact promotion—moving an image from a build registry to a staging registry to a production registry—must preserve digest identity, which requires content-addressed copying rather than re-pushing through a tag.

Tools like crane copy and skopeo copy copy by digest and preserve the manifest without rebuild. The digest of the artifact in the destination registry must be verified to match the source digest after copy. If it does not, the copy operation encountered a mid-flight corruption or the destination registry modified the manifest (some registries rewrite media types on ingest—this is a known incompatibility vector with older ECR behavior).

# Promote by digest, then verify
crane copy \
  build-registry.internal/myapp@sha256:abc123 \
  prod-registry.internal/myapp:v1.4.2

PROMOTED=$(crane digest prod-registry.internal/myapp:v1.4.2)
[ "$PROMOTED" = "sha256:abc123" ] || { echo "digest mismatch after promotion"; exit 1; }

This verification step is frequently omitted and represents a silent integrity gap in most promotion pipelines.

The SBOM Attachment Problem

Software Bill of Materials (SBOM) documents attached as OCI referrers are growing in regulatory relevance. The mechanical problem: an SBOM attached to an image digest is itself an OCI artifact, and its attachment is recorded in the referrers API response for that digest. If your promotion pipeline copies the image without copying its referrers, the SBOM does not follow the artifact into production.

crane copy --all and Oras copy with referrers support handle this, but not all registry implementations expose the referrers API consistently. Verify that your production registry supports the referrers API (GET /v2/<name>/referrers/<digest>) before depending on it in policy enforcement.

Decision Framework

Apply these checks when designing your artifact delivery system:

1. Tag vs. digest in manifests. If any Kubernetes manifest in your production path references an image by tag, you have a reproducibility and drift risk. Enforce digest-only at admission or in the CD manifest generation step.

2. Post-push digest resolution. CI must resolve and record the digest immediately after push, before any downstream step runs. Treat the digest as the artifact ID for the rest of the pipeline.

3. Promotion integrity check. After every cross-registry copy, verify the destination digest matches the source. Fail the pipeline on mismatch.

4. Signature verification placement. Verify signatures in CD before manifest generation if latency at admission is unacceptable. Verify at admission if you want an enforcement point that CD bypasses cannot circumvent. Both when the threat model demands it.

5. Referrer completeness. If SBOMs or attestations are part of your compliance posture, confirm your registry supports the referrers API and that your promotion tooling copies referrers. Otherwise, attestations exist only in the build registry and are invisible in production.

6. Webhook failure policy. Fail-closed admission webhooks block all pod creation when the webhook is unavailable. Scope webhook coverage to namespaces that require it, keep the webhook stateless and fast, and define a runbook for webhook outage that does not involve disabling it silently.

Digest pinning is not a hardening step you add after the delivery system is built. It is the foundation on which reproducible, auditable, and safe artifact delivery is constructed. Every other control—signature verification, SBOM policy, rollback safety—depends on the artifact identifier being stable. Get that primitive right first.