Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 6 min read

Turning On ECR Tag Immutability Breaks Pipelines That Re-Push the latest Tag

Originally published on KBY Technologies. If you push images to ECR from CI and a security review has asked you to make the repository immutable, this is the part we would check in your pipeline first: whether anything

Originally published on KBY Technologies.

If you push images to ECR from CI and a security review has asked you to make the repository immutable, this is the part we would check in your pipeline first: whether anything pushes the same tag twice. We walk through the exclusion that keeps release tags locked while the build goes green, and the read-only checks to run before you change anything.

Switching an Amazon ECR repository to IMMUTABLE stops every pipeline that pushes the same tag twice, and a pipeline that pushes latest on every build does exactly that. From the moment the setting changes, ECR answers a push of an existing tag with ImageTagAlreadyExistsException: β€œThe specified image is tagged with a tag that already exists. The repository is configured for tag immutability.” The fix is not to turn immutability off. Set the repository to IMMUTABLE_WITH_EXCLUSION with a single WILDCARD filter for latest, so release tags stay protected and the build goes green, then move the pipeline to a unique tag per build and drop the exclusion once nothing depends on latest.

Symptom

The image push step fails with ImageTagAlreadyExistsException on the first build after the repository setting changed. The build and test stages pass, the image exists locally, and nothing in the pipeline definition changed. The PutImage API documents the error as HTTP 400 with the message quoted above. If the pipeline pushes a unique tag first and latest second, the unique tag lands and only the second push fails, which makes it look like an intermittent registry fault rather than a setting.

Repositories that serve a pull through cache rule are a related case. AWS recommends Mutable for those so that ECR can update cached images, so a blanket β€œmake every repository immutable” change can stop cached images refreshing as well as breaking builds.

Why it looks safe to switch on

Tag immutability is usually turned on as a hardening task, often after a scanner or audit flags mutable repositories. The mental model is β€œrelease tags can no longer be overwritten”, and that part is true. What is easy to miss is that plain IMMUTABLE makes no distinction between a release tag such as v2.3.1 and a moving tag such as latest, main or dev. The ECR user guide says it directly: tag immutability affects all tags. A pipeline written to push latest after every merge has always relied on overwriting it.

Root cause

The repository’s imageTagMutability setting decides whether any existing tag can be pointed at a new image. With MUTABLE, image tags can be overwritten. With IMMUTABLE, every tag in the repository is immutable, and pushing an image with a tag that is already in the repository is rejected. The first push of latest after the change is refused because latest already exists from the previous build.

Since ECR added exclusion filters there are four values, not two: MUTABLE, IMMUTABLE, IMMUTABLE_WITH_EXCLUSION and MUTABLE_WITH_EXCLUSION. IMMUTABLE_WITH_EXCLUSION keeps the repository immutable but lets the tags that match a filter be updated; MUTABLE_WITH_EXCLUSION does the reverse and locks only the matching tags. Each filter has filterType WILDCARD and a filter of 1 to 128 characters drawn from letters, digits, ., _, * and -, and a repository takes between one and five filters.

Impact

Every pipeline that re-pushes a moving tag to the repository stops delivering images until the setting or the pipeline changes. Deployments that pull latest keep running the last image that was successfully pushed, so the service looks healthy while new code never arrives.

Under pressure the quick fix is put-image-tag-mutability --image-tag-mutability MUTABLE. It unblocks the build in seconds and removes the protection for every tag, including the release tags the change was meant to protect: any principal allowed to push can again point v2.3.1 at a different image. A filter of * under IMMUTABLE_WITH_EXCLUSION would match every tag and amount to the same thing. Nothing in a green build log shows that the protection has gone. If you need to find who changed the setting in the first place, the CloudTrail steps in finding who changed a security group with CloudTrail Event history work the same way for ECR, because CloudTrail records every ECR API call.

How to confirm it without changing anything

Two read-only calls settle the diagnosis. First, read the repository setting and any filters:

aws ecr describe-repositories --repository-names <repo> \
  --query "repositories[].[repositoryName,imageTagMutability,imageTagMutabilityExclusionFilters]"

An answer of IMMUTABLE, or IMMUTABLE_WITH_EXCLUSION with no filter that matches the tag the pipeline pushes, confirms the cause. Second, check that the tag already exists and which image holds it:

aws ecr describe-images --repository-name <repo> --image-ids imageTag=latest

The output shows the imageDigest, the imageTags on that image and imagePushedAt. A push time that matches the last green build, before the setting changed, is the image the failing push is trying to replace. To see when the setting changed and by whom, look up the event in the repository’s Region; Event history covers the last 90 days:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutImageTagMutability

Correction

Unblock the pipeline with a one-tag exclusion first, then remove the pipeline’s dependence on the moving tag. Run every step in the repository’s own Region with a role that can call ecr:PutImageTagMutability, and try it on a non-production repository first.

  1. Record the current state. Save the setting so rollback has an exact target:

    aws ecr describe-repositories --repository-names <repo> > ecr-mutability-before.json
    
  2. Exclude only the moving tag. Precondition: the backup file exists and shows IMMUTABLE or IMMUTABLE_WITH_EXCLUSION. The call carries the repository’s whole filter list, so include any existing filters you want to keep.

    aws ecr put-image-tag-mutability --repository-name <repo> \
      --image-tag-mutability IMMUTABLE_WITH_EXCLUSION \
      --image-tag-mutability-exclusion-filters filterType=WILDCARD,filter=latest
    

    The response echoes imageTagMutability and the filter list. Re-run the failed build: the latest push should now succeed, and every other tag stays immutable.

  3. Give every build a unique tag. Change the pipeline to push a tag that is never reused, such as the commit SHA or build number, alongside latest. Point deployments at that tag, or at the image digest, rather than at latest.

  4. Drop the exclusion when nothing reads latest. Precondition: no task definition, manifest or script still pulls latest and the pipeline no longer pushes it. Then set plain immutability:

    aws ecr put-image-tag-mutability --repository-name <repo> --image-tag-mutability IMMUTABLE
    

Keep the exclusion as narrow as the tag name allows. A filter such as dev-* is fine for a family of moving tags; a filter that also matches release names defeats the point.

Verification

The fix holds when a moving-tag push succeeds and a release-tag overwrite is still refused. Run describe-repositories again and confirm imageTagMutability and the filter list read exactly as intended. Run describe-images --image-ids imageTag=latest and confirm imagePushedAt now matches the new build. In a non-production repository, push a second image under an existing release tag: it should still fail with ImageTagAlreadyExistsException. If it succeeds, a filter is wider than you meant.

Rollback

Rollback is one call that restores the value saved in ecr-mutability-before.json. If the repository was IMMUTABLE before step 2, run put-image-tag-mutability --image-tag-mutability IMMUTABLE with no filters; if it already had filters, send the same value and the same filter list. Restoring IMMUTABLE brings the failing latest push back until the pipeline stops pushing that tag. AWS documents the setting’s effect on pushes, not on images already stored. Stop and escalate if the backup shows MUTABLE: that repository was never immutable and the failure has another cause.

Stop it happening on the next repository

Before switching a repository to immutable, search the pipelines that push to it for tags that are reused, and set the exclusion in the same change. New repositories can be created with the right setting from the start using create-repository --image-tag-mutability IMMUTABLE_WITH_EXCLUSION --image-tag-mutability-exclusion-filters filterType=WILDCARD,filter=latest. Leave pull through cache repositories mutable, as AWS recommends. Immutability fixes what a tag points to, not what is inside the image, so pair it with scanning; scan on push leaves images pushed before it was enabled unscanned, and cross-account pull policies need source conditions to keep the right accounts reading those images.

Read the original on KBY Technologies

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.