Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 5 min read

Use dedicated IAM roles for every data pipeline job

Most data engineers assume that a single "ETL-Role" shared across an entire orchestration platform is sufficient because the services are internal. This assumption is a common precursor to lateral movement if a single jo

Most data engineers assume that a single "ETL-Role" shared across an entire orchestration platform is sufficient because the services are internal. This assumption is a common precursor to lateral movement if a single job is compromised, as that job inherits the broad permissions of every other pipeline in your environment. When a job that only needs to read from a specific S3 prefix suddenly has the ability to delete tables in your data warehouse or read sensitive secrets from Parameter Store, your blast radius is effectively the entire platform.

This article shows how to break the "shared service role" pattern by implementing job-specific IAM roles. We will focus on defining discrete execution roles for individual tasks, ensuring that a failure in one pipeline component cannot be used to escalate privileges or access data outside of that component’s narrow scope. This scope covers AWS IAM policy design and resource-based policy attachments for common data orchestration patterns.

Why this matters: Shared roles violate the principle of least privilege by granting aggregate permissions to every job, which complicates auditing and increases the risk of data exfiltration. By assigning a dedicated role to each job, you ensure that even a compromised process remains sandboxed within its intended data domain.

What you'll build

You will create a granular IAM infrastructure that isolates a data ingestion job from a transformation job. Instead of a single powerful role, each job uses a unique identity with only the permissions required for its specific task.

flowchart TD
    JobA["Data Ingestion Job (IAM Role A)"] --> S3Raw["Amazon S3 (Raw Bucket)"]
    JobB["Transformation Job (IAM Role B)"] --> S3Raw
    JobB --> S3Clean["Amazon S3 (Clean Bucket)"]
    JobA -.-> S3Clean
    style JobA fill:#f9f,stroke:#333
    style JobB fill:#ccf,stroke:#333
    style S3Raw fill:#eee,stroke:#333
    style S3Clean fill:#eee,stroke:#333

Prerequisites

  • An active AWS account.
  • AWS CLI v2 installed and configured.
  • Basic familiarity with AWS IAM JSON policy syntax.
  • You should have two S3 buckets: data-raw-bucket and data-clean-bucket.

Step 1: Create the ingestion execution role

The ingestion job only needs s3:PutObject access to the raw bucket. We create a dedicated role for this purpose to prevent it from ever reading or writing to the clean bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::data-raw-bucket/*"
    }
  ]
}

Use the AWS CLI to create this role and attach the policy.

aws iam create-role --role-name IngestionJobRole --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam put-role-policy --role-name IngestionJobRole --policy-name S3WriteOnly --policy-document file://ingest-policy.json

Verify the role exists: aws iam get-role --role-name IngestionJobRole.

Step 2: Create the transformation execution role

The transformation job requires s3:GetObject on the raw bucket and s3:PutObject on the clean bucket. It must not have any permissions to modify existing raw data.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::data-raw-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::data-clean-bucket/*"
    }
  ]
}

Apply this policy to a distinct role:

aws iam create-role --role-name TransformationJobRole --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam put-role-policy --role-name TransformationJobRole --policy-name S3TransformPolicy --policy-document file://transform-policy.json

Verify the role: aws iam get-role --role-name TransformationJobRole.

Step 3: Enforce the boundary with bucket policies

Even with distinct roles, it is a best practice to verify bucket policies to deny access from unauthorized entities. This provides a secondary layer of protection should an IAM policy be accidentally modified.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::data-clean-bucket/*",
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::<account-id>:role/TransformationJobRole"
        }
      }
    }
  ]
}

Apply this policy via aws s3api put-bucket-policy. This ensures that even if an administrator mistakenly adds a new permission to a different role, the bucket itself will reject the request if the principal is not TransformationJobRole.

Design decisions

  • Role-per-job, not Role-per-service. By isolating at the job level, you gain granular CloudTrail logs that identify exactly which business process performed an action. You give up the simplicity of managing a single global role but gain a significant reduction in blast radius.
  • Inline policies, not Managed policies. For highly specific jobs, inline policies ensure the policy cannot be accidentally shared or modified by updates to a managed policy. Use managed policies only for common, broad permissions that are truly ubiquitous across your platform.
  • Deny-by-default in Resource Policies. Use bucket policies to explicitly deny access to everyone except your designated role. This provides a safety net against "permission creep" where IAM roles are modified over time without corresponding updates to the data layer.

Troubleshooting

  • Access Denied on S3: If your job fails with an Access Denied error, verify the PrincipalArn in your bucket policy matches the ARN of the role being assumed by your job. Use aws sts get-caller-identity inside your job environment to confirm the active role.
  • Conflicting Policies: If a role has both an inline policy and a managed policy, the effective permission is the union of both. Check all attached policies if a job seems to have "extra" access it shouldn't possess.
  • Implicit Deny: Remember that IAM evaluates all policies; if any policy contains a Deny, the request is rejected regardless of any Allow statements. If you see unexpected access errors, scan for explicit Deny statements in your Service Control Policies (SCPs) or bucket policies.

Clean up

To remove the resources created in this tutorial, execute:

aws iam delete-role-policy --role-name IngestionJobRole --policy-name S3WriteOnly
aws iam delete-role --role-name IngestionJobRole
aws iam delete-role-policy --role-name TransformationJobRole --policy-name S3TransformPolicy
aws iam delete-role --role-name TransformationJobRole

Conclusion

By mapping every data job to a unique IAM role, you transition from a "trusted platform" model to a "zero-trust" execution model where every pipeline component is individually permissioned. This reduces the risk of lateral movement and makes your infrastructure significantly easier to audit and secure.

Try it: Audit your current infrastructure by listing all roles used by your data pipelines and checking for overlap; consolidate any "general purpose" roles by splitting them into job-specific identities today.

References

About the examples: automated checks confirmed that the JSON blocks parse and that 12 AWS CLI commands match the AWS service API models (operation and parameter names). The examples have not been run end to end, so test them in a non-production environment before relying on them.

Cover photo by Harvey Power on Unsplash.

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.