External Secrets Operator + CSI Driver: The Setup That Survived Our SOC 2 Audit

Our SOC 2 audit exposed a hard truth: secrets stored in Git (even encrypted) and a separate secrets manager don't reconcile automatically. Here's the ESO + CSI driver stack that took us from "we encrypt everything at rest" to "we have a documented, auditable secret rotation workflow."

The setup

AWS Secrets Manager
    ↓ (sync every 60s)
External Secrets Operator (k8s controller)
    ↓ (creates)
SecretStore / ExternalSecret (k8s CRD)
    ↓ (CSI driver projection)
Pod-mounted secret file (tmpfs)

Pods never reference k8s Secret objects. The CSI driver mounts a tmpfs volume at /var/run/secrets/ and syncs values on file write. If a secret rotates, the CSI driver rewrites the file and sends SIGHUP to the pod's process via refreshInterval.

The 4 components you need

1. SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-prod
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-sa

Use JWT auth via IRSA (IAM Roles for Service Accounts) instead of static credentials. The service account is bound to an IAM role with secretsmanager:GetSecretValue only.

2. ExternalSecret CRD

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payments-db-credentials
spec:
  refreshInterval: 60s
  secretStoreRef:
    name: aws-prod
    kind: SecretStore
  target:
    name: payments-db-cred
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

refreshInterval: 60s is the audit-safe default. For compliance-sensitive secrets, drop to 15s.

3. SecretProviderClass (CSI driver)

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payments-db
  namespace: payments
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/payments/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: "username"
            objectAlias: "DB_USER"
          - path: "password"
            objectAlias: "DB_PASS
   secretObjects:
    - data:
        - key: password
          objectName: DB_PASS
      secretName: payments-db-mounted
      type: Opaque

The CSI driver projects the AWS secret into the pod's filesystem at /var/run/secrets/.

4. Pod spec

volumes:
  - name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: payments-db
volumeMounts:
  - name: secrets
    mountPath: /var/run/secrets
    readOnly: true

The 3 pitfalls we hit

Pitfall 1: SyncInterval race conditions.

ESO refreshes every 60s. The CSI driver mounts synchronously on pod start. If a secret rotates between pod start and the first sync, the pod has stale creds. We added a lifecycle.preStop hook that calls /var/run/secrets/refresh before SIGTERM.

Pitfall 2: Pod restarts on rotation.

By default, the CSI driver uses the CSI driver rotation feature which writes new secrets without restarting the pod. But this only works if your app reads the secret file periodically, not just at startup. We added fileEventRefresh: true for our 12 apps that cache creds at startup. The other 8 read on every request — no restart required.

Pitfall 3: Audit trail.

AWS Secrets Manager logs GetSecretValue calls to CloudTrail, but the identity logged is the ESO service account, not the pod that consumed the secret. Our audit required "which pod consumed this secret at 14:32." We solved it by enabling CloudTrail data events on Secrets Manager (not just management events). Cost: $0.10 per 100K events. Worth it.

The compliance evidence we produce

For SOC 2 audit, we generate monthly:

1. AWS Config rule secretsmanager-rotation-enabled-check — 100% of secrets rotated within 90 days.
2. ESO reconciliation report via kubectl get externalsecret -A -o json | jq — every secret has a status.refreshTime within the last refreshInterval.
3. CSI driver Synchronization Report — kubectl get csi-driver-config -o yaml shows every pod has a mounted secret volume.

The metrics we care about

• Secret rotation compliance: target 100% (audit requirement).
• Mean time to secret propagation: ESO 60s + CSI sync 5s = 65s. Acceptable.
• Pods with stale creds incident rate: 0 in 6 months.

On adjacent tooling

For Windows build agents that need to authenticate to AWS services from containers, ScsDriver WebDAV mount tool for Windows can mount the same S3 buckets that ESO syncs to k8s — useful when a Windows CI runner needs to download a fresh secret bundle during build. The credential rotation flow works because the Windows agent re-mounts the share on every CI job.

What does your secret rotation stack look like? Vault + ESO, AWS Secrets Manager + ESO, or something else?