Managing Encrypted Secrets in Kubernetes with Flux CD and Mozilla SOPS (GPG)

Managing Encrypted Secrets in Kubernetes with Flux CD and Mozilla SOPS (GPG)

One of the biggest challenges in GitOps is secret management: How do we store everything in Git without leaking sensitive credentials like API keys, database passwords, and TLS certificates?

In this practical tutorial, we explore how to manage encrypted Kubernetes secrets using Mozilla SOPS (Secrets OPerationS) with GPG/PGP and Flux CD. Secrets are encrypted on the developer machine, safely committed to Git, and automatically decrypted in-memory by Flux inside your Kubernetes cluster.

📁 GitHub Repository: The complete working sample code for this tutorial is available at github.com/ZakariaZermani/gitops-lab-flux.

Table of Contents

1. Why SOPS vs. External Secret Stores (OpenBao / Vault)?
2. Architecture Overview
3. Prerequisites
4. Step-by-Step Tutorial
   • Step 1: GPG Key Setup
   • Step 2: Repository SOPS Configuration (.sops.yaml)
   • Step 3: In-Cluster Decryption Setup for Flux
   • Step 4: Creating and Encrypting Secrets
   • Step 5: Testing & Verifying Reconciliation
5. Best Practices & Common Pitfalls
   • Use stringData instead of data
   • Avoid Double Base64 Encoding
   • The Flux Self-Management Bootstrap Trap
6. Conclusion

Why SOPS vs. External Secret Stores (OpenBao / Vault)?

When managing secrets in GitOps, there are two primary architectures:

1. In-Git Encryption (Mozilla SOPS): Encrypted secret files live directly alongside your manifests in Git and are decrypted in-memory by Flux during reconciliation.
2. External Secret Stores (OpenBao, HashiCorp Vault, AWS Secrets Manager) + Operator: Secrets live in an external server, and an operator (like External Secrets Operator - ESO) fetches them into the cluster.

Why choose SOPS?

• Zero Server Infrastructure: No need to deploy, secure, patch, or maintain a high-availability Vault/OpenBao cluster.
• Pure GitOps (Single Source of Truth): Git commits contain both the application code and exact matching encrypted secret versions. Rolling back a Git commit automatically rolls back secret configurations.
• Native Flux CD Support: Flux supports SOPS out-of-the-box (PGP/GPG, Age, AWS KMS, Azure Key Vault, GCP KMS) without requiring third-party operators.
• Ideal for Edge / Dev / Lean Environments: Lightweight, offline-capable, and simple to set up for small-to-medium infrastructures.

(Use OpenBao/Vault + ESO instead when you require dynamic secrets with TTL, cross-cluster centralized rotation, or strict compliance audit trails for every secret read access.)

Architecture Overview

 Developer Machine                         Git Repository (GitHub)                  Kubernetes Cluster (Flux CD)
+-----------------------+                 +-------------------------+             +----------------------------------+
| 1. Plaintext Secret   |                 | Encrypted Secret        |             | kustomize-controller             |
|    (stringData)       |                 | (test-app/              |             |   * reads in-cluster sops-gpg    |
|          │            |                 |  basic-auth.yaml)       |             |   * decrypts Secret in-memory    |
|   sops --encrypt      | ── git push ──> |                         | ── sync ──> |   * creates Secret in cluster    |
|          ▼            |                 |                         |             |          │                       |
| Encrypted Secret      |                 |                         |             |          ▼                       |
| (data is encrypted)   |                 |                         |             | Pod mounts / consumes Secret     |
+-----------------------+                 +-------------------------+             +----------------------------------+

Prerequisites

Before starting, ensure you have the following CLI tools installed:

• kubectl connected to your Kubernetes cluster
• flux CLI (v2+)
• sops CLI (v3.8+)
• gpg / GnuPG

Step-by-Step Tutorial

Step 1: GPG Key Setup

1. Generate a new GPG key (if you don't already have one):

   export KEY_NAME="flux-secrets"
   export KEY_COMMENT="Flux SOPS Key"

   gpg --batch --full-generate-key <<EOF
   %no-protection
   Key-Type: RSA
   Key-Length: 4096
   Subkey-Type: RSA
   Subkey-Length: 4096
   Name-Real: ${KEY_NAME}
   Name-Comment: ${KEY_COMMENT}
   Expire-Date: 0
   EOF

1. Retrieve the Key Fingerprint:

   export KEY_FP=$(gpg --list-secret-keys --with-colons "${KEY_NAME}" | grep '^fpr:' | head -n 1 | cut -d: -f10)
   echo "GPG Key Fingerprint: ${KEY_FP}"

1. Export the Public Key to the repository:

The public key can safely be committed to Git so team members can encrypt secrets:

   gpg --armor --export "${KEY_FP}" > flux-system/.sops.pub.asc

Step 2: Repository SOPS Configuration (.sops.yaml)

Define a .sops.yaml configuration at the repository root (or in flux-system/):

creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    pgp: CDDAD9230C22E207B2AB2BD183E0D8C9439B21A2

Note: Replace the pgp: value with your actual GPG key fingerprint (${KEY_FP}).
encrypted_regex: ^(data|stringData)$ ensures only secret values are encrypted, keeping metadata (names, namespaces, labels) readable.

Step 3: In-Cluster Decryption Setup for Flux

Flux's kustomize-controller needs the private GPG key to decrypt secrets inside the cluster.

1. Export the private key into Kubernetes as a Secret in namespace flux-system:

   gpg --export-secret-keys --armor "${KEY_FP}" | \
     kubectl create secret generic sops-gpg \
     --namespace=flux-system \
     --from-file=sops.asc=/dev/stdin

1. Configure Decryption on your Flux Kustomization:

In flux-system/gotk-sync.yaml (or your application's Flux Kustomization manifest), add spec.decryption:

   apiVersion: kustomize.toolkit.fluxcd.io/v1
   kind: Kustomization
   metadata:
     name: flux-system
     namespace: flux-system
   spec:
     decryption:
       provider: sops
       secretRef:
         name: sops-gpg
     interval: 10m0s
     path: ./
     prune: true
     sourceRef:
       kind: GitRepository
       name: flux-system

1. Apply the configuration to the cluster:

   kubectl apply -f flux-system/gotk-sync.yaml

Step 4: Creating and Encrypting Secrets

Always use stringData: for plaintext secrets when working with SOPS.

1. Create an unencrypted secret manifest (test-app/basic-auth.yaml):

   apiVersion: v1
   kind: Secret
   metadata:
     name: basic-auth
     namespace: default
   stringData:
     user: admin
     password: supersecretpassword123
     name: test-app-user

1. Encrypt the file with SOPS:

   sops --encrypt --in-place test-app/basic-auth.yaml

(SOPS automatically uses the rule in .sops.yaml to encrypt all fields matching stringData)

1. Verify the encrypted file:

The file should now contain ENC[AES256_GCM,...] under stringData: and a sops: metadata section at the bottom.

1. Commit and push to Git:

   git add test-app/basic-auth.yaml
   git commit -m "feat: add encrypted basic-auth secret"
   git push

Step 5: Testing & Verifying Reconciliation

1. Trigger Flux reconciliation:

   flux reconcile kustomization flux-system --with-source

1. Check the Kustomization status:

   flux get kustomizations

Output should show READY: True.

1. Verify the decrypted Secret in Kubernetes:

   kubectl get secret basic-auth -n default -o yaml

1. Verify that a Pod consumes the decrypted secret:

In test-app/pod.yaml, a test pod mounts the secret and injects environment variables:

   kubectl get pod basic-auth-app -n default

• Check environment variables:
  

   kubectl exec -it basic-auth-app -n default -- env | grep BASIC_AUTH
  

• Check mounted secret files (in plaintext):
  

   kubectl exec -it basic-auth-app -n default -- sh -c 'for f in /etc/secrets/*; do echo -n "$f: "; cat "$f"; echo ""; done'
  

Best Practices & Common Pitfalls

1. Use stringData instead of data

• data: requires base64-encoded values. If you write plaintext under data: and encrypt it with SOPS, Kubernetes will fail dry-run validation with illegal base64 data at input byte ....
• stringData: accepts human-readable plaintext strings and Kubernetes automatically base64-encodes them upon storage.

2. Avoid Double Base64 Encoding

• Never pre-encode values with base64 before putting them into stringData:.
• If you base64-encode a value manually and put it into stringData:, Kubernetes will base64-encode it a second time. When mounted into the container, the file will still contain base64 instead of the actual plaintext secret.

3. The Flux Self-Management Bootstrap Trap

• When running flux bootstrap, Flux regenerates flux-system/gotk-sync.yaml and removes custom modifications like spec.decryption.
• If this happens:
  1. Re-add spec.decryption to flux-system/gotk-sync.yaml.
  2. Apply it directly with kubectl apply -f flux-system/gotk-sync.yaml so the controller has the decryption configuration before it attempts to process encrypted secrets.
  3. Recreate sops-gpg secret if the flux-system namespace was recreated.

Conclusion

By combining Mozilla SOPS and Flux CD, you achieve true GitOps: your entire state—including secrets—lives declaratively in Git with full audit history and seamless rollback capabilities, without exposing sensitive plaintext data.

Have questions, feedback, or a different GitOps secret setup? Feel free to drop a comment below or check out the sops-tuto branch on the GitHub repository!