Secrets Management

This homelab uses Sealed Secrets for GitOps-compatible secrets management, allowing encrypted secrets to be safely stored in Git and automatically decrypted by the cluster.

Overview

Solution: Bitnami Sealed Secrets
Namespace: kube-system (controller)
Deployment: Managed by ArgoCD
Sync Wave: -25 (deploys early, before applications needing secrets)

Why Sealed Secrets?

The Problem with Kubernetes Secrets

Standard Kubernetes Secrets are base64-encoded (not encrypted) and cannot be safely committed to Git repositories. This creates challenges for GitOps workflows where all configuration should be version-controlled.

Previous Approach: git-crypt

Previously, secrets were encrypted with git-crypt in the repository. However:

ArgoCD cannot decrypt git-crypt files
Required manual kubectl apply before ArgoCD sync
Not truly GitOps-compliant

Sealed Secrets Solution

Sealed Secrets solves this by:

Encrypting secrets client-side using the cluster's public key
Storing encrypted SealedSecret CRDs in Git (safe to commit)
Decrypting at runtime via the Sealed Secrets controller
Full GitOps compatibility - ArgoCD can sync SealedSecrets directly

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Sealed Secrets Flow                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Developer Workstation              Kubernetes Cluster          │
│  ┌──────────────────┐              ┌──────────────────────┐    │
│  │                  │   kubeseal   │  Sealed Secrets      │    │
│  │  Plain Secret    │ ──────────►  │  Controller          │    │
│  │  (local only)    │              │  (kube-system)       │    │
│  └──────────────────┘              └──────────┬───────────┘    │
│           │                                   │                 │
│           ▼                                   ▼                 │
│  ┌──────────────────┐              ┌──────────────────────┐    │
│  │  SealedSecret    │   Git Push   │  Decrypted Secret    │    │
│  │  (encrypted)     │ ──────────►  │  (runtime only)      │    │
│  │  Stored in Git   │   ArgoCD     │  Used by Pods        │    │
│  └──────────────────┘              └──────────────────────┘    │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Managed Secrets

Secret Name	Namespace	Application	Purpose
`unipoller-secret`	unipoller	UniFi Poller	UniFi controller API key
`cloudflare-api-token`	external-dns	External DNS	Cloudflare DNS API token
`unifi-credentials`	external-dns	External DNS	UniFi webhook credentials
`alertmanager-smtp-credentials`	default	Alertmanager	Email notification SMTP credentials
`snmp-exporter-credentials`	default	SNMP Exporter	Synology NAS SNMPv3 credentials
`cloudflare-api-token-secret`	cert-manager	cert-manager	DNS01 challenge API token
`client-info-secret`	synology-csi	Synology CSI	NAS iSCSI authentication
`velero-b2-credentials`	velero	Velero	Backblaze B2 backup storage credentials

Secrets NOT Managed by Sealed Secrets

Secret	Reason	How to Apply
`my-ssh-repo-secret-homelab`	Bootstrap secret - ArgoCD needs this to read the repo	`kubectl apply -f secrets/argocd-git-access.yaml`
`kube-prometheus-stack-grafana`	Auto-generated by Helm chart	Managed automatically

Using kubeseal

Prerequisites

Install the kubeseal CLI:

# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar -xvzf kubeseal-0.24.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Creating a New SealedSecret

Step 1: Create a standard Kubernetes Secret YAML:

# my-secret.yaml (DO NOT commit this file)
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: my-namespace
type: Opaque
stringData:
  username: myuser
  password: mysecretpassword

Step 2: Seal the secret using kubeseal:

kubeseal --controller-name=sealed-secrets-controller \
         --controller-namespace=kube-system \
         --format yaml \
         < my-secret.yaml \
         > my-sealed.yaml

Step 3: Commit the SealedSecret:

# The sealed secret is safe to commit
git add my-sealed.yaml
git commit -m "feat: Add my-secret as SealedSecret"

Step 4: Delete the unencrypted secret:

rm my-secret.yaml

Secret Rotation Procedures

Rotating a Secret Value

Use this procedure when a credential needs to be updated (e.g., password change, API token refresh, compromised secret).

Step 1: Create the New Plain Secret

Create a temporary file with the new secret value. Never commit this file to Git.

# Example: Rotating cloudflare-api-token for external-dns
cat > /tmp/new-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: external-dns
type: Opaque
stringData:
  cloudflare_api_token: "NEW_TOKEN_VALUE_HERE"
EOF

Step 2: Seal the New Secret

kubeseal --controller-name=sealed-secrets-controller \
         --controller-namespace=kube-system \
         --format yaml \
         < /tmp/new-secret.yaml \
         > manifests/base/external-dns/cloudflare-sealed.yaml

Step 3: Clean Up Plain Secret

rm /tmp/new-secret.yaml

Step 4: Commit and Deploy

# Create feature branch
git checkout -b rotate/external-dns-cloudflare-token

# Commit the updated sealed secret
git add manifests/base/external-dns/cloudflare-sealed.yaml
git commit -m "chore: Rotate cloudflare-api-token for external-dns"

# Push and create PR
git push -u origin rotate/external-dns-cloudflare-token
gh pr create --title "chore: Rotate cloudflare-api-token" \
             --body "Rotates the Cloudflare API token for external-dns."

Step 5: Verify Deployment

After the PR is merged and ArgoCD syncs:

# Verify the secret was updated (check annotation timestamp)
kubectl get secret cloudflare-api-token -n external-dns -o jsonpath='{.metadata.annotations}'

# Verify the application is working
kubectl logs -n external-dns deployment/external-dns-cloudflare | tail -20

Step 6: Restart Affected Pods (if needed)

Some applications cache secrets and need a restart to pick up new values:

kubectl rollout restart deployment/external-dns-cloudflare -n external-dns
kubectl rollout status deployment/external-dns-cloudflare -n external-dns

Batch Secret Rotation

For rotating multiple secrets at once (e.g., after a security incident):

#!/bin/bash
# batch-rotate-secrets.sh

SECRETS=(
  "unipoller:unipoller-secret:manifests/base/unipoller/unipoller-sealed.yaml"
  "external-dns:cloudflare-api-token:manifests/base/external-dns/cloudflare-sealed.yaml"
  "external-dns:unifi-credentials:manifests/base/external-dns/unifi-sealed.yaml"
)

for entry in "${SECRETS[@]}"; do
  IFS=':' read -r namespace name path <<< "$entry"
  echo "Processing: $name in $namespace"
  echo "  - Update /tmp/${name}.yaml with new values"
  echo "  - Then run: kubeseal --controller-name=sealed-secrets-controller --controller-namespace=kube-system --format yaml < /tmp/${name}.yaml > $path"
done

Sealing Key Management

The sealing key is an RSA key pair stored as a Kubernetes Secret in kube-system. The controller uses the private key to decrypt SealedSecrets.

Backup Sealing Key

Critical

Back up the sealing key immediately after cluster creation. Without this backup, all SealedSecrets become unrecoverable if the cluster is rebuilt.

# Export all sealing keys
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > sealed-secrets-key-backup.yaml

# Verify the backup contains the key
grep -c "tls.crt" sealed-secrets-key-backup.yaml
grep -c "tls.key" sealed-secrets-key-backup.yaml

Store securely:

Password manager (1Password, Bitwarden)
Encrypted USB drive
Cloud storage with client-side encryption

Rotate Sealing Key

Key rotation creates a new sealing key while keeping old keys for decryption.

Manual Key Rotation (Recommended)

# 1. Backup current key first
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > sealed-secrets-key-backup-$(date +%Y%m%d).yaml

# 2. Delete the controller pod to trigger new key generation
kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

# 3. Wait for controller to restart
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=sealed-secrets \
  -n kube-system \
  --timeout=60s

# 4. Verify new key was created
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  --show-labels

Re-seal All Secrets After Key Rotation

After rotating the sealing key, re-seal all secrets to use the new key:

#!/bin/bash
# reseal-all-secrets.sh

SEALED_SECRETS=(
  "manifests/base/unipoller/unipoller-sealed.yaml:unipoller-secret:unipoller"
  "manifests/base/external-dns/cloudflare-sealed.yaml:cloudflare-api-token:external-dns"
  "manifests/base/external-dns/unifi-sealed.yaml:unifi-credentials:external-dns"
  "manifests/base/kube-prometheus-stack/alertmanager-smtp-sealed.yaml:alertmanager-smtp-credentials:default"
  "manifests/base/kube-prometheus-stack/snmp-exporter-sealed.yaml:snmp-exporter-credentials:default"
  "manifests/base/cert-manager/cloudflare-sealed.yaml:cloudflare-api-token-secret:cert-manager"
  "manifests/base/synology-csi/client-info-sealed.yaml:client-info-secret:synology-csi"
  "manifests/base/velero/b2-credentials-sealed.yaml:velero-b2-credentials:velero"
)

for entry in "${SEALED_SECRETS[@]}"; do
  IFS=':' read -r path name namespace <<< "$entry"
  echo "Re-sealing: $name ($namespace)"

  # Get current decrypted secret from cluster and re-seal
  kubectl get secret "$name" -n "$namespace" -o yaml | \
    kubectl neat | \
    kubeseal --controller-name=sealed-secrets-controller \
             --controller-namespace=kube-system \
             --format yaml > "$path"

  echo "  Updated: $path"
done

echo "Done! Review changes with: git diff"

note

This script requires kubectl-neat plugin: kubectl krew install neat

Restore Sealing Key

Use this procedure when rebuilding the cluster:

# 1. Install Sealed Secrets controller first (ArgoCD will do this)

# 2. Delete the auto-generated key
kubectl delete secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key

# 3. Restore from backup
kubectl apply -f sealed-secrets-key-backup.yaml

# 4. Restart the controller to load the restored key
kubectl delete pod -n kube-system -l app.kubernetes.io/name=sealed-secrets

# 5. Verify restoration
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets | grep -i "sealed secrets"

Disaster Recovery

Scenario 1: Cluster Rebuilt, Backup Available

Deploy new cluster with ArgoCD
Restore sealing key backup (see above)
ArgoCD will sync all SealedSecrets

Verify secrets are decrypted:

kubectl get secrets -A | grep -v default-token

Scenario 2: Cluster Rebuilt, No Backup

warning

If the sealing key backup is lost, all SealedSecrets are unrecoverable. You must regenerate all credentials.

Secret	Service	Rotation Procedure
cloudflare-api-token	External DNS	Generate new API token in Cloudflare dashboard
cloudflare-api-token-secret	Cert Manager	Use same token as External DNS or create dedicated one
unifi-credentials	External DNS	Create new local user in UniFi controller
unipoller-secret	UniPoller	Use existing UniFi read-only user or create new one
alertmanager-smtp-credentials	AlertManager	Generate app password in email provider
snmp-exporter-credentials	SNMP Exporter	SNMP community string from Synology NAS
client-info-secret	Synology CSI	Client credentials from Synology DSM
velero-b2-credentials	Velero	Create new B2 application key in Backblaze dashboard

Scenario 3: Single Secret Corrupted

If a single SealedSecret is corrupted in Git:

# Get the current decrypted secret from cluster
kubectl get secret <name> -n <namespace> -o yaml | kubectl neat > /tmp/current-secret.yaml

# Re-seal
kubeseal --controller-name=sealed-secrets-controller \
         --controller-namespace=kube-system \
         --format yaml \
         < /tmp/current-secret.yaml \
         > path/to/sealed-secret.yaml

# Clean up and commit
rm /tmp/current-secret.yaml
git add path/to/sealed-secret.yaml
git commit -m "fix: Regenerate corrupted sealed secret"

Troubleshooting

SealedSecret Not Decrypting

Symptoms: Secret not created, SealedSecret shows error in events

# Check SealedSecret status
kubectl describe sealedsecret <name> -n <namespace>

# Check controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets --tail=50

Common causes:

Wrong namespace (SealedSecrets are namespace-scoped by default)
Sealing key mismatch (sealed with different key)
Corrupted encrypted data (re-seal from plain secret)

"Unable to decrypt" Error

# Error: "no key could decrypt secret"

# Verify available keys
kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key

# If key was rotated, re-seal the secret with current key

kubeseal Connection Failed

# Error: "cannot fetch certificate"

# Verify controller is running
kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

# Verify service exists
kubectl get svc -n kube-system sealed-secrets-controller

Secret Updated But Application Uses Old Value

Some applications cache secrets. Force a refresh:

kubectl rollout restart deployment/<name> -n <namespace>

Controller Deployment

ArgoCD Application

Location: manifests/applications/seal-controller.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "-25"
spec:
  project: infrastructure
  sources:
    - repoURL: https://bitnami-labs.github.io/sealed-secrets
      chart: sealed-secrets
      targetRevision: 2.18.0
      helm:
        valueFiles:
          - $values/manifests/base/sealed-secrets/values.yaml
    - repoURL: git@github.com:imcbeth/homelab.git
      path: manifests/base/sealed-secrets
      targetRevision: HEAD
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: kube-system

Resource Configuration

Optimized for Raspberry Pi cluster:

resources:
  requests:
    cpu: 25m
    memory: 32Mi
  limits:
    cpu: 100m
    memory: 64Mi

Actual usage: ~1m CPU, ~9Mi memory

Quick Reference

Seal a New Secret

kubeseal --controller-name=sealed-secrets-controller \
         --controller-namespace=kube-system \
         --format yaml \
         < plain-secret.yaml \
         > sealed-secret.yaml

Backup Sealing Key

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o yaml > sealed-secrets-key-backup.yaml

View Controller Logs

kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets -f

List All SealedSecrets

kubectl get sealedsecrets -A

Best Practices

Security

Never commit unencrypted secrets - Always use kubeseal
Back up sealing keys - Store securely for disaster recovery
Rotate keys periodically - Re-seal secrets with new keys
Limit secret scope - Use namespace-specific secrets when possible

GitOps Workflow

Use output redirect - Always use kubeseal ... > file.yaml instead of copy-paste
Name convention - Use *-sealed.yaml suffix for SealedSecret files
Pre-commit exclusions - SealedSecrets are excluded from yamllint (long encrypted lines)
Test in staging - Verify secrets work before promoting to production

Helm-Managed Secrets

Don't use SealedSecrets for secrets managed by Helm charts:

Grafana admin password (auto-generated)
Other chart-created secrets

These secrets are created by Helm and would conflict with SealedSecrets.

References

Last Updated: 2026-02-12 Status: Production, All secrets migrated Managed By: ArgoCD (manifests/applications/seal-controller.yaml)

Overview​

Why Sealed Secrets?​

The Problem with Kubernetes Secrets​

Previous Approach: git-crypt​

Sealed Secrets Solution​

Architecture​

Managed Secrets​

Secrets NOT Managed by Sealed Secrets​

Using kubeseal​

Prerequisites​

Creating a New SealedSecret​

Secret Rotation Procedures​

Rotating a Secret Value​

Step 1: Create the New Plain Secret​

Step 2: Seal the New Secret​

Step 3: Clean Up Plain Secret​

Step 4: Commit and Deploy​

Step 5: Verify Deployment​

Step 6: Restart Affected Pods (if needed)​

Batch Secret Rotation​

Sealing Key Management​

Backup Sealing Key​

Rotate Sealing Key​

Manual Key Rotation (Recommended)​

Re-seal All Secrets After Key Rotation​

Restore Sealing Key​

Disaster Recovery​

Scenario 1: Cluster Rebuilt, Backup Available​

Scenario 2: Cluster Rebuilt, No Backup​

Scenario 3: Single Secret Corrupted​

Troubleshooting​

SealedSecret Not Decrypting​

"Unable to decrypt" Error​

kubeseal Connection Failed​

Secret Updated But Application Uses Old Value​

Controller Deployment​

ArgoCD Application​

Resource Configuration​

Quick Reference​

Seal a New Secret​

Backup Sealing Key​

View Controller Logs​

List All SealedSecrets​

Best Practices​

Security​

GitOps Workflow​

Helm-Managed Secrets​

Related Documentation​

References​

Overview

Why Sealed Secrets?

The Problem with Kubernetes Secrets

Previous Approach: git-crypt

Sealed Secrets Solution

Architecture

Managed Secrets

Secrets NOT Managed by Sealed Secrets

Using kubeseal

Prerequisites

Creating a New SealedSecret

Secret Rotation Procedures

Rotating a Secret Value

Step 1: Create the New Plain Secret

Step 2: Seal the New Secret

Step 3: Clean Up Plain Secret

Step 4: Commit and Deploy

Step 5: Verify Deployment

Step 6: Restart Affected Pods (if needed)

Batch Secret Rotation

Sealing Key Management

Backup Sealing Key

Rotate Sealing Key

Manual Key Rotation (Recommended)

Re-seal All Secrets After Key Rotation

Restore Sealing Key

Disaster Recovery

Scenario 1: Cluster Rebuilt, Backup Available

Scenario 2: Cluster Rebuilt, No Backup

Scenario 3: Single Secret Corrupted

Troubleshooting

SealedSecret Not Decrypting

"Unable to decrypt" Error

kubeseal Connection Failed

Secret Updated But Application Uses Old Value

Controller Deployment

ArgoCD Application

Resource Configuration

Quick Reference

Seal a New Secret

Backup Sealing Key

View Controller Logs

List All SealedSecrets

Best Practices

Security

GitOps Workflow

Helm-Managed Secrets

Related Documentation

References