cert-manager - TLS Certificate Management

Overview

cert-manager is a Kubernetes controller that automates the management and issuance of TLS certificates from various sources, including Let's Encrypt. It ensures certificates are valid and up-to-date, automatically renewing them before expiry.

Key Features

Automated Certificate Issuance: Automatically requests and installs TLS certificates
Multiple Issuers: Supports Let's Encrypt, private CA, and other ACME providers
DNS-01 Challenge: Uses Cloudflare DNS for wildcard certificate support
Auto-Renewal: Certificates automatically renewed 60 days before expiry
Kubernetes Native: Managed via Custom Resource Definitions (CRDs)

Deployment Details

ArgoCD Application

Name: cert-manager
Namespace: cert-manager
Project: infrastructure
Sync Wave: -10
Helm Chart: jetstack/cert-manager v1.19.3
Auto-Sync: Enabled (prune, selfHeal)

Resources

# Controller resources
resources:
  requests:
    cpu: 10m
    memory: 128Mi
  limits:
    cpu: 100m
    memory: 256Mi

# Webhook resources
webhook:
  resources:
    requests:
      cpu: 10m
      memory: 128Mi
    limits:
      cpu: 50m
      memory: 128Mi

# CA Injector resources
cainjector:
  resources:
    requests:
      cpu: 10m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 256Mi

Resource Limits Updated (2026-02-14)

Resource limits were added to all cert-manager components (PR #451) to comply with Gatekeeper's require-resource-limits policy after removing cert-manager from the Gatekeeper exclusion list.

Total Resource Usage:

CPU Requests: 30m (0.15% of 20 cores)
Memory Requests: 384Mi

Configuration

ClusterIssuers

Two ClusterIssuers are configured for Let's Encrypt:

1. Production Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lets-encrypt-k8s-n37-ca-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: imcbeth1980@gmail.com
    privateKeySecretRef:
      name: lets-encrypt-k8s-n37-ca-key-prod
    solvers:
    - dns01:
        cloudflare:
          email: imcbeth1980@gmail.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

When to use: Production ingresses with public-facing domains

2. Staging Issuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lets-encrypt-k8s-n37-ca-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: imcbeth1980@gmail.com
    privateKeySecretRef:
      name: lets-encrypt-k8s-n37-ca-key-staging
    solvers:
    - dns01:
        cloudflare:
          email: imcbeth1980@gmail.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token

When to use: Testing certificate issuance without rate limits

DNS-01 Challenge with Cloudflare

Why DNS-01?

Wildcard Certificates: Supports *.k8s.n37.ca certificates
Internal Services: Works for services not publicly accessible
Firewall Friendly: No need to expose port 80 publicly
Shared Secret: Reuses Cloudflare API token from cert-manager configuration

Cloudflare API Token

The API token is stored as a SealedSecret for GitOps compatibility:

SealedSecret: manifests/base/cert-manager/cloudflare-sealed.yaml
Decrypted Secret: cloudflare-api-token-secret in cert-manager namespace

# View decrypted secret (base64 encoded)
kubectl get secret cloudflare-api-token-secret -n cert-manager -o yaml

# View SealedSecret status
kubectl get sealedsecret cloudflare-api-token-secret -n cert-manager

Permissions Required:

Zone: DNS: Edit
Zone: Zone: Read

Token Scope: k8s.n37.ca zone only

See Secrets Management for details on managing SealedSecrets.

Certificate Management

Automatic Certificate Creation

Certificates are automatically created when an Ingress resource includes cert-manager annotations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    cert-manager.io/cluster-issuer: "lets-encrypt-k8s-n37-ca-prod"
spec:
  tls:
  - hosts:
    - example.k8s.n37.ca
    secretName: example-k8s-n37-ca-nginx-tls
  rules:
  - host: example.k8s.n37.ca
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

Currently Managed Certificates

Domain	Namespace	Secret Name	Status	Age
argocd.k8s.n37.ca	argocd	argocd-k8s-n37-ca-nginx-tls	Ready	4d6h
grafana.k8s.n37.ca	default	grafana-k8s-n37-ca-nginx-tls	Ready	31h
localstack.k8s.n37.ca	localstack	localstack-k8s-n37-ca-nginx-tls	Ready	4d5h

Certificate Lifecycle

Ingress Created: With cert-manager.io/cluster-issuer annotation
Certificate Resource Created: Automatically by cert-manager
ACME Challenge: DNS-01 challenge initiated with Cloudflare
TXT Record Created: _acme-challenge.example.k8s.n37.ca
Verification: Let's Encrypt verifies domain ownership
Certificate Issued: Stored as Kubernetes Secret
Ingress Updated: Uses the secret for TLS termination
Auto-Renewal: 60 days before expiry

Certificate Renewal

Renewal Window: 30 days before expiry
Process: Fully automated, no intervention required
Monitoring: Check certificate expiry in Grafana (future: Blackbox Exporter)

Operations

Verify cert-manager Status

# Check all cert-manager pods
kubectl get pods -n cert-manager

# Expected output:
# NAME                                       READY   STATUS
# cert-manager-xxxxxxxxxx-xxxxx              1/1     Running
# cert-manager-cainjector-xxxxxxxxxx-xxxxx   1/1     Running
# cert-manager-webhook-xxxxxxxxxx-xxxxx      1/1     Running

Check ClusterIssuers

# List all ClusterIssuers
kubectl get clusterissuer

# Expected output:
# NAME                                READY   AGE
# lets-encrypt-k8s-n37-ca-prod        True    4d
# lets-encrypt-k8s-n37-ca-staging     True    4d

# Check issuer status
kubectl describe clusterissuer lets-encrypt-k8s-n37-ca-prod

List All Certificates

# List certificates in all namespaces
kubectl get certificates -A

# Check specific certificate details
kubectl describe certificate grafana-k8s-n37-ca-nginx-tls -n default

View Certificate Details

# Get certificate information
kubectl get certificate grafana-k8s-n37-ca-nginx-tls -n default -o yaml

# Check TLS secret
kubectl get secret grafana-k8s-n37-ca-nginx-tls -n default -o yaml

# Decode certificate
kubectl get secret grafana-k8s-n37-ca-nginx-tls -n default -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

Manually Trigger Certificate Renewal

# Delete certificate to force renewal (it will be recreated)
kubectl delete certificate grafana-k8s-n37-ca-nginx-tls -n default

# Or annotate certificate to force renewal
kubectl annotate certificate grafana-k8s-n37-ca-nginx-tls -n default cert-manager.io/issue-temporary-certificate="true" --overwrite

Troubleshooting

Certificate Not Issuing

Check Certificate Status:
```
kubectl describe certificate <cert-name> -n <namespace>
```
Look for events describing the issue.

Check CertificateRequest:

kubectl get certificaterequest -n <namespace>
kubectl describe certificaterequest <request-name> -n <namespace>

Check Challenge:

kubectl get challenge -n <namespace>
kubectl describe challenge <challenge-name> -n <namespace>

Check Order:

kubectl get order -n <namespace>
kubectl describe order <order-name> -n <namespace>

Common Issues

Issue: "Waiting for DNS propagation"

Cause: DNS TXT record not yet visible to Let's Encrypt servers

Solution:

Wait 2-5 minutes for DNS propagation
Check Cloudflare DNS records for _acme-challenge entries
Verify Cloudflare API token has correct permissions

Verify DNS:

# Check DNS TXT record
dig _acme-challenge.grafana.k8s.n37.ca TXT +short

Issue: "Secret cloudflare-api-token-secret not found"

Cause: Cloudflare API token secret missing or SealedSecret not synced

Solution:

# Verify secret exists
kubectl get secret cloudflare-api-token-secret -n cert-manager

# Check SealedSecret status
kubectl get sealedsecret cloudflare-api-token-secret -n cert-manager
kubectl describe sealedsecret cloudflare-api-token-secret -n cert-manager

# If SealedSecret exists but secret doesn't, check Sealed Secrets controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

The secret is managed via SealedSecret (manifests/base/cert-manager/cloudflare-sealed.yaml) and should be automatically created by the Sealed Secrets controller.

Issue: "Rate limit exceeded"

Cause: Too many certificate requests to Let's Encrypt production

Solution:

Use staging issuer for testing: lets-encrypt-k8s-n37-ca-staging
Wait for rate limit reset (limits are per domain, per week)
See: Let's Encrypt Rate Limits

Issue: Certificate shows as "False" or "Unknown"

Cause: Various - check logs

Solution:

# Check cert-manager controller logs
kubectl logs -n cert-manager deployment/cert-manager -f

# Look for errors related to your certificate
kubectl logs -n cert-manager deployment/cert-manager | grep <cert-name>

Monitoring

Metrics

cert-manager exposes Prometheus metrics on port 9402:

# Certificate expiry time
certmanager_certificate_expiration_timestamp_seconds

# Certificate renewal success/failure
certmanager_certificate_renewal_count

# ACME client requests
certmanager_http_acme_client_request_count

Alerts (Future)

Recommended alerts to configure:

Certificate Expiring Soon: Alert if certificate expires in < 14 days
Certificate Renewal Failed: Alert on renewal failures
ClusterIssuer Not Ready: Alert if issuer status is not Ready

Adding New Certificates

For a New Ingress

Simply add the cert-manager annotation to your Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: my-app
  annotations:
    cert-manager.io/cluster-issuer: "lets-encrypt-k8s-n37-ca-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.k8s.n37.ca
    secretName: myapp-k8s-n37-ca-nginx-tls
  rules:
  - host: myapp.k8s.n37.ca
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80

Process:

Apply Ingress manifest
cert-manager detects annotation
Creates Certificate resource automatically
Initiates ACME challenge
Certificate issued and stored in secret
Ingress uses secret for TLS

For Testing (Use Staging)

Replace the issuer annotation:

annotations:
  cert-manager.io/cluster-issuer: "lets-encrypt-k8s-n37-ca-staging"

Integration with external-dns (Planned)

When external-dns is deployed, it will work alongside cert-manager:

external-dns: Creates DNS A/CNAME records pointing to ingress
cert-manager: Creates DNS TXT records for ACME challenges

Both use the same Cloudflare API token, no conflict.

Best Practices

Use Staging First: Test new certificates with staging issuer
Monitor Expiry: Set up alerts for expiring certificates
Secret Management: Keep cloudflare-api-token-secret secure and backed up
Rate Limits: Be aware of Let's Encrypt rate limits
Consistent Naming: Use pattern: <app>-k8s-n37-ca-nginx-tls for secrets
Namespace Isolation: Certificates are namespace-scoped, plan accordingly

Security Considerations

API Token Security: Cloudflare token managed via SealedSecret (encrypted in Git, decrypted at runtime)
Least Privilege: Token has minimal required permissions (DNS edit only)
Token Rotation: When rotating, create new SealedSecret with kubeseal
ACME Account: Private key stored securely in cert-manager namespace
Certificate Secrets: Contain private keys, protect namespace access
GitOps Safe: SealedSecrets can be safely committed to Git repository

Backup and Disaster Recovery

Critical Resources to Backup

ClusterIssuers:

kubectl get clusterissuer -o yaml > clusterissuers-backup.yaml

Cloudflare API Token Secret:

kubectl get secret cloudflare-api-token-secret -n cert-manager -o yaml > cloudflare-secret-backup.yaml

ACME Account Keys:

kubectl get secret lets-encrypt-k8s-n37-ca-key-prod -n cert-manager -o yaml > acme-key-prod-backup.yaml
kubectl get secret lets-encrypt-k8s-n37-ca-key-staging -n cert-manager -o yaml > acme-key-staging-backup.yaml

Recovery Process

Restore cert-manager via ArgoCD (from Git repository)
Restore secrets: kubectl apply -f <backup-file>.yaml
Verify ClusterIssuers are Ready: kubectl get clusterissuer
Certificates will automatically regenerate from Ingresses

Upgrade Procedure

cert-manager is managed by ArgoCD using Helm. To upgrade:

Update Chart Version: Edit manifests/applications/cert-manager.yaml in homelab repo
Check Release Notes: Review breaking changes at cert-manager Releases
Create PR: Follow GitOps workflow
ArgoCD Sync: Automatic after merge
Verify: Check pods and certificate renewal

Note: Always test in staging environment first if available.

Useful Commands Reference

# cert-manager status
kubectl get pods -n cert-manager
kubectl get clusterissuer

# Certificates
kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>

# Certificate chain (request → order → challenge)
kubectl get certificaterequest -n <namespace>
kubectl get order -n <namespace>
kubectl get challenge -n <namespace>

# Logs
kubectl logs -n cert-manager deployment/cert-manager -f

# Force certificate renewal
kubectl delete certificate <name> -n <namespace>

# Check certificate expiry
kubectl get secret <cert-secret> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

Resources

Official Documentation: cert-manager.io
Let's Encrypt: letsencrypt.org
Cloudflare API Tokens: Cloudflare API Tokens Guide
ACME Challenge Types: Let's Encrypt Challenge Types
Rate Limits: Let's Encrypt Rate Limits

nginx-ingress - Ingress controller using these certificates
ArgoCD - GitOps deployment of cert-manager
external-dns - Complementary DNS automation (planned)

Last Updated: 2026-02-14 Status: Production, Healthy Managed By: ArgoCD (manifests/applications/cert-manager.yaml) Secrets: SealedSecret (manifests/base/cert-manager/cloudflare-sealed.yaml)

Overview​

Key Features​

Deployment Details​

ArgoCD Application​

Resources​

Configuration​

ClusterIssuers​

1. Production Issuer​

2. Staging Issuer​

DNS-01 Challenge with Cloudflare​

Why DNS-01?​

Cloudflare API Token​

Certificate Management​

Automatic Certificate Creation​

Currently Managed Certificates​

Certificate Lifecycle​

Certificate Renewal​

Operations​

Verify cert-manager Status​

Check ClusterIssuers​

List All Certificates​

View Certificate Details​

Manually Trigger Certificate Renewal​

Troubleshooting​

Certificate Not Issuing​

Common Issues​

Issue: "Waiting for DNS propagation"​

Issue: "Secret cloudflare-api-token-secret not found"​

Issue: "Rate limit exceeded"​

Issue: Certificate shows as "False" or "Unknown"​

Monitoring​

Metrics​

Alerts (Future)​

Adding New Certificates​

For a New Ingress​

For Testing (Use Staging)​

Integration with external-dns (Planned)​

Best Practices​

Security Considerations​

Backup and Disaster Recovery​

Critical Resources to Backup​

Recovery Process​

Upgrade Procedure​

Useful Commands Reference​

Resources​

Related Documentation​

Overview

Key Features

Deployment Details

ArgoCD Application

Resources

Configuration

ClusterIssuers

1. Production Issuer

2. Staging Issuer

DNS-01 Challenge with Cloudflare

Why DNS-01?

Cloudflare API Token

Certificate Management

Automatic Certificate Creation

Currently Managed Certificates

Certificate Lifecycle

Certificate Renewal

Operations

Verify cert-manager Status

Check ClusterIssuers

List All Certificates

View Certificate Details

Manually Trigger Certificate Renewal

Troubleshooting

Certificate Not Issuing

Common Issues

Issue: "Waiting for DNS propagation"

Issue: "Secret cloudflare-api-token-secret not found"

Issue: "Rate limit exceeded"

Issue: Certificate shows as "False" or "Unknown"

Monitoring

Metrics

Alerts (Future)

Adding New Certificates

For a New Ingress

For Testing (Use Staging)

Integration with external-dns (Planned)

Best Practices

Security Considerations

Backup and Disaster Recovery

Critical Resources to Backup

Recovery Process

Upgrade Procedure

Useful Commands Reference

Resources

Related Documentation