Falco Runtime Security

Production Status

Status: ✅ OPERATIONAL (Deployed: 2026-01-29, PR #336)

Component	Status	Resources
Falco DaemonSet	Running on all 5 nodes	50m/128Mi → 500m/512Mi
Falcosidekick	Running	10m/32Mi → 100m/64Mi
Falcosidekick WebUI	Running	10m/32Mi → 100m/128Mi
Redis (redis-stack)	Running	50m/1536Mi → 200m/2Gi

Operational Highlights:

✅ Falco monitoring syscalls on all nodes
✅ Modern eBPF driver (efficient on ARM64)
✅ Prometheus metrics being scraped (30s interval)
✅ Grafana dashboard displaying real-time events
✅ AlertManager receiving critical/warning alerts
✅ Loki receiving all security events
✅ Custom rules for homelab environment

Overview

Falco is a cloud-native runtime security tool that detects unexpected application behavior and alerts on threats at runtime. It uses eBPF to monitor system calls and can detect:

Container escape attempts
Privilege escalation
Reverse shell connections
Cryptocurrency mining
Sensitive file access
Anomalous network activity

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                           FALCO ARCHITECTURE                                  │
└─────────────────────────────────────────────────────────────────────────────┘

  NODE 1              NODE 2              NODE 3              NODE 4              NODE 5
┌──────────┐       ┌──────────┐       ┌──────────┐       ┌──────────┐       ┌──────────┐
│  Falco   │       │  Falco   │       │  Falco   │       │  Falco   │       │  Falco   │
│ DaemonSet│       │ DaemonSet│       │ DaemonSet│       │ DaemonSet│       │ DaemonSet│
│  (eBPF)  │       │  (eBPF)  │       │  (eBPF)  │       │  (eBPF)  │       │  (eBPF)  │
└────┬─────┘       └────┬─────┘       └────┬─────┘       └────┬─────┘       └────┬─────┘
     │                  │                  │                  │                  │
     │                  │    gRPC :5060    │                  │                  │
     └──────────────────┴──────────────────┴──────────────────┴──────────────────┘
                                           │
                                    ┌──────▼──────┐
                                    │ Falcosidekick│
                                    │  (router)    │
                                    └──────┬──────┘
                                           │
              ┌────────────────────────────┼────────────────────────────┐
              │                            │                            │
       ┌──────▼──────┐              ┌──────▼──────┐              ┌──────▼──────┐
       │ Alertmanager │              │    Loki     │              │  WebUI      │
       │  :9093       │              │   :3100     │              │  :2802      │
       │ (critical)   │              │  (all logs) │              │ (visual)    │
       └─────────────┘              └─────────────┘              └─────────────┘

Deployment

Falco is deployed via ArgoCD using the official Falcosecurity Helm chart.

ArgoCD Application: manifests/applications/falco.yaml

Configuration: manifests/base/falco/values.yaml

Version: Helm chart 8.0.1 (App version 0.43.0)

Sync Wave: -5 (after monitoring stack for Prometheus/Loki integration)

Resource Configuration

Optimized for Raspberry Pi 5 cluster:

Component	CPU Request	CPU Limit	Memory Request	Memory Limit
Falco	50m	500m	128Mi	512Mi
Falcosidekick	10m	100m	32Mi	64Mi
Falcosidekick WebUI	10m	100m	32Mi	128Mi
Redis (redis-stack)	50m	200m	1536Mi	2Gi
WebUI init container	10m	50m	32Mi	64Mi

Redis Memory Sizing (Updated 2026-02-27)

The redis-stack server (with RediSearch, TimeSeries, JSON, Bloom, Gears modules) requires significantly more memory than plain redis. The RDB dump can exceed 1GB. maxmemory only caps key data, not module overhead (indexes, metadata). Container memory limit must account for both.

Configuration:

Storage: 2Gi PVC (expanded from 1Gi when 99% full, PR #473)
Memory limit: 2Gi (Gatekeeper max), request 1536Mi
TTL: 30 days to prevent unbounded data growth
Eviction: allkeys-lru with maxmemory: 1000mb

falcosidekick:
  webui:
    ttl: "30d"
    redis:
      storageSize: "2Gi"
      resources:
        requests:
          memory: 1536Mi
        limits:
          memory: 2Gi
      config:
        maxmemory: "1000mb"
        maxmemory-policy: "allkeys-lru"

Security Detection

Built-in Rules

Falco comes with extensive default rules covering:

Container Threats: Privileged containers, namespace escapes
File Access: Sensitive files (/etc/shadow, /etc/passwd)
Process Activity: Unexpected shell spawns, package managers
Network: Outbound connections, suspicious DNS queries
System: Kernel module loading, ptrace usage

Custom Homelab Rules

Custom rules tuned for the homelab environment:

# Cryptocurrency mining detection
- rule: Detect Cryptocurrency Mining
  desc: Detect cryptocurrency mining processes
  condition: spawned_process and proc.name in (xmrig, minerd, minergate)
  priority: CRITICAL
  tags: [cryptomining, mitre_execution]

# Reverse shell detection
- rule: Reverse Shell Detected
  desc: Detect reverse shell connections
  condition: spawned_process and proc.cmdline contains "/dev/tcp"
  priority: CRITICAL
  tags: [shell, mitre_execution]

Disabled Rules (Noise Reduction)

The following rules are disabled to reduce noise in a development/homelab environment:

Terminal shell in container - Common for debugging
Attach/Exec Pod - Common kubectl usage

Monitoring and Alerts

Grafana Dashboard

Access at: https://grafana.k8s.n37.ca

Dashboard: "Falco Runtime Security"

Panels include:

Overview: Critical/Error/Warning event counts (24h)
Event Timeline: Security events by priority over time
Event Analysis: Events by rule, namespace, and top pods
System Performance: Syscall event rate and memory usage
Drop Rate: Event processing efficiency

Prometheus Alerts

PrometheusRule: falco-security-alerts

Critical Alerts:

Alert	Description	Action Required
`FalcoCriticalSecurityEvent`	Any critical security event detected	Immediate investigation
`FalcoReverseShellDetected`	Reverse shell attempt	Isolate affected pod
`FalcoCryptominingDetected`	Cryptocurrency mining detected	Terminate and investigate

Warning Alerts:

Alert	Description	Action Required
`FalcoErrorSecurityEvent`	Error-level security events	Review within 24h
`FalcoHighEventRate`	>10 events/sec sustained	Investigate source
`FalcoDown`	Falco instance not running	Restore monitoring
`FalcoHighDropRate`	>1% event drop rate	Check resources

Info Alerts:

Alert	Description	Action Required
`FalcoNoEvents`	No events for 15 minutes	Verify Falco health

Integration Points

System	Purpose	Port
Alertmanager	Critical/warning alerts	9093
Loki	All security events	3100
Prometheus	Metrics scraping	8765
WebUI	Visual event browser	2802

Access WebUI

The Falcosidekick WebUI provides a visual interface for browsing security events.

# Port forward to access locally
kubectl port-forward -n falco svc/falco-falcosidekick-ui 2802:2802

# Open in browser
open http://localhost:2802

Common Operations

View Recent Events

# Check Falcosidekick logs for recent events
kubectl logs -n falco deployment/falco-falcosidekick --tail=50

# Query Loki for Falco events
# In Grafana Explore, use LogQL:
{namespace="falco"} |= "priority"

Test Falco Detection

# Trigger a test rule (read sensitive file)
kubectl exec -it -n default <any-pod> -- cat /etc/shadow

# Check for the event
kubectl logs -n falco daemonset/falco --tail=20 | grep shadow

Check Falco Health

# Verify all Falco pods are running
kubectl get pods -n falco -o wide

# Check Falco driver status
kubectl logs -n falco daemonset/falco | grep -i driver

# View syscall processing rate
kubectl exec -n falco daemonset/falco -- cat /proc/falco/stats

Update Falco Rules

# Falco rules are automatically updated via falcoctl
# To force an update:
kubectl rollout restart daemonset/falco -n falco

Troubleshooting

Falco Pods Failing to Start

Symptom: Falco pods in CrashLoopBackOff

Common Causes:

Kernel headers missing: eBPF driver needs kernel headers

# Check driver logs
kubectl logs -n falco daemonset/falco -c falco-driver-loader

Insufficient privileges: Falco needs privileged mode

# Verify securityContext
kubectl get daemonset -n falco falco -o yaml | grep -A5 securityContext

High Event Drop Rate

Symptom: FalcoHighDropRate alert firing

Solution:

Increase buffer size in values.yaml:

driver:
  modernEbpf:
    bufSizePreset: 8  # Increase from 4

Or reduce event volume with rule tuning

No Events Being Generated

Symptom: Falco running but no events in logs

Check:

Verify driver is loaded:

kubectl exec -n falco daemonset/falco -- cat /proc/falco/loaded

Check if rules are loaded:

kubectl exec -n falco daemonset/falco -- falcoctl artifact list

Alertmanager Not Receiving Alerts

Check Falcosidekick configuration:

kubectl logs -n falco deployment/falco-falcosidekick | grep -i alertmanager

Verify network connectivity:

kubectl exec -n falco deployment/falco-falcosidekick -- \
  wget -qO- --timeout=5 http://alertmanager-operated.monitoring:9093/-/healthy

Network Policy

Falco namespace has a NetworkPolicy restricting traffic:

Allowed Ingress:

Prometheus (metrics scraping on 8765, 2801)
Internal namespace communication (gRPC 5060, HTTP 2801, WebUI 2802)

Allowed Egress:

DNS (kube-system:53)
Kubernetes API (6443)
Alertmanager (monitoring:9093)
Loki (loki:3100)
HTTPS for rule downloads (443)

Configuration Files

File	Purpose
`manifests/applications/falco.yaml`	ArgoCD Application definition
`manifests/base/falco/values.yaml`	Helm values (driver, resources, rules)
`manifests/base/falco/falco-alerts.yaml`	PrometheusRule for security alerts
`manifests/base/grafana/dashboards/falco-security-dashboard.yaml`	Grafana dashboard
`manifests/base/network-policies/falco/network-policy.yaml`	Network isolation

Security Considerations

Privileged Mode: Falco requires privileged containers to access syscalls
Host PID Namespace: Required for process monitoring
eBPF: Uses kernel-level tracing (requires compatible kernel)
gRPC: Events sent to Falcosidekick over unencrypted gRPC (internal only)
Rule Updates: Automatic updates may introduce new detection rules

Resources

Official Documentation: falco.org/docs
Helm Chart: github.com/falcosecurity/charts
Rules Reference: falco.org/docs/rules
Falcosidekick: github.com/falcosecurity/falcosidekick

Trivy Operator - Container vulnerability scanning
Network Policies - Falco namespace isolation
Grafana Dashboards - Security dashboards

Production Status​

Overview​

Architecture​

Deployment​

Resource Configuration​

Security Detection​

Built-in Rules​

Custom Homelab Rules​

Disabled Rules (Noise Reduction)​

Monitoring and Alerts​

Grafana Dashboard​

Prometheus Alerts​

Integration Points​

Access WebUI​

Common Operations​

View Recent Events​

Test Falco Detection​

Check Falco Health​

Update Falco Rules​

Troubleshooting​

Falco Pods Failing to Start​

High Event Drop Rate​

No Events Being Generated​

Alertmanager Not Receiving Alerts​

Network Policy​

Configuration Files​

Security Considerations​

Resources​

Related Documentation​