Dev.to Security 🔐 Cybersecurity 👁 0 📖 11 min read

Detect a Runtime Compromise with Falco: 5 Useful Rules and Zero Alert Noise

Your cluster has image scanning, admission control and NetworkPolicies. All of that is prevention. None of it tells you what a process actually did at 03:14 last night inside a running pod. That is runtime detection, and

Your cluster has image scanning, admission control and NetworkPolicies. All of that is prevention. None of it tells you what a process actually did at 03:14 last night inside a running pod. That is runtime detection, and on Kubernetes the default answer is Falco.

The problem: a fresh Falco install ships ~100 default rules tuned for "demo on a laptop", not for a 60-node OpenShift cluster running CI jobs, Istio sidecars and log shippers. Within an hour you get thousands of Read sensitive file untrusted and Contact K8S API Server From Container events. Two weeks later nobody reads the channel, and your detection capability is worth exactly zero.

This article takes the opposite approach: install Falco with the modern eBPF driver, disable most of the default ruleset, keep five rules that map to real attacker behaviour, tune them with explicit allowlists, and route the result to Alertmanager through Falcosidekick. The goal is a handful of alerts per week that a human will actually open.

Context: what Falco sees, and why it's loud

Falco is a userspace process (deployed as a DaemonSet, one pod per node) that consumes a stream of syscalls from the kernel — execve, openat, connect, ptrace and friends — enriches them with container and Kubernetes metadata, and evaluates them against rules written in a small boolean DSL.

Three driver options exist:

  • modern eBPF (CO-RE) — recommended. Needs a kernel ≥ 5.8 with BTF. RHCOS on OpenShift 4.12+ qualifies. No kernel module, no compilation, no DKMS.
  • legacy eBPF probe — downloaded/compiled per kernel version.
  • kernel module — avoid on immutable OSes.

The noise comes from the fact that generic rules cannot know your workload. "A shell was opened in a container" is suspicious — unless it's your CI runner building images, in which case it fires 4000 times a day. Falco tuning is therefore not about writing clever rules; it's about writing precise exceptions for your own platform.

Installing Falco with the modern eBPF driver

Prerequisites 

# Check kernel version and BTF availability on a node
oc debug node/worker-0 -- chroot /host uname -r
oc debug node/worker-0 -- chroot /host ls -l /sys/kernel/btf/vmlinux

If /sys/kernel/btf/vmlinux exists, modern_ebpf will work.

Helm values 

# falco-values.yaml
driver:
  enabled: true
  kind: modern_ebpf

collectors:
  crio:
    enabled: true
    socket: /var/run/crio/crio.sock
  containerd:
    enabled: false
  docker:
    enabled: false

# Content of falco.yaml
falco:
  rules_files:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/rules.d
  json_output: true
  json_include_output_property: true
  buffered_outputs: true
  # Global floor: anything below NOTICE is dropped before it reaches an output
  priority: notice
  # Token bucket: 0.1 alert/s sustained, burst of 10. Stops alert storms.
  outputs:
    rate: 0.1
    max_burst: 10
  syscall_event_drops:
    actions:
      - log
    rate: 0.03333
    max_burst: 10

resources:
  requests:
    cpu: 100m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 1Gi

tolerations:
  - operator: Exists

Install it:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

oc new-project falco

helm upgrade --install falco falcosecurity/falco \
  --namespace falco \
  --values falco-values.yaml \
  --wait

OpenShift: grant the SCC

Falco needs privileged access to load the eBPF program and read /proc:

oc adm policy add-scc-to-user privileged -z falco -n falco
oc rollout restart daemonset/falco -n falco

Verify the driver is actually loaded 

oc logs -n falco ds/falco -c falco | head -30
# Expect: "Falco initialized with configuration file ..."
#         "Loading rules from file /etc/falco/falco_rules.yaml"
#         "Starting health webserver with threadiness ..."
#         "Enabled event sources: syscall"
#         "Opening 'syscall' source with modern BPF probe."

oc get pods -n falco -o wide

If you see Opening 'syscall' source with Kernel module, your values were not applied.

Step 1: silence the default ruleset

Before adding anything, remove what you will never act on. Falco lets you disable rules by name or by tag at startup. Keep the default file loaded (it provides the shared macros and lists that your own rules reuse), but mute the chatty families:

# falco-values.yaml (append)
falco:
  rules_files:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/rules.d
  # Disable entire noisy families by tag
  disabled_tags:
    - network
    - users
  # Re-enable nothing by default; our local file re-declares what we want

This is deliberately brutal. The five rules below are declared in your own file, so they survive. Everything else falls back to "logged at DEBUG, never emitted".

Step 2: five rules that are worth waking up for

Put them in falco_rules.local.yaml, the conventional override file. Two patterns are used:

  • override: condition: append — extend an existing default rule with exceptions (requires Falco ≥ 0.36).
  • a brand-new rule — when the default is too broad to salvage. 
# falco_rules.local.yaml
- required_engine_version: 0x000028

#############################################
# Shared allowlists — the heart of the tuning
#############################################
- list: shell_allowed_images
  items:
    - registry.internal.example.com/ops/debug-toolbox
    - quay.io/openshift/origin-cli

- list: build_namespaces
  items: ["openshift-builds", "ci-runners", "tekton-pipelines"]

- list: etc_write_allowed_procs
  items: [nginx, envoy, pilot-agent, update-ca-trust, dpkg, rpm]

- list: sa_token_readers
  items: [kubectl, oc, kubelet, pilot-agent, envoy, fluent-bit, vector, otelcol, prometheus]

- list: egress_allowed_namespaces
  items: ["openshift-monitoring", "openshift-logging", "egress-proxy"]

#############################################
# RULE 1 — Interactive shell in a container
# MITRE T1059 — Command and Scripting Interpreter
#############################################
- rule: Terminal shell in container
  condition: >
    and not container.image.repository in (shell_allowed_images)
    and not k8s.ns.name in (build_namespaces)
  override:
    condition: append

#############################################
# RULE 2 — Write below /etc inside a container
# MITRE T1222 / T1098 — persistence & config tampering
#############################################
- rule: Write below etc in container
  desc: A process inside a container opened a file under /etc for writing
  condition: >
    container
    and evt.type in (open, openat, openat2, creat)
    and evt.is_open_write = true
    and fd.name startswith /etc
    and not fd.name startswith /etc/nginx/conf.d
    and not fd.name startswith /etc/pki/ca-trust
    and not proc.name in (etc_write_allowed_procs)
    and not k8s.ns.name in (build_namespaces)
  output: >
    Write below /etc in container
    (file=%fd.name proc=%proc.cmdline ppid=%proc.pname uid=%user.uid
     container=%container.name image=%container.image.repository
     ns=%k8s.ns.name pod=%k8s.pod.name)
  priority: ERROR
  tags: [container, filesystem, persistence, T1222]

#############################################
# RULE 3 — ServiceAccount token read by an unexpected process
# MITRE T1552.001 — Credentials in files
#############################################
- rule: Service account token read by unexpected process
  desc: Something other than the known K8s clients opened the projected SA token
  condition: >
    open_read
    and container
    and fd.name contains "secrets/kubernetes.io/serviceaccount"
    and fd.name endswith "token"
    and not proc.name in (sa_token_readers)
  output: >
    SA token read by unexpected process
    (proc=%proc.cmdline exe=%proc.exepath uid=%user.uid
     container=%container.name image=%container.image.repository
     ns=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, k8s, credentials, T1552]

#############################################
# RULE 4 — Egress to the public internet (exfiltration)
# MITRE T1041 — Exfiltration over C2 channel
#############################################
- macro: private_destination
  condition: >
    (fd.snet = "10.0.0.0/8" or fd.snet = "172.16.0.0/12"
     or fd.snet = "192.168.0.0/16" or fd.snet = "169.254.0.0/16"
     or fd.snet = "127.0.0.0/8")

- rule: Unexpected outbound connection from container
  desc: A container connected to a public IP from a namespace that should only talk internally
  condition: >
    outbound
    and container
    and not private_destination
    and not k8s.ns.name in (egress_allowed_namespaces)
    and not container.image.repository in (shell_allowed_images)
  output: >
    Outbound connection to public IP from restricted namespace
    (dest=%fd.sip:%fd.sport proto=%fd.l4proto proc=%proc.cmdline
     container=%container.name image=%container.image.repository
     ns=%k8s.ns.name pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, network, exfiltration, T1041]

#############################################
# RULE 5 — A binary written at runtime is executed
# MITRE T1105 — Ingress tool transfer
#############################################
- rule: Drop and execute new binary in container
  condition: >
    and not k8s.ns.name in (build_namespaces)
  override:
    condition: append

Why these five

  1. Interactive shell — nearly every hands-on-keyboard intrusion starts with /bin/sh. Almost no production container legitimately spawns one.
  2. Write below /etc — covers cron persistence, /etc/ld.so.preload hijacking, CA trust tampering. A container filesystem should be immutable at runtime.
  3. SA token read — the token is the pivot from "code execution in a pod" to "cluster API access". Reading it from a process that is not a Kubernetes client is a strong signal.
  4. Unexpected egress — exfiltration and C2 beaconing. This is the rule most likely to be noisy, which is exactly why it is scoped by namespace allowlist rather than applied cluster-wide.
  5. Drop and execute — proc.is_exe_upper_layer=true means the executed binary was written into the container's writable layer after the image was built. Crypto miners, reverse shells and kernel exploits all land here.

Note what is not in the list: "Contact K8S API Server From Container", "Read sensitive file untrusted", "Launch Privileged Container". They are useful on paper and unusable in practice without weeks of tuning.

Step 3: ship the rules and validate them

The Helm chart renders customRules into a ConfigMap mounted at /etc/falco/rules.d/:

# falco-values.yaml (append)
customRules:
  falco_rules.local.yaml: |-
    - required_engine_version: 0x000028
    # ... paste the content above ...

helm upgrade --install falco falcosecurity/falco \
  -n falco -f falco-values.yaml

# Syntax check from inside a running pod
oc exec -n falco ds/falco -c falco -- \
  falco --validate /etc/falco/rules.d/falco_rules.local.yaml

# List the rules Falco actually loaded
oc exec -n falco ds/falco -c falco -- falco -L | grep -A1 "Terminal shell"

A malformed rule makes the pod crash-loop. Always validate before rolling the DaemonSet.

Step 4: route to Alertmanager with Falcosidekick

Falcosidekick is a small forwarder that receives Falco's JSON output over HTTP and fans it out to ~50 backends. Alertmanager is the right target when you already have an on-call rotation wired to it: you inherit grouping, inhibition, silences and escalation for free.

# falco-values.yaml (append)
falcosidekick:
  enabled: true
  replicaCount: 2
  webui:
    enabled: false
  config:
    debug: false
    customfields: "cluster:prod-eu-west-1,env:production"
    alertmanager:
      hostport: "http://alertmanager-operated.monitoring.svc.cluster.local:9093"
      endpoint: "/api/v2/alerts"       # v1 was removed in Alertmanager 0.27
      minimumpriority: "warning"       # second filter, after falco.priority
      checkcert: false
      expireafter: "10m"
      extralabels: "source:falco,team:security"
      customseveritymap: "critical:critical,error:critical,warning:warning,notice:info"

Two filters now stack: Falco drops anything below notice at the source, Falcosidekick drops anything below warning before it reaches Alertmanager. Debug-level events remain in the pod logs for forensics without paging anyone.

On the Alertmanager side, Falcosidekick sends alertname=<rule name>, source=falco, severity=<mapped>, plus every Falco output field as a label (dots become underscores: k8s.ns.name → k8s_ns_name).

# alertmanager.yaml (excerpt)
route:
  receiver: default
  group_by: [alertname, cluster]
  routes:
    - matchers:
        - source = "falco"
        - severity = "critical"
      receiver: security-oncall
      group_by: [alertname, k8s_ns_name]
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 4h
    - matchers:
        - source = "falco"
      receiver: security-slack
      group_by: [alertname, k8s_ns_name]
      group_wait: 60s
      repeat_interval: 24h

inhibit_rules:
  - source_matchers: [alertname = "Drop and execute new binary in container"]
    target_matchers: [source = "falco"]
    equal: [k8s_pod_name]

The inhibit rule matters: once a binary was dropped and executed in a pod, the follow-up shell/egress alerts for that same pod are noise. One incident, one page.

# Check your config before applying
amtool check-config alertmanager.yaml

# Watch what actually arrives
oc port-forward -n monitoring svc/alertmanager-operated 9093:9093
amtool alert query --alertmanager.url=http://localhost:9093 'source="falco"'

Step 5: prove it works

Never trust an untested detection pipeline.

oc new-project falco-test
oc run canary --image=registry.access.redhat.com/ubi9/ubi:latest -- sleep 3600
oc wait --for=condition=Ready pod/canary -n falco-test

# Rule 1: interactive shell
oc exec -it -n falco-test canary -- /bin/bash -c "echo hello"

# Rule 2: write below /etc
oc exec -n falco-test canary -- bash -c 'echo "* * * * * root id" >> /etc/crontab'

# Rule 3: SA token read
oc exec -n falco-test canary -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

# Rule 5: drop and execute
oc exec -n falco-test canary -- bash -c 'cp /bin/echo /tmp/notecho && /tmp/notecho pwned'

# Observe
oc logs -n falco ds/falco -c falco --tail=50 | jq -r '.rule + " | " + .output'
oc logs -n falco deploy/falco-falcosidekick --tail=20

For a broader smoke test, the upstream event-generator replays known attack patterns:

oc run event-generator -n falco-test --rm -it \
  --image=falcosecurity/event-generator:latest \
  --restart=Never -- run syscall --loop

Clean up afterwards: oc delete project falco-test.

Security best practices

  • Protect the pipeline itself. Falco runs privileged. Restrict the falco namespace with a dedicated SCC, forbid oc exec into it via RBAC, and ship Falco's own logs off-cluster — an attacker with node access will try to kill the DaemonSet first. Alert on falco pod count dropping below node count with a Prometheus rule.
  • Version the rules in Git, not in helm --set. Every change to falco_rules.local.yaml is a change to your detection coverage. It deserves a PR, a reviewer, and the falco --validate step in CI.
  • Allowlist by image digest or namespace, never by process name alone. proc.name in (nginx) is trivially bypassed by renaming a binary. Combine with container.image.repository or proc.exepath.
  • Watch for dropped events. syscall_event_drops in the logs means Falco could not keep up and you have blind spots. Scrape the Falco metrics endpoint and alert on falco_n_drops_total increasing.
  • Make containers immutable. Set readOnlyRootFilesystem: true wherever possible — rules 2 and 5 then become near-zero-false-positive, because any write is already abnormal.
  • Review exceptions quarterly. An allowlist entry added for a debug session in March is an attacker's open door in November.
  • Keep raw events longer than alerts. Alertmanager is for paging; send the full JSON stream to Loki/Elasticsearch for incident response and threat hunting.

Conclusion

Runtime detection fails for organisational reasons far more often than technical ones. A Falco install that emits 3000 events a day is not "more secure" than no Falco at all — it is strictly worse, because it creates the illusion of coverage.

Key takeaways:

  • Deploy with driver.kind: modern_ebpf on any kernel ≥ 5.8 with BTF: no module, no compilation, no breakage on node upgrades.
  • Start by disabling the default ruleset, then add back only rules you have a response procedure for. Five well-tuned rules beat a hundred ignored ones.
  • Tune with explicit lists and macros in falco_rules.local.yaml, using override: condition: append for built-in rules and custom rules when the default is unsalvageable.
  • Stack filters: falco.priority, outputs.rate, and minimumpriority in Falcosidekick — then let Alertmanager handle grouping, inhibition and escalation.
  • Test your detections with a canary pod and event-generator on every rule change, exactly as you would test application code.

Once the five rules are quiet in steady state, add a sixth. Then a seventh. Growing a ruleset from a silent baseline is sustainable; pruning one from a screaming baseline never happens.

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.

Share 𝕏 Twitter ✈ Telegram ↗ Reddit 🟧 HN
← News 🔐 Cybersecurity 🏠 Home