Kernel-level container insights: Utilizing eBPF with Cilium, Tetragon, and SBOMs for security
As applications become more distributed, traditional monitoring and security tools are failing to keep pace. This article explores how eBPF, when utilized by the graduated CNCF Cilium and its sub-project Tetragon, combined with Software Bills of Materials (SBOMs), can provide insights and a security feedback loop for modern systems.
We’ll create a container image and its SBOM. We’ll then launch it, simulate a breach, and see how our eBPF-based setup with Tetragon captures the issue. This will show how the SBOM can be used to debug and understand the incident.
Combining eBPF-driven observability and SBOM context
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that is adopted to enhance networking performance, provide deep observability, and strengthen security. It’s an enabling technology for CNCF projects such as Cilium, Falco, and Pixie. Tech companies like Google, Netflix, Cloudflare, and Microsoft use eBPF: Meta, for example, shared that it reduced CPU usage by 20% through kernel-level profiling with eBPF.
The SBOM, which provides an inventory of software components, is an essential component in understanding and securing these systems. Cloud Native Buildpacks, utilized by enterprises like Broadcom-VMware, Salesforce, and Red Hat and integrated into Google Cloud services, offer an automated way to produce SBOMs.
Why is the combination of eBPF-based tools and SBOMs useful for security? eBPF allows tools like Cilium’s Tetragon to observe kernel-level activities such as system calls, network traffic, and process executions in real time with low overhead. This provides raw data on what is happening inside containers.
Raw data, however, often needs context to be useful for security investigations. An SBOM provides this context by listing every software component and its version within the container.
Monitoring containers
This tutorial demonstrates how to use Cilium and Tetragon for eBPF-based security observability in Kubernetes, and how an SBOM helps contextualize runtime events.
Prerequisites:
- A Kubernetes cluster (e.g., Kind).
- pack CLI, kubectl, and Helm v3 installed.
- A sample application.
Step 1: Build container & generate SBOM
We’ll use Paketo Buildpacks to produce our application image and its SBOM. While SBOMs can be generated through various means, a system like Cloud Native Buildpacks, via a Paketo Buildpacks, provides an automated method to include them in your build pipeline. In your application’s source code directory, run:
pack build my-java-app \
--builder paketobuildpacks/builder-jammy-full \
--path . \
--sbom-output-dir ./sbom-output
This creates my-java-app:latest locally and outputs SBOMs to ./sbom-output.
Step 2: Deploy to Kubernetes
Create my-app-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-java-app-deployment
spec:
replicas: 1
selector:
matchLabels:
app: my-java-app
template:
metadata:
labels:
app: my-java-app
spec:
containers:
- name: my-java-app-container
image: my-java-app
imagePullPolicy: IfNotPresent # Needed for local images with Kind
ports:
- containerPort: 8080
For local clusters like kind, load the image first:
kind load docker-image my-java-app:latest
Then apply the deployment:
kubectl apply -f my-app-deployment.yaml
Step 3: Install Cilium CNI and Tetragon
Cilium provides cluster networking and is Tetragon’s parent project. Add and update its Helm repo and install it:
helm repo add cilium https://helm.cilium.io/
helm repo update
helm install cilium cilium/cilium --version 1.15.5 --namespace kube-system
Tetragon uses eBPF for security observability. Add and update the Isovalent Helm repo and install it:
helm repo add isovalent https://helm.isovalent.com
helm repo update
helm install tetragon isovalent/tetragon --version 1.16.1 --namespace kube-system
Verify Tetragon pods are running (it may take some time): kubectl get pods -n kube-system -l app.kubernetes.io/name=tetragon
Step 4: Observe system activity with Tetragon
Stream Tetragon events. Keep this running in a separate terminal:
kubectl exec -it -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact
Step 5: Simulate an anomaly
Real-world attacks often involve a compromised binary making unexpected outbound calls. We’ll simulate this using curl. Imagine this is your my-java-app (or a component within it) attempting to exfiltrate data or contact a C2 server after being compromised.
Get your application pod’s name: kubectl get pods -l app=my-java-app
Then, execute this command (replace
kubectl exec -it
'curl -s -L --connect-timeout 5 https://www.sylvainkalache.com -o /dev/null'
This attempts to fetch and discard content from https://www.sylvainkalache.com. The key is that this network activity might be out of character for your container.
Step 6: Detect with Tetragon and leverage the SBOM for context
Your tetra getevents stream should show events like these:
process default/my-java-app-deployment-85b5674f45-vj6l5 /bin/sh -c "curl -s -L --connect-timeout 5 https://www.sylvainkalache.com -o /dev/null"
process default/my-java-app-deployment-85b5674f45-vj6l5 /usr/bin/curl -s -L --connect-timeout 5 https://www.sylvainkalache.com -o /dev/null
From these Tetragon events, the outbound network request is identified. We see the shell execution, the curl process, and the TCP connect event to an external site on port 443.
Correlating Tetragon’s findings with the SBOM: A Log4Shell example
Consider a scenario like the Log4Shell vulnerability (CVE-2021-44228). An attacker exploits this in a Java application, causing the Java process to make an unexpected outbound JNDI/LDAP network connection. Tetragon, using eBPF, would detect this network activity from your Java application process.
This is where the SBOM generated in Step 1 provides context:
1. Alert contextualization: Tetragon flags an outbound connection from your my-java-app process to an unknown external IP on an LDAP/RMI port.
2. SBOM inspection: You consult the SBOM for my-java-app, since it is a Java application, you would search its dependencies for “log4j-core” (the vulnerable component in Log4Shell). The SBOM reveals the exact version of any log4j-core library present.
3. Vulnerability confirmation: Cross-referencing this version against CVE-2021-44228, if a vulnerable log4j-core is listed, this suggests the unexpected network call is due to Log4Shell exploitation.
4. Impact assessment and targeted response: The SBOM confirms which applications use the vulnerable Log4j version. You understand the Java process is compromised.
A clearer view of container security
This example is a very trivial one, and a production setup would need to be much more sophisticated, but it demonstrates the goal and a path that cloud native security team can use to monitor their containers.
While this can be achieved with other tools, Cilium has become an industry standard, with over 500 contributing companies to the project, including Google, Cisco, and RedHat, as well as fast-growing mid-size enterprises, like Datadog, Cloudflare, and SUSE. It’s a solid project worth learning and investing in.