Linux Observability: eBPF & Kernel Tracing
Traditional application performance monitoring (APM) agents rely on user-space code instrumentation (such as bytecode manipulation in Java or monkey-patching in Node.js). Inserting monitoring probes inside user-space applications adds CPU overhead, increases memory allocation noise, and requires modifying application source code or container images.
Extended Berkeley Packet Filter (eBPF) revolutionizes system observability by allowing developers to run sandboxed C-like programs directly inside the Linux Kernel without modifying kernel source code or loading dangerous kernel modules (kmods). eBPF programs hook into kernel syscalls, network sockets, and file system operations with near-zero overhead. This guide details eBPF architecture, in-kernel verifier safety, eBPF Maps, kprobes/tracepoints, and Cilium Kubernetes networking.
Mental Model: Linux Kernel Space vs User Space Observability
User-space monitoring tools (like top, ps, or application-level loggers) possess limited visibility. They inspect system state via periodic polling of the /proc filesystem or HTTP metrics endpoints, missing transient microsecond spikes and kernel-level network packet drops.
eBPF (Extended Berkeley Packet Filter) grants programmable access to the Linux Kernel.
When a user-space application executes a system call (e.g., sys_enter_connect or sys_enter_write), the Linux kernel triggers attached eBPF programs instantly. eBPF captures kernel context (process ID, memory addresses, socket IP tuples) and streams telemetry to user-space daemons without context-switching overhead. For Linux fundamentals, review linux essentials for developers and observability opentelemetry grafana.
Quick reference
- eBPF runs sandboxed bytecode directly within the Linux Kernel without kernel module instability.
- Captures 100% of system calls, network packets, and disk I/O events without user-space code changes.
- Zero-overhead execution: operates in-kernel without expensive user/kernel context switches.
- Replaces intrusive application instrumentation with transparent, universal kernel-level telemetry.
- Powers modern cloud-native observability and security tools including Cilium, Falco, and Pixie.
Remember this
Run eBPF programs in the Linux kernel to achieve transparent, zero-overhead observability across all workloads.
eBPF Architecture: Bytecode Verification, JIT Compilation, & eBPF Maps
Because eBPF code executes inside the Linux Kernel, a buggy or malicious program could crash the operating system or cause kernel panics. To prevent catastrophic failure, Linux enforces a strict safety workflow:
1. Compilation: C code is compiled into eBPF bytecode using clang -target bpf.
2. In-Kernel Verifier: Before loading, the eBPF Verifier performs static code analysis. It verifies that the program contains no infinite loops, invalid pointer dereferences, or uninitialized memory reads.
3. JIT Compilation: Upon passing verification, the JIT (Just-In-Time) Compiler translates eBPF bytecode into native host machine instructions (x86_64 or ARM64) for maximum execution speed.
4. eBPF Maps: Key-value data structures (e.g., BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_RINGBUF) shared between kernel eBPF programs and user-space daemons for state storage.
Quick reference
- Clang/LLVM compiles C/Rust source code into platform-independent eBPF bytecode.
- eBPF Verifier performs static analysis to guarantee memory safety and loop termination.
- JIT Compiler translates eBPF bytecode into native machine instructions for native speed.
- eBPF Maps (Hash Maps, Ring Buffers) share state efficiently between kernel and user-space.
- Guarantees that kernel-space eBPF execution will never cause host operating system crashes.
Remember this
Rely on the in-kernel eBPF Verifier and JIT Compiler to execute memory-safe bytecode at native kernel speeds.
Attaching Programs to Kprobes, Uprobes, Tracepoints, & XDP Socket Hooks
eBPF programs achieve versatility by attaching to various kernel and user-space event hooks:
- Kprobes / Kretprobes: Dynamic kernel function entry/exit hooks. Allows tracing any internal kernel function (e.g., tcp_v4_connect).
- Tracepoints: Static kernel instrumentation points embedded by Linux kernel maintainers (e.g., sched_process_exec). Stable across Linux kernel updates.
- Uprobes / Uretprobes: Dynamic user-space function hooks (e.g., tracing SSL/TLS handshakes inside OpenSSL libssl.so).
- XDP (eXpress Data Path): Ultra-high-speed network packet processing hook running at the network driver level (NIC) before the Linux networking stack allocates a sk_buff buffer.
Quick reference
- Kprobes dynamically attach to kernel functions for deep kernel runtime diagnostic tracing.
- Tracepoints provide stable, backward-compatible kernel instrumentation across Linux versions.
- Uprobes trace user-space library functions (OpenSSL, Go runtimes) without recompiling apps.
- XDP processes network packets directly at the NIC driver layer for sub-microsecond DDoS mitigation.
- BCC (BPF Compiler Collection) and libbpf-bootstrap simplify writing eBPF probes in C and Go.
Remember this
Attach eBPF programs to Tracepoints for stability, Uprobes for app SSL tracing, and XDP for high-speed networking.
Cilium Container Networking, Hubble Flow Visualization, & Zero-Overhead Tracing
In Kubernetes clusters, traditional iptables-based kube-proxy routing creates linear latency scaling bottlenecks ($O(N)$ rule lookup overhead) as the number of Services grows into the thousands.
Cilium replaces kube-proxy with eBPF-based container networking.
Cilium uses eBPF socket maps (BPF_MAP_TYPE_SOCKMAP) to route network packets directly between pod sockets in kernel space, bypassing iptables and IPVS entirely. Paired with Hubble, Cilium visualizes real-time L3/L4 network flows, HTTP REST paths, and DNS queries without installing sidecar proxies (like Envoy) inside user application pods.
Quick reference
- Cilium replaces iptables kube-proxy routing with eBPF BPF_MAP_TYPE_SOCKMAP direct socket routing.
- Eliminates O(N) iptables packet processing bottlenecks in large multi-thousand pod Kubernetes clusters.
- Hubble provides real-time L3/L4 network flow visualization and HTTP/gRPC protocol inspection.
- Enforces network security policies at the Linux kernel layer without sidecar proxy overhead.
- Reduces pod-to-pod network latency by up to 60% compared to legacy flannel/kube-proxy setups.
Remember this
Deploy Cilium and Hubble to replace Kubernetes kube-proxy with high-performance eBPF networking.
Key takeaway
To test eBPF, install bcc-tools (sudo apt install bcc-tools). Run sudo execsnoop-bpfcc to trace all new OS process executions in real time.
Related Articles
Explore this topic