Coding interview: How would you write portable eBPF tracing with CO-RE?

Prompt and context

An observability agent must measure syscall latency across distributions and kernel versions. Building separately for every kernel creates an unmanageable release matrix, while hard-coded struct offsets break after upgrades. Explain BTF, CO-RE relocations, and libbpf responsibilities, then design a verifiable build, load, event, and compatibility path.

What the interviewer is testing

• Understanding the relationship between BTF type information, compiler relocation records, and load-time patching.
• Distinguishing layout portability from verifier, helper, kfunc, and program-type constraints.
• Designing event lifetime, ring-buffer or perf-buffer delivery, and loss accounting.
• Handling missing BTF, absent fields, insufficient capabilities, and load failures.

Questions to clarify first

1. Does the target kernel enable BTF and expose /sys/kernel/btf/vmlinux?
2. Which probe types are available—tracepoints, kprobes, fentry, or LSM—and what are the privilege boundaries?
3. What are the event rate, latency budget, tolerated loss, and user-space consumption model?
4. Which architectures, kernel versions, and container environments must be supported?
5. If a field is absent or the verifier rejects the program, should the agent disable the metric or use a fallback probe?

A 30-second answer framework

I would compile one BPF object with BTF and CO-RE relocation data, then have libbpf patch field offsets at load time from the running kernel's BTF. A generated skeleton would manage maps, programs, and the ring buffer while checking capabilities, BTF, and program type. Missing fields or load failures would be recorded and would disable the metric or switch to a stable tracepoint; the agent would never emit fabricated values. Benchmarks would cover multiple kernels, concurrent events, loss, and unload recovery.

Step-by-step deep answer

Step 1: Separate compile-time and load-time responsibilities

The compiler writes the BPF program, type information, and CO-RE relocation records into the object. At load time, libbpf uses target-kernel BTF to resolve structs, fields, and offsets and updates instruction immediates or offsets. CO-RE reduces per-kernel builds, but it does not make every helper, kfunc, or program type available.

Step 2: Choose a stable probe boundary

Prefer stable tracepoints or fentry and confirm the event fields exposed by the target kernel. Kprobes cover more paths but depend more on symbols and parameter layout. Express field access with CO-RE macros instead of fixed offsets. Read only necessary fields to reduce verifier work and event size.

Step 3: Design event structure and transport

Record pid, timestamp, syscall identity, and the minimum latency data in BPF, then send it through a ring buffer or perf buffer. Use a fixed-size event or an explicit version field, and validate length and version in user space. Track loss per CPU, process, and queue; never silently convert buffer overflow into zero latency.

Step 4: Make verifier constraints part of the design

Bounds checks, loop limits, pointer types, and helper return values must satisfy the verifier. Move complex parsing to user space and avoid unprovable traversal in BPF. Ensure compiler- and skeleton-generated map definitions match program accesses. Keep verifier-failure summaries and kernel versions in logs for diagnosis.

Step 5: Handle BTF and field differences

Check that target BTF is readable and use optional CO-RE fields or feature probes to detect field presence. If a layout change makes relocation fail, disable the metric or choose a compatible tracepoint; do not fill an unreadable field with a default and report precise latency. Validate architecture differences in a multi-kernel CI matrix.

Step 6: Design the build and release path

Pin clang, libbpf, and skeleton-generation versions, produce a BTF-carrying object, and record its build ID. Ship the user-space loader, BPF object, and minimal capability requirements. Run self-checks before loading. Container deployment must state required BPF filesystems, privileges, and kernel settings and provide an actionable failure reason.

Step 7: Verify performance, correctness, and rollback

Compare field values and event counts with an independent tool across distributions, architectures, and kernels. Stress CPU, memory, ring-buffer loss, user-space lag, and tail latency at high event rates. Keep the old object or a stable probe as a rollback path and confirm that links, maps, and threads are released on unload.

High-quality sample answer

I would compile one object containing BTF and CO-RE relocations and let libbpf patch offsets from target-kernel BTF. Probes would prefer stable tracepoints or fentry; BPF would emit a small versioned event, and a user-space skeleton would decode it through a ring buffer while counting loss. Startup checks would cover BTF, program type, helper capability, and field presence. A relocation or verifier failure would disable the metric or switch probes, never fabricate data. CI would span architectures, kernels, high event rates, and unload recovery, with the old probe retained for rollback.

Common mistakes

• Treating CO-RE as compatibility for every helper, kfunc, and verifier constraint.
• Hard-coding struct offsets and bypassing BTF and relocations.
• Omitting event version and length checks and treating truncated data as valid.
• Ignoring buffer loss and reporting an observation gap as zero latency.
• Filling missing fields with defaults and hiding a compatibility failure.

Follow-up questions and responses

Follow-up 1: When does CO-RE relocation happen?

Compilation writes relocation records into the BPF object. At load time, libbpf resolves them with target-kernel BTF and patches field information. Offsets are not recalculated for every event at runtime.

Follow-up 2: Can it run without vmlinux BTF?

It depends on available BTF, probe type, distribution, and libbpf capabilities. Startup checks should establish what is safe; if the field cannot be resolved, disable the program or use a stable probe that does not depend on it.

Follow-up 3: Why not use kprobes everywhere?

Kprobes are flexible but depend more on symbols, parameters, and inlining, so upgrade semantics are less stable. Tracepoints or fentry can reduce maintenance while still requiring availability checks.

Follow-up 4: How do you prove there are no missing reports?

Reconcile BPF event count, ring-buffer loss, user-space consumption, and an independent syscall count by CPU and time window. Any mismatch must surface as missing or degraded state, not silently become zero.

Follow-up 5: How do you debug a verifier rejection?

Save loader logs, kernel version, target architecture, BTF ID, and the verifier summary. Narrow the issue to pointer bounds, loop limits, helper use, or map definitions with a minimal reproducer, then rerun the multi-kernel matrix after fixing it.