Scenario
You own a polyglot microservice platform. Traces, metrics, and logs exist, but performance regressions still require manual, host-by-host profiling. The team wants OpenTelemetry Profiles to send CPU, off-CPU, and heap data through OTLP to a Collector and correlate it with traces and spans. The signal entered Public Alpha in 2026. Design the evaluation, pilot, data path, and rollback plan.
What the interviewer evaluates
- Distinguishing the questions answered by profiles, logs, metrics, and traces.
- Recognizing Alpha maturity, backend readiness, and language-specific collection gaps.
- Controlling sampling overhead, storage cost, sensitive data, and access.
- Defining measurable pilot gates, isolation boundaries, and rollback.
Clarifying questions
Confirm whether the target problem is CPU, memory, lock wait, or tail latency; which languages and runtimes must be covered; current profilers, SLOs, sampling budget, retention, and compliance boundaries. Ask whether a backend supports OTLP Profiles and whether pprof, JFR, or the current APM can remain during failures.
30-second answer
I would run a pilot but keep the Alpha signal out of critical alerting. Select a few Linux services, collect at low frequency through a separate Collector, and retain pprof/JFR as the baseline. Measure time to diagnosis, CPU overhead, sampling coverage, cost per GB, trace-correlation rate, and redaction defects. Expand only after backend, access-control, and rollback drills pass. A format or Collector failure must be able to stop export without affecting request traffic.
Step-by-step reasoning
1. Define signal boundaries
Logs describe discrete events, metrics describe system-level values, traces describe a request’s path, and profiles describe which code consumes resources. Profiles complement rather than replace the other signals; correlate them through resource, trace, or span identifiers to shorten root-cause analysis.
2. Design collection
Sampling profilers periodically record stacks for continuous, low overhead. Instrumentation profilers can record runtime events such as allocations, locks, or garbage collection. Start with an eBPF agent or language-native tool on isolated nodes, then let the Collector filter, rate-limit, batch, and route data before OTLP export.
3. Control cost and privacy
Set sampling rates and CPU budgets by service tier, prioritizing services with 99th-percentile latency anomalies. Restrict symbols, arguments, and sensitive path data; isolate tenants; and use separate retention for raw and aggregated profiles. Monitor drop rate, Collector queues, egress bandwidth, and storage cost.
4. Handle Alpha risk
The official documentation labels Profiles Alpha, and the announcement says it should not be used for critical production workloads while production-ready backends are still emerging. Use a feature flag, an independent resource quota, an old-tool baseline, and a configuration-only rollback. Do not migrate every language and backend merely for standardization.
High-quality sample answer
I would optimize for shorter performance-diagnosis time, not immediate replacement of existing profilers. Phase one selects two Linux services, one Go and one JVM, keeps pprof/JFR output, and records baseline diagnosis time and resource cost. Phase two deploys an isolated Collector with low-frequency CPU profiling; raise the rate only for SLO or CPU anomalies. The Collector filters by environment, enforces limits, redacts fields, and routes OTLP through a queue separate from production traffic. Phase three enables trace/span correlation and tests whether a slow span can reach the responsible stack. Gates are bounded CPU overhead, sufficient coverage, lower P95 diagnosis time, acceptable monthly cost, and zero sensitive-field leaks. Because Profiles is Alpha, keep the current backend and tools. If the Collector backs up, the backend rejects data, or access boundaries fail, disable export; business requests never depend on this path.
Common mistakes
- Treating Alpha Profiles as a stable universal replacement.
- Replacing traces, metrics, or logs with profiles without explaining correlation.
- Retaining every raw stack and symbol indefinitely despite cost and sensitivity.
- Sharing a business-critical queue with profile export so failures delay requests.
- Saying “deploy eBPF” without sampling budgets, backend compatibility, or a kill switch.
Follow-up questions and responses
“Why not keep pprof or JFR?”
Keep them as baseline and fallback. Profiles adds a common model, an OTLP pipeline, and cross-signal correlation; migration is justified only if the pilot proves those benefits.
“How do you correlate traces and profiles?”
Record resource and available traceid or spanid metadata on profile samples and preserve the fields through the Collector and backend. Accept that not every sample can map to a request.
“When would you stop the pilot?”
Disable export when CPU, cost, or privacy risk exceeds its threshold, or backend stability cannot support the rollback window. Keep existing tools and revisit after the signal matures.