OpenTelemetry: Distributed Tracing with Jaeger
In monolithic architectures, diagnosing slow API endpoints involves inspecting a single application log stream. However, in distributed microservices architectures — where an API request traverses multiple API gateways, authentication services, backend databases, and asynchronous message queues — identifying which service introduced latency is difficult without distributed tracing.
Distributed Tracing tracks the complete execution path of a request across multi-service boundaries. OpenTelemetry (OTel) provides vendor-neutral APIs and SDKs to capture traces, while Jaeger visualizes request paths as interactive waterfall graphs. This guide details W3C Trace Context propagation, OpenTelemetry Collector routing, and Jaeger bottleneck diagnosis.
Mental Model: Trace Context Propagation Across Service Boundaries
Distributed tracing connects fragmented logs into a single coherent request story using two core primitives: Traces and Spans.
A Trace represents the end-to-end execution path of a request through the system, identified by a globally unique 128-bit TraceID. A Span represents a single unit of work (such as an HTTP handler execution or a SQL database query) containing a SpanID, start/end timestamps, and key-value tags (http.status_code=200, db.statement).
To preserve trace continuity across microservice HTTP/gRPC boundaries, services pass a W3C Trace Context header (traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01). The downstream service extracts the parent TraceID and attaches its new local SpanID to the parent trace. For observability infrastructure, review observability opentelemetry grafana and securing microservices istio service mesh mtls.
Quick reference
- TraceID uniquely identifies the entire end-to-end request path across all services.
- SpanID tracks individual execution units with precise microsecond duration timers.
- W3C TraceContext header (traceparent) propagates trace identity over HTTP and gRPC headers.
- Child spans reference parent SpanIDs to reconstruct hierarchical request call trees.
- Eliminates guesswork when troubleshooting latency spikes in multi-tier microservices.
Remember this
Propagate W3C traceparent headers across microservice calls to stitch isolated spans into unified request traces.
OpenTelemetry Architecture: API, SDK, & OTel Collector
OpenTelemetry decouples telemetry instrumentation from backend storage vendors. OTel consists of three architectural components:
1. OTel API: Defines abstract tracing primitives embedded into application source code without backend dependencies.
2. OTel SDK: Implements the API, handling in-memory span batching, tail sampling, and export protocols (OTLP/gRPC).
3. OTel Collector: A proxy daemon that receives telemetry from SDKs, processes and filters data (attributes/redact), and exports traces to Jaeger, Grafana Tempo, or Datadog.
Deploying the OTel Collector as a sidecar or DaemonSet offloads export network retries and data transformation CPU load from application pods.
Quick reference
- OTel API provides vendor-neutral instrumentation abstractions for application code.
- OTel SDK manages in-memory span batching and async OTLP/gRPC network exports.
- OTel Collector receives, processes, redacts, and exports trace batches out-of-process.
- Prevents vendor lock-in by allowing backend exporter changes via YAML collector configs.
- Reduces application pod CPU overhead by offloading span compression to collector daemons.
Remember this
Deploy the OpenTelemetry Collector daemon to batch, filter, and export traces without vendor lock-in.
Configuring W3C Trace Context Header Propagation in HTTP/gRPC
Automatic trace propagation requires configuring OTel HTTP client instrumentations (opentelemetry/instrumentation-http, otelhttp).
When an application uses instrumented HTTP clients (like Axios, Fetch, or Go net/http), OTel automatically injects the traceparent header into outgoing requests. Incoming HTTP servers parse the header, setting the current active span in Context.
For asynchronous messaging (Kafka, RabbitMQ), OTel injects trace headers into message metadata headers (record.headers()), allowing consumer workers to link asynchronous background job execution back to the original HTTP request trace.
Quick reference
- Auto-instrumentation libraries inject traceparent headers into outbound HTTP/gRPC requests.
- Extractors parse incoming HTTP headers to set the active TraceContext in application context.
- Inject trace context into Kafka record headers for asynchronous event-driven trace linking.
- Record exception events (.recordException(err)) inside active spans for error root-cause tracking.
- Configure OTLP gRPC exporters with TLS encryption for secure telemetry transport.
Remember this
Inject trace context into HTTP, gRPC, and Kafka headers to maintain continuous trace paths across sync and async calls.
Visualizing Latency Spans & Bottlenecks in Jaeger UI
Once spans flow from OTel Collectors into Jaeger, the Jaeger UI transforms raw trace JSON into interactive timeline waterfall charts.
In Jaeger, engineers inspect request paths to spot latency bottlenecks immediately. Long horizontal bars indicate slow database queries, external API call timeouts, or serialization delays. Clicking a span reveals structured metadata: SQL statements, HTTP status codes, pod hostnames, and error stack traces.
Use Probabilistic Tail-Based Sampling in OTel Collector to retain 100% of error traces and high-latency outlier traces (>2s) while sampling only 1% of normal 200 OK requests, reducing storage disk costs by 90%.
Quick reference
- Jaeger UI renders trace timelines as interactive Gantt waterfall charts for instant bottleneck spot checks.
- Inspect span attributes to view exact SQL queries, HTTP endpoints, and container hostnames.
- Correlate trace logs by embedding TraceID tags directly into application log messages.
- Apply Tail-Based Sampling in OTel Collector to preserve 100% of error and slow traces.
- Export Jaeger trace metrics to Grafana for high-level P99 latency trend dashboards.
Remember this
Use Jaeger waterfall views to identify latency bottlenecks and deploy tail-based sampling to optimize storage costs.
Key takeaway
To test distributed tracing, send a request through your API gateway. Open Jaeger UI (http://localhost:16686), search for the TraceID, and verify all microservice spans appear in the waterfall view.
Related Articles
Explore this topic