Distributed Tracing: OpenTelemetry & Jaeger
When a user request traverses ten distinct microservices, database clusters, and external payment APIs, diagnosing a sudden 3-second latency spike using isolated application log files is nearly impossible. Searching through disconnected stdout logs across hundreds of pod instances cannot reveal which specific service or database query delayed the user request.
Distributed Tracing tracks the entire execution path of a user request as it flows across microservice network boundaries. OpenTelemetry (OTel) is the CNCF vendor-neutral observability standard for generating, collecting, and exporting telemetry data (Traces, Metrics, Logs), while Jaeger provides deep trace visualization and latency waterfall root-cause analysis. This guide details OpenTelemetry SDK setup, W3C Trace Context header propagation, OTel Collector pipelines, and tail-based sampling.
Mental Model: Isolated Application Logs vs W3C Distributed Trace Context Spans
Isolated application logs describe local events inside a single microservice container, but lack correlation identifiers linking upstream HTTP requests to downstream RPC calls.
Distributed Tracing Data Model structures request lifecycles as a directed acyclic graph (DAG) of Spans grouped under a single Trace:
- Trace ID: A unique 128-bit global identifier generated at the ingress API gateway that follows the request across every microservice hop. - Span: Represents a single timed unit of work (e.g. an HTTP handler, gRPC call, or SQL query), recording start/end timestamps, status codes, and error attributes. For kernel observability comparisons, review mastering ebpf linux kernel observability and implementing service mesh traffic management envoy proxy.
Quick reference
- Traces represent the global end-to-end execution path of a user request across microservices.
- Spans measure individual units of work with microsecond start/end timestamps and key-value attributes.
- W3C Trace Context headers (traceparent) propagate trace IDs across HTTP/gRPC boundaries.
- Vendor-neutral OpenTelemetry standard unifies tracing, metrics, and log telemetry instrumentation.
- Powers real-time production observability at Uber, Red Hat, Shopify, and CoreConcept.
Remember this
Adopt OpenTelemetry distributed tracing to visualize microservice request lifecycles and pinpoint latency bottlenecks.
OpenTelemetry Collector Architecture, Receivers, Processors, & Exporters
Instead of embedding vendor-specific SDK exporters directly inside application microservices, deploy the OpenTelemetry Collector as an agent or gateway proxy:
1receivers:2 otlp:3 protocols:4 grpc: { endpoint: "0.0.0.0:4317" }5processors:6 batch:7 timeout: 1s8 send_batch_size: 81929 memory_limiter:10 check_interval: 1s11 limit_percentage: 7512exporters:13 jaeger:14 endpoint: "jaeger-collector:14250"15service:16 pipelines:17 traces:18 receivers: [otlp]19 processors: [memory_limiter, batch]20 exporters: [jaeger]Quick reference
- OTel Collector decouples telemetry collection from backend storage destinations.
- Receivers ingest OTLP (OpenTelemetry Protocol), Zipkin, or Jaeger spans over gRPC/HTTP.
- Processors batch, filter, scrub PII data, and limit memory consumption safely.
- Exporters forward sanitized traces to Jaeger, Tempo, Datadog, or Cloud Monitoring.
- Zero application code changes required when switching telemetry storage vendors.
Remember this
Deploy OpenTelemetry Collector pipelines to decouple telemetry processing from storage backends.
W3C Trace Context (traceparent) Propagation Across Microservice Hops
For distributed traces to span multiple services, microservices must inject and extract W3C Trace Context headers on outgoing network calls:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
- 00: Version identifier.
- 4bf92f35...: 128-bit global Trace ID.
- 00f067aa...: 64-bit Parent Span ID.
- 01: Trace Flags (sampled bit set).
OpenTelemetry HTTP and gRPC client instrumentation automatically injects traceparent headers into outgoing requests, allowing downstream services to attach child spans seamlessly.
Quick reference
- W3C traceparent header standardizes cross-vendor context propagation across HTTP/gRPC headers.
- Propagates global Trace ID and Parent Span ID to maintain causal execution relationships.
- Automatic SDK instrumentation wraps HTTP client fetch() and gRPC client stubs transparently.
- Context propagation preserves span causality through asynchronous messaging queues (Kafka/NATS).
- Ensures complete end-to-end visibility across polyglot microservice language stacks.
Remember this
Inject W3C traceparent headers across HTTP and gRPC boundaries to maintain trace causality.
Head-Based vs Tail-Based Sampling Strategies for Cost Control
Tracing 100% of high-volume production requests generates terabytes of telemetry data, creating massive storage bills. Tracing requires smart Sampling Strategies:
1. Head-Based Sampling: The root service decides whether to sample a trace at request start (e.g. Probabilistic 1% sampling). Highly efficient, but risks missing rare 5xx error traces. 2. Tail-Based Sampling: The OpenTelemetry Collector buffers all spans in memory until the entire trace completes. The Collector then evaluates rules: Retain 100% of traces containing HTTP 5xx errors or latency > 2000ms, while sampling only 0.1% of successful HTTP 200 responses.
Quick reference
- Head-Based sampling makes instant probabilistic decisions at request ingress to minimize overhead.
- Tail-Based sampling buffers full traces in OTel Collector memory before making sampling decisions.
- Tail-based rules retain 100% of error traces and high-latency outlier requests automatically.
- Reduces telemetry storage costs by 90%+ while preserving 100% of actionable incident data.
- Essential for high-scale microservice networks processing billions of requests per day.
Remember this
Implement Tail-Based Sampling in OTel Collector to capture 100% of error traces while controlling storage spend.
Key takeaway
To test Jaeger distributed tracing locally, run docker run -d --name jaeger -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one:latest. Send OTLP traces to localhost:4317 and view waterfalls at http://localhost:16686.
Related Articles
Explore this topic