Distributed Tracing Pipeline: OpenTelemetry & Grafana Tempo
Debugging microservice latency spikes and error cascades across dozens of distributed services using traditional isolated log files is nearly impossible. When an edge HTTP request fails with a 500 error or takes 3 seconds to complete, engineering teams must trace the exact request path across API gateways, auth services, database queries, and async queues.
OpenTelemetry (OTel) provides a vendor-neutral observability framework for generating, collecting, and exporting trace telemetry. OpenTelemetry Collector processes OTLP (OpenTelemetry Protocol) trace spans in high-throughput pipelines, while Grafana Tempo delivers cost-effective object-storage trace persistence without expensive search indexing. This guide covers OTel W3C traceparent propagation, Collector pipeline configuration, tail-based sampling, and Grafana TraceQL querying.
Mental Model: Monolithic Application Logging vs Distributed Microservice Tracing
Distributed tracing tracks the complete lifecycle of a single incoming request across microservice boundaries using a unified Trace ID and nested Span IDs:
1. W3C Trace Context Propagation: The edge gateway injects HTTP header traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 into downstream service requests.
2. Span Graph Assembly: Each microservice creates child spans covering HTTP calls, gRPC requests, and database queries, reporting telemetry asynchronously. For OpenTelemetry Jaeger and GCP observability, review implementing distributed tracing opentelemetry jaeger and observability opentelemetry gcp.
Quick reference
- Trace IDs correlate logs, metrics, and spans across polyglot microservice boundaries.
- W3C traceparent headers propagate span context over HTTP, gRPC, and Kafka message headers.
- Asynchronous OTLP gRPC exporters prevent telemetry overhead from stalling application threads.
- Pinpoints exact bottleneck spans causing $p_{99}$ latency spikes in production.
- Powers distributed observability at Grafana Labs, Uber, Shopify, Slack, and CoreConcept.
Remember this
Inject W3C traceparent headers to correlate distributed trace spans across microservice boundaries.
OpenTelemetry Collector Architecture: Receivers, Processors, & Exporters
The OpenTelemetry Collector decouples application SDK instrumentation from backend telemetry storage:
1# otel-collector-config.yaml2receivers:3 otlp:4 protocols:5 grpc: { endpoint: 0.0.0.0:4317 }6 http: { endpoint: 0.0.0.0:4318 }7 8processors:9 memory_limiter:10 check_interval: 1s11 limit_percentage: 7512 batch:13 send_batch_size: 819214 timeout: 500ms15 16exporters:17 otlp/tempo:18 endpoint: tempo:431719 tls: { insecure: true }20 21service:22 pipelines:23 traces:24 receivers: [otlp]25 processors: [memory_limiter, batch]26 exporters: [otlp/tempo]Quick reference
- Receivers accept incoming telemetry in OTLP, Jaeger, Zipkin, or Prometheus formats.
- Processors filter, batch, scrub PII data, and limit memory usage in pipeline RAM.
- Exporters push processed trace batches to storage backends (Tempo, Jaeger, Cloud Trace).
- Batching processor reduces network overhead by grouping spans into high-speed OTLP payloads.
- Eliminates vendor lock-in by allowing instant backend storage target changes in YAML.
Remember this
Deploy OpenTelemetry Collector pipelines to batch, filter, and export trace data without application changes.
Tail-Based Sampling Strategies vs Head-Based Sampling
Sampling every single trace in high-throughput production environments generates petabytes of telemetry data, leading to massive storage costs. Sampling decisions must be made intelligently:
- Head-Based Sampling (SDK Level): Probabilistically samples 1% or 10% of requests at the edge SDK. Misses rare $p_{99.9}$ latency spikes and 500 internal server error traces that occurred in the unsampled 99%. - Tail-Based Sampling (Collector Level): Holds all trace spans in Collector memory buffers until the entire trace completes. Retains 100% of error traces and 100% of slow traces ($>500\text{ms}$), while sampling only 1% of fast HTTP 200 traces.
Quick reference
- Tail-based sampling guarantees 100% capture of production errors and latency outliers.
- Saves up to 90% in cloud storage costs by dropping repetitive successful fast traces.
- Requires routing all spans of a given Trace ID to the same Collector instance using load-balancing exporters.
- Configures memory_limiter processor to prevent OOM spikes during traffic surges.
- Ensures high-value diagnostic traces are never dropped.
Remember this
Implement Collector tail-based sampling to capture 100% of errors while reducing storage costs by 90%.
Grafana Tempo Trace Storage & TraceQL High-Speed Querying
Grafana Tempo is an open-source, high-scale distributed tracing backend designed for object storage (AWS S3, GCP GCS, Azure Blob):
- Index-Free Storage Architecture: Tempo does not build expensive search indexes for trace data. It writes compressed trace blocks directly to cloud object storage, drastically lowering storage costs. - TraceQL Query Engine: Enables searching traces by duration, span attributes, and status code:
1{ status = error && duration > 500ms && .http.status_code = 500 }Quick reference
- Index-free architecture reduces tracing backend storage costs by 10x compared to Elasticsearch.
- Stores raw compressed trace blocks natively in AWS S3 or GCP GCS bucket storage.
- TraceQL query engine filters traces by span duration, HTTP status code, and k8s metadata.
- Seamless integration with Grafana dashboards, Prometheus metrics, and Loki log lines.
- Delivers sub-second trace retrieval for production incident troubleshooting.
Remember this
Use Grafana Tempo and TraceQL to store petabytes of trace telemetry affordably in cloud object storage.
Key takeaway
To test OpenTelemetry Collector and Grafana Tempo locally, run docker-compose up -d with otel/opentelemetry-collector-contrib and grafana/tempo. Query traces in Grafana at http://localhost:3000.
Related Articles
Explore this topic