Distributed Tracing — how you find which service ate the 3 seconds
Distributed tracing mints one trace ID at the edge, propagates it through every service, and reassembles spans into a waterfall to show exactly which hop ate the latency. Compare: logs alone vs. traces with parent pointers.
A checkout click touches five services. The page takes 3.1 seconds. You check the logs of each service separately and see nothing obviously wrong. Which one is slow?
Without distributed tracing, you are guessing. With it, you see that 2.9 of those 3.1 seconds happened inside a single inventory query, and now you know where to look.
Mental model: A trace ID is a thread that runs through every service, and each service records its own span (start time, end time, parent span) so a collector can stitch them into a waterfall and show you the exact nested breakdown of latency.
The problem: logs are local, requests are global
A log line is just one service writing to its own file, on its own clock. When you run a checkout, that request doesn't log itself — it gets logged five times, once per service, in five different places:
- Service A (checkout):
[2024-01-15T10:23:45.001Z] Order created for order_id=12345 - Service B (auth):
[2024-01-15T10:23:45.031Z] Token validated - Service C (inventory):
[2024-01-15T10:23:45.061Z] Stock check started - Service C (inventory):
[2024-01-15T10:23:47.961Z] Stock check complete - Service D (payment):
[2024-01-15T10:23:47.990Z] Charge processed
You can grep for order_id=12345 and find those five lines. But now what? The times tell you the request took 2.96 seconds total, but which service caused the delay? Service C's inventory check is clearly slow — it started at 45ms and finished at 2.96s — but the logs don't tell you whether inventory was slow because of something it did, or because it was waiting on something else, or because it was the bottleneck that made everything else wait.
You also can't see the nesting: did checkout call auth, or auth call inventory? The logs don't answer that. You have to know your architecture by hand.
Five services is manageable. Fifty services, and you are lost.
The mechanism: trace ID → propagation → spans → waterfall
Distributed tracing solves this by following one principle: one request, one identifier, threaded through every hop.
Step 1: The edge mints a trace ID
When a user clicks checkout, a load balancer or API gateway generates a single, universally unique identifier — the trace ID. Let's call it trace_id=abc123xyz. This ID is born once and never changes.
User click → Load balancer generates trace_id=abc123xyz
Step 2: Every service propagates it via a header
Service A (checkout) receives the request with a header:
GET /checkout HTTP/1.1
X-Trace-ID: abc123xyz
When checkout calls auth, it does not invent a new trace ID. It passes the same one:
POST /validate HTTP/1.1
X-Trace-ID: abc123xyz
Auth passes it to inventory. Inventory passes it to payment. The same ID flows through every hop, unchanged, like a bloodstream carrying a dye marker through an organism.
Step 3: Each hop records its own span
While propagating the trace ID, each service also records its own span — a record that says: "I started at time T1, finished at time T2, and my parent span was S_parent."
For example, checkout might record:
Span {
span_id: "span_001",
trace_id: "abc123xyz",
parent_span_id: null, // checkout is the root
service: "checkout",
start_time: 10:23:45.001Z,
end_time: 10:23:45.041Z,
duration_ms: 40
}
Auth records:
Span {
span_id: "span_002",
trace_id: "abc123xyz",
parent_span_id: "span_001", // auth was called by checkout
service: "auth",
start_time: 10:23:45.031Z,
end_time: 10:23:45.061Z,
duration_ms: 30
}
Inventory records:
Span {
span_id: "span_003",
trace_id: "abc123xyz",
parent_span_id: "span_001", // inventory was also called by checkout
service: "inventory",
start_time: 10:23:45.061Z,
end_time: 10:23:47.961Z,
duration_ms: 2900
}
Step 4: A collector reassembles them into a waterfall
All those spans get sent to a central collector (Jaeger, Zipkin, or an OpenTelemetry backend). The collector sorts them by trace ID, then uses the parent_span_id pointers to build a tree:
Trace: abc123xyz
├── [40ms] checkout (span_001)
│ ├── [30ms] auth (span_002, parent: span_001)
│ └── [2900ms] inventory (span_003, parent: span_001)
└── [remaining service calls...]
And there it is: a waterfall. One nested bar per service. The depth shows the call tree. The width shows the duration. The red bar is inventory at 2900ms.
The payoff: latency attribution
Before tracing, the request looked like this in your monitoring:
Checkout endpoint: 3.1 seconds (slow alert triggered)
You have no idea where those 3.1 seconds went. You check each service's CPU, memory, database queries — and they all look normal. At 2 a.m., you page your most experienced engineer, and they guess based on experience.
With distributed tracing, you see:
Trace for order_id=12345 (total: 3.1s)
├── checkout: 40ms
├── auth: 30ms
└── inventory: 2900ms ← THIS
You didn't have a slow system. You had one slow call. You now know exactly which service, which endpoint, and which query to investigate. You go straight to the inventory database, find the missing index, and the problem is solved before your engineer finishes their coffee.
How traces get lost: dropped headers and sampling
Distributed tracing is powerful, but it has two honest failure modes.
Dropped headers
If a developer writes a new service and forgets to read the X-Trace-ID header or fails to propagate it downstream, the trace breaks at that point.
Trace abc123xyz starts:
Checkout → Auth → [Header lost here]
↓
Inventory records trace_id=null
Now the inventory span is orphaned. The collector has no way to tie it to the original trace. The waterfall has a gap, and you lose visibility downstream.
Sampling
At scale, you cannot afford to store and process every single trace. If you run a million requests per second, storing all of them would overwhelm your backend. Instead, you sample: maybe keep 1 in 100 traces, or 1 in 10,000.
The trade-off is obvious: the one request you actually want to debug might be the one thrown away. You will never see the trace for the exact user who complained at 3 a.m.
Modern tracing systems (like OpenTelemetry) use adaptive sampling: they keep more traces if the request is slow or errored, and fewer if it succeeded quickly. But sampling is still a tax.
Traces in the context of microservices and async messaging
If you are still on a monolith, you do not need distributed tracing. One process, one request context, one log stream — you can usually follow the call stack or use a profiler.
Distributed tracing is essential once a request crosses process boundaries. That happens in microservices (multiple services over HTTP/gRPC), and it also happens in any asynchronous system: if Service A writes a message to a queue that Service B consumes later, the same trace ID should flow through the message header so B knows it came from A.
If you are using Kafka or RabbitMQ, you propagate the trace ID in the message headers. Same idea. The queue is just another hop.
Tools: Jaeger, Zipkin, and OpenTelemetry
Three major open-source projects implement this:
| Tool | Role | Strengths |
|---|---|---|
| OpenTelemetry | Instrumentation standard | Vendor-neutral, works with any backend. You add one SDK to your code, and it can send traces to Jaeger, Zipkin, Datadog, or any other backend. |
| Jaeger | Trace backend + UI | CNCF graduated project. Stores and queries traces. Fast, scalable, good UI for exploring waterfalls. Popular in Kubernetes environments. |
| Zipkin | Trace backend + UI | Older, battle-tested. Simpler than Jaeger. Good for getting started, but less feature-rich. Solid choice if you have small teams or lower trace volume. |
Most teams use OpenTelemetry to instrument their code, then pick a backend (Jaeger, Zipkin, or a managed service like Datadog, New Relic, or Honeycomb) to store and query the traces.
Reframing for LLM inference pipelines
If you are building an LLM inference system with multiple stages (embedding → retrieval → ranking → generation), distributed tracing maps directly to the inference pipeline.
Each stage is a "span": embedding takes 50ms, retrieval takes 200ms, ranking takes 100ms, generation takes 2000ms. If a user complains that inference is slow, the trace waterfall shows you instantly that generation is the bottleneck. You can then decide whether to optimize the model, cache outputs, or use a smaller model for that stage.
Batch inference complicates this: if you batch 100 requests together, each one still gets its own trace ID, and the waterfall shows you the queueing delay, the batch assembly time, and the per-token latency. Tracing is how you see the difference between "my model is slow" and "my batching strategy is causing high latency."
Verdict
Reach for distributed tracing when a single request crosses multiple services or processes. Without it, you are flying blind at latency. With it, you have a map. The cost is modest: a few lines of SDK initialization, header propagation in your RPC client, and a backend to store the traces. The payoff is eliminating 2 a.m. guessing games.
Watch the 90-second reel for this concept in action.