Day 43 of 100
The Trace Context Was Lost
🔥 Problem Statement
A request flows through three services — A calls B directly, B hands off work to C asynchronously through a message queue rather than a direct call. Tracing shows a clean, complete picture for the A-to-B hop: proper parent-child spans, correct timing, everything as expected. Then the trace simply ends. Whatever C does in response to that message shows up as a completely separate, disconnected trace, with no link back to the original request.
🏗️ Environment
- Request path: Service A -> Service B (async queue) -> Service C
- Tracing: instrumented on all three services
- Trace behavior: complete for A and B, breaks before C
- B to C communication: via a message queue, not direct HTTP
🤔 Your Challenge
Why would trace context specifically break at the point where communication switches from direct HTTP to an asynchronous queue?
- How does trace context normally travel between two services in a direct HTTP call?
- Does a message queue automatically carry the same kind of context that HTTP headers do?
- What would need to happen, specifically, for context to survive a hop through an async queue?
- Why would this same instrumentation work fine for direct calls but not for queue-based ones?
Solution Hidden
Think through the problem yourself before looking at the answer.
💡Solution
Step 1 — Understand the symptoms
Trace context — the identifiers that let a downstream span know which trace and which parent span it belongs to — travels via HTTP headers in a typical direct service-to-service call, and most tracing instrumentation automatically injects and extracts those headers without any custom code. A message queue has no equivalent built-in mechanism by default; a message's headers or attributes aren't automatically populated with trace context just because the publishing service happens to be inside an active trace. Unless something explicitly puts that context into the message and something on the consuming side explicitly reads it back out, the link between publish and consume simply doesn't exist.
Step 2 — Identify the likely bottleneck
The A-to-B hop working perfectly confirms the tracing instrumentation itself is functioning correctly for HTTP — the SDK, the exporter, the collector, all fine. The B-to-C hop breaking specifically at the queue boundary points at one particular gap: propagation was never implemented for the async messaging path. This is an extremely common gap precisely because most tracing libraries handle HTTP propagation automatically, giving a false sense that "tracing is set up" everywhere, when in practice it silently stops at any communication path that isn't a direct, instrumented HTTP call.
Step 3 — Investigation
- Check the message payload or headers being published by Service B for any trace context fields — if none exist, propagation was never implemented for this path, which is close to confirming the cause directly.
- Check Service C's message-consuming code for any attempt to extract trace context from the incoming message — even if B did inject it, C also needs to actively read it and use it to continue the same trace, rather than automatically starting a new, disconnected one.
- Check whether the messaging library or platform in use has any OpenTelemetry-specific instrumentation available — some popular queue/broker clients have semi-automatic context propagation support that simply wasn't enabled here.
- Confirm this is consistent across every message, not intermittent — a consistent, total break at this specific hop (rather than an occasional one) supports "propagation was never built for this path" over a flakier, harder-to-reproduce cause.
Step 4 — Recommended action
Implement explicit context propagation across the queue boundary: when Service B publishes a message, inject the current trace context into the message's headers or attributes (OpenTelemetry provides standard propagators for exactly this purpose); when Service C consumes a message, extract that context and use it to start its own span as a child of the original trace rather than a new, disconnected root. Many OpenTelemetry messaging instrumentation libraries handle this automatically once enabled — worth checking before writing this by hand.
Once implemented, confirm with a real end-to-end trace that a single trace ID now spans all three services, including the async hop, not just the two directly-connected ones.
Step 5 — Engineering lesson
Distributed tracing is only as complete as its weakest propagation path, and asynchronous communication — queues, pub/sub, event streams — is one of the most common places that path breaks, precisely because it doesn't get the same automatic handling that direct HTTP calls do. "Tracing is instrumented" across a system can be true while still leaving meaningful gaps at exactly the boundaries — async messaging chief among them — where context doesn't propagate on its own.
🧠 Today’s Takeaway
A distributed trace is only useful when context travels with the request.
🔔 Don’t Miss Tomorrow’s Challenge
A new real-world engineering challenge is released every day.
100 Days → 100 Challenges → 100 New Things Learned.
Get the useful stuff, not the noise.
Occasional notes on engineering, Platform Engineering, AI, cloud and things I’m learning along the way.

