A request creates a job. The job calls an external API and writes to the database. Something in this chain is slow, and the logs tell four disconnected stories.
Grepping a request id across services is our folklore. OpenTelemetry finally makes the boring standard version practical in PHP.
The core is small. One trace id for the whole causal chain, a span id for every operation, and one rule: pass the context along. Over HTTP it is a single header:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
Incoming middleware reads it and opens a server span. Every outgoing HTTP call opens a child span and sends the header further. Nothing new so far, Zipkin did this years ago. The part people forget is the queue. A job is not a new story. It was caused by a request. So the producer writes the trace context into message metadata, next to the payload, and the worker restores it before running the handler. Now the slow API call inside the job hangs from the same trace as the click that caused it. That is the whole point: causality that survives process boundaries.
Add SQL spans and the waterfall is complete. On one project the first full trace paid for the setup in a day. The “slow endpoint” was a job doing three retries against a dead cache, invisible in request logs.
Two warnings.
Sampling. You do not need every trace, you need enough of them plus all the errors. Decide the percentage early, exporters are not free.
Attributes go to a backend that stores and indexes them. Statement text is fine, bound values are not. No emails, no tokens, and keep attribute values low cardinality, or the tracing backend becomes the most expensive database you run.
We had the request id in the logs for three years before this. I never once managed to follow it into a job.