Observability

Last year we turned tracing on everywhere. This year the telemetry invoice is a line item a manager asks about by name. The dashboards did not get more useful. More data, same confusion at 2 a.m. Time to design instead of collect. The design starts from the questions I actually ask when the pager goes off. In practice there are four: is the latency of key endpoints normal, did the error rate jump, how far behind are the queues, is the database saturated. That is the core set. Metrics for these must be cheap, always on, with alerts. Everything else can be sampled traces I pull up on demand. ...

A request creates a job. The job calls an external API and writes to the database. Something in this chain is slow, and the logs tell four disconnected stories. Grepping a request id across services is our folklore. OpenTelemetry finally makes the boring standard version practical in PHP. The core is small. One trace id for the whole causal chain, a span id for every operation, and one rule: pass the context along. Over HTTP it is a single header: ...