Last year we turned tracing on everywhere. This year the telemetry invoice is a line item a manager asks about by name. The dashboards did not get more useful. More data, same confusion at 2 a.m.
Time to design instead of collect.
The design starts from the questions I actually ask when the pager goes off. In practice there are four: is the latency of key endpoints normal, did the error rate jump, how far behind are the queues, is the database saturated. That is the core set. Metrics for these must be cheap, always on, with alerts. Everything else can be sampled traces I pull up on demand.
The main cost is cardinality. Someone puts a user id into a metric label and now you pay for one time series per user. Forever. Per-user detail belongs in traces, and traces belong under sampling: head sampling low and boring, tail sampling for errors and slow requests if the pipeline can do it.
Same discipline for logs. A log line is a set of fields you will filter by. Request id, route, status, duration, tenant. And no personal data in spans or logs at all. Nobody audits telemetry storage the way they audit the main database, so keep it the least interesting one.
The rule we adopted: every dashboard panel must answer a question that was asked in a real incident. Once a quarter we delete the panels that answer nothing. A metric nobody queries is a log with worse compression.
The user id label was mine. Added during an incident, marked temporary. It outlived the incident by a year.