telemetry schemas are production api contracts
OpenTelemetry semantic conventions make telemetry more portable. They also make metric names, attributes, and log fields production contracts that need compatibility management.
You can break production confidence without breaking production traffic.
Rename a Kubernetes attribute. Move a resource field. Change a label’s meaning just enough that an old query no longer matches. The application keeps serving requests. Pods keep running. Deployments stay green.
Meanwhile, the burn-rate alert goes quiet, the cost report groups a workload under “unknown”, and the incident dashboard cheerfully tells everyone there is nothing to see.
That is not a dashboard problem.
It is an API compatibility problem.
stable is good news. stable is also a promise.
The OpenTelemetry Kubernetes attributes processor reaching v1.0.0 is good news. Stable semantic conventions make telemetry more portable, collector configuration less weird, and vendor migrations less dependent on folklore.
But stability has a consequence that is easy to miss when we are busy celebrating standards: it turns telemetry fields into interfaces other systems are allowed to depend on.
The old mental model was that metrics, logs, and traces were debugging exhaust. Useful during an incident, maybe, but not important enough to deserve the same care as a public API.
That model is now wildly out of date.
An attribute such as k8s.namespace.name, a metric label such as service, or a log field carrying an environment name may be read by far more than Grafana:
- alert rules and SLO burn-rate queries
- incident timelines and routing automation
- autoscalers and release gates
- chargeback and FinOps reports
- audit and compliance exports
- anomaly detection and AI incident assistants
- the exhausted engineer trying to answer “what changed?” at 03:17
If any of those consumers stop understanding the field, your system has lost operational capability. The request path may be fine. Your ability to operate it is not.
the collector is not the whole contract
It is tempting to treat a collector upgrade as plumbing. Update the image, adjust a config file, watch it restart, move on.
That works only when the collector is the sole consumer of its own output, which is almost never true.
Telemetry crosses boundaries. An OpenTelemetry Collector may receive Prometheus metrics, enrich them with Kubernetes metadata, relabel them, export them to a managed backend, and leave a trail of dashboards, recording rules, and automation depending on the result. The recent Prometheus and OpenTelemetry interoperability work is useful precisely because real teams run these mixed pipelines. They are not waiting for a clean replacement day that will never arrive.
So a schema change can have a long blast radius:
1
instrumentation -> collector -> enrichment -> storage -> queries -> decisions
The dangerous changes are often small enough to look harmless in a pull request:
k8s.pod.namebecomes a differently named or scoped attribute- a custom
teamlabel disappears during relabeling service.nameis populated differently after a library upgrade- a log field moves from body text into structured attributes
- a dimension gains an unbounded value and turns a useful metric into a small billing emergency
None of this is an argument against semantic conventions. It is an argument for taking them seriously.
schema migrations, not dashboard cleanup
Elastic’s migration guidance around the Kubernetes semantic-convention change is the practical version of this story: custom queries that relied on older attribute names need to be migrated, and dual emission gives users time to move dashboards and alerts safely.
That is the correct instinct. Treat a telemetry schema migration like an API migration.
If you rename a field in a public HTTP response, you do not deploy it on Friday, shrug, and tell consumers to update their parsers when they notice. At least, you should not. You publish the change, preserve compatibility where possible, measure adoption, give people a deadline, and keep a way back.
Telemetry deserves the same routine:
- Name an owner. Someone owns the semantic contract, not merely the collector deployment.
- Declare the change. Say which field changes, why, which consumers are affected, and when the old form disappears.
- Dual-emit deliberately. Produce old and new fields for a defined window where storage cost and cardinality allow it.
- Migrate the consumers. Alerts, recording rules, dashboards, SLOs, exports, and automations are code. Change them and review them.
- Prove the new path. Test the important queries against representative production-shaped data before deprecating the old field.
- Keep rollback boring. The previous collector configuration and its operating instructions should be near the migration, not hidden in somebody’s browser history.
This is not bureaucracy added to observability. This is the missing part of operating an observability system people actually trust.
the hidden dependency graph is growing
OpenTelemetry’s growing maturity across metrics, traces, and logs makes this more important, not less. The Go Logs API and SDK reaching release-candidate status is another sign that logs are moving into a world of structured, interoperable data rather than remaining an ungoverned pile of strings.
That is excellent. I would like fewer bespoke parsers and fewer dashboards held together by regular expressions and personal optimism.
But a structured field is easier to consume. Which means more things will consume it.
The moment a release gate asks whether error rate is safe in environment=production, that value has a consumer. The moment a FinOps report allocates cost by team, that label has a consumer. The moment an incident assistant uses k8s.cluster.name to assemble a timeline, that attribute has a consumer.
You do not need a giant platform program to acknowledge this. You need to stop pretending that an attribute with fifteen downstream consumers is “just metadata.”
monday morning checklist
If I were planning a collector or semantic-convention upgrade, I would start here:
- Inventory high-value consumers. Search alert rules, recording rules, dashboards, SLO definitions, automation, exports, and saved queries for the field being changed.
- Write a small schema changelog. Include old name, new name, owner, compatibility window, and removal date. A Markdown file beside the collector config is already better than memory.
- Dual-emit with a deadline. Compatibility without an expiry becomes permanent clutter. Set the removal date when the old field is introduced, then revisit it with evidence.
- Test operational outcomes. Check that the alert fires, the burn-rate query returns data, the deployment gate behaves, and the cost grouping still makes sense. A pretty dashboard alone is not proof.
- Put cardinality in the design review. Every new dimension should answer: what values can it take, what limits it, and who pays when it grows?
- Assign owners to custom dimensions.
team,product,tenant_tier, and similar fields are useful only when their meaning does not drift quietly across services. - Keep a rollback path with the config. Include the last known-good collector version, the reversal change, and the few queries that prove recovery.
The checklist is deliberately unglamorous. So are seat belts. Both are more appreciated after the sudden stop.
standards do not replace governance
The nice story about OpenTelemetry is that standards will make observability portable and less vendor-shaped. I think that story is mostly true, and it is worth building toward.
The lazy version of the story is that standards remove the need for governance.
They do not.
They make the hidden contracts visible enough to govern.
That is progress. A metric name, resource attribute, label, or structured log field is now something we can document, test, version, migrate, and own. The work is not glamorous, but it is much cheaper than finding out during an incident that the system was healthy while the only trustworthy alarm had quietly stopped speaking your language.
references
- OpenTelemetry: Kubernetes attributes processor reaches v1.0.0
- Elastic Observability Labs: Kubernetes semantic conventions v1 and EDOT Collector migration
- OpenTelemetry: Prometheus and OpenTelemetry interoperability survey results
- OpenTelemetry: Go Logs API and SDK reach release candidate status
- CNCF: Observability Day at KubeCon + CloudNativeCon North America 2026
To test my projects, I use Railway. If you want $20 USD to get started, use this link.