The fragmentation problem in observability
Monitoring a distributed system requires three types of data: traces that follow a request across services, metrics that measure aggregate behaviour and logs that record point-in-time events. Until 2019, collecting this data means choosing between incompatible tools and formats. Each observability backend — Jaeger, Zipkin, Prometheus, commercial platforms — requires its own instrumentation library. Switching backends means rewriting collection code. The result is a technical vendor lock-in that constrains infrastructure decisions for years.
Two open source projects tackle the problem from different angles: OpenTracing, incubated by the CNCF, defines standard APIs for distributed tracing; OpenCensus, born at Google, provides libraries for traces and metrics with export to multiple backends. Both gain adoption, but the coexistence of two partial standards creates further fragmentation.
The merger into OpenTelemetry
In May 2019 the two projects announce their merger into OpenTelemetry, a single project under the Cloud Native Computing Foundation umbrella. The goal is to define an open standard for collecting observability data that is independent of backend, language and platform.
The fundamental principle is: instrument once, export anywhere. An application instrumented with OpenTelemetry can send its data to any compatible backend without code changes.
Architecture: SDKs, Collector and OTLP
OpenTelemetry’s architecture spans three layers. The multi-language SDKs — available for Java, Python, Go, JavaScript, .NET, C++, Rust and others — provide APIs to create spans (units of work in tracing), record metrics and emit logs. Instrumentation can be manual or automatic, with libraries that intercept frameworks and HTTP clients without changes to application code.
The Collector is a standalone component that receives, processes and forwards telemetry data. It works as a configurable intermediary: it can filter, enrich, aggregate and route data to one or more backends. The pipeline architecture — receiver, processor, exporter — makes the Collector extensible through plugins.
The transport protocol is OTLP (OpenTelemetry Protocol), designed to transmit traces, metrics and logs in a unified format over gRPC or HTTP. OTLP is the contract between SDKs and Collector, and between Collector and backend.
Observability as infrastructure
OpenTelemetry transforms observability from a product choice into standard infrastructure. Organisations can adopt a single instrumentation library knowing that the collected data will be compatible with any analysis platform, today and in the future. For distributed systems, where understanding behaviour across dozens of services is an operational necessity, an open standard reduces adoption costs and eliminates lock-in to a single vendor.
Link: opentelemetry.io