Dev Note

The Observability Wars Are Over. OTel Won.
Three years ago, the observability space was a warzone. Datadog vs New Relic vs Dynatrace. OpenTracing vs OpenCensus vs vendor-specific agents. Every company had a different stack, and switching vendors meant re-instrumenting everything.
Then OpenTelemetry (OTel) hit v1.0 for traces, then metrics, then logs. Vendors fell in line. By 2026, OTel is the default instrumentation standard for any serious distributed system.
The benefits compound:
Instrument once, export anywhere
Vendor-neutral data format
Massive ecosystem of auto-instrumentation
CNCF-backed, not going anywhere
This post covers what you need to know to use OTel effectively in production today.
Photo by Carlos Muza on Unsplash
The Three Pillars, Unified
OpenTelemetry covers all three observability signals:
Signal
      Status
      What It Covers
    
Traces
      Stable (v1.0)
      Request flow across services
    
Metrics
      Stable (v1.0)
      Measurements over time
    
Logs
      Stable (v1.0)
      Discrete events with context
    
Profiles
      Beta
      CPU/memory profiling
    
The magic is correlation: a trace can link to the logs emitted during that trace, and to the metrics that spiked during that request. One unified data model.
The OTel Architecture
Your Application
     │
     ▼
[OTel SDK] ← auto-instrumentation + manual instrumentation
     │
     ▼
[OTel Collector] ← receive, process, export
     │
     ├──► Jaeger (traces)
     ├──► Prometheus (metrics)
     ├──► Loki (logs)
     ├──► Datadog
     ├──► Honeycomb
     └──► Any OTLP-compatible backend
The OTel Collector is the key piece: a vendor-neutral proxy that receives telemetry from your apps, processes/enriches it, and forwards to any backend. Change backends by updating collector config — zero code changes.
Getting Started: Auto-Instrumentation
The fastest path to value: zero-code auto-instrumentation.
Python (FastAPI example)
pip install opentelemetry-sdk opentelemetry-instrumentation-fastapi opentelemetry-exporter-otlp
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
# Configure SDK
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)
# One line to instrument FastAPI
FastAPIInstrumentor.instrument_app(app)
Every HTTP request now gets a trace automatically. Incoming/outgoing calls, DB queries, Redis operations — all captured without writing a single span manually.
Node.js
// otel.js - load before your app
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
# Start with auto-instrumentation
node --require ./otel.js app.js
Java (Spring Boot)
# Download the OTel Java agent
curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -o otel-agent.jar
# Run with agent - zero code changes needed
java -javaagent:otel-agent.jar \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -Dotel.service.name=payment-service \
  -jar app.jar
The Java agent instruments 80+ libraries automatically: Spring, Hibernate, gRPC, Kafka, Redis, and more.
Manual Instrumentation: Adding Business Context
Auto-instrumentation captures infrastructure. Manual spans add business context.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
async def process_payment(order_id: str, amount: float):
    with tracer.start_as_current_span("process_payment") as span:
        # Add business attributes to the span
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.currency", "USD")
        
        try:
            result = await charge_card(order_id, amount)
            span.set_attribute("payment.status", "success")
            span.set_attribute("payment.transaction_id", result.transaction_id)
            return result
            
        except PaymentDeclinedException as e:
            span.set_attribute("payment.status", "declined")
            span.set_attribute("error.type", "payment_declined")
            span.record_exception(e)
            raise
Now when you search for a specific order.id in Jaeger/Honeycomb, you see the entire trace with business context. Debug a specific customer’s failed payment in seconds, not hours.
The OTel Collector: Your Observability Control Plane
The Collector is where the real power lives.
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  # Add resource attributes to all telemetry
  resource:
    attributes:
    - key: deployment.environment
      value: production
      action: insert
  
  # Filter out noisy health check traces
  filter:
    traces:
      span:
        - 'attributes["http.route"] == "/health"'
  
  # Probabilistic sampling - keep 10% of low-value traces
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 10
  
  # Batch for efficiency
  batch:
    timeout: 5s
    send_batch_size: 1000
exporters:
  # Export to multiple backends simultaneously
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  
  prometheus:
    endpoint: "0.0.0.0:8889"
  
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${DD_API_KEY}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, filter, probabilistic_sampler, batch]
      exporters: [otlp/jaeger, otlp/datadog]
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheus]
The Collector handles sampling, filtering, enrichment, and fanout — all without touching your application code.
Sampling Strategies in Production
At scale, you can’t store every trace. Sampling is essential.
Head-Based Sampling (Simple)
Decide at trace start whether to sample. Fast, low overhead.
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
# Sample 5% of all traces
sampler = TraceIdRatioBased(0.05)
provider = TracerProvider(sampler=sampler)
Problem: You sample 5% of normal traces and 5% of error traces. You might miss important errors.
Tail-Based Sampling (Smart)
Keep all interesting traces; sample boring ones.
# Collector tail sampling processor
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
    - name: errors-policy
      type: status_code
      status_code: {status_codes: [ERROR]}  # Keep all errors
    
    - name: slow-traces-policy
      type: latency
      latency: {threshold_ms: 1000}  # Keep traces > 1s
    
    - name: sample-rest
      type: probabilistic
      probabilistic: {sampling_percentage: 2}  # Sample 2% of the rest
Tail-based sampling requires the Collector to buffer spans until the trace completes. More complex, but far more valuable — you never lose an error trace.
Connecting Traces to Logs
The killer feature of OTel in 2026: correlated logs.
import logging
from opentelemetry import trace
class OtelLogHandler(logging.Handler):
    def emit(self, record):
        span = trace.get_current_span()
        if span.is_recording():
            ctx = span.get_span_context()
            record.trace_id = format(ctx.trace_id, '032x')
            record.span_id = format(ctx.span_id, '016x')
        
        # Your existing log handling
        super().emit(record)
Now every log line contains the trace ID. In Grafana/Loki, you can click from a trace span directly to the logs from that span, in the same time window.
The debugging flow becomes:
Alert fires → find the trace
Trace shows which service is slow → click to logs
Logs show the exception → fix the bug
Kubernetes: OTel Operator
For Kubernetes deployments, the OTel Operator handles injection automatically:
# Install operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
# Annotate namespace for auto-injection
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    instrumentation.opentelemetry.io/inject-python: "true"
    instrumentation.opentelemetry.io/inject-nodejs: "true"
Every pod in the annotated namespace gets auto-instrumented. No Dockerfile changes. No app code changes. The operator injects the OTel SDK via init containers.
The State of OTel Backends in 2026
Backend
      Best For
      OTel Support
    
Grafana Stack (Tempo+Loki+Mimir)
      Self-hosted, unified
      Native OTLP
    
Honeycomb
      Developer-friendly, high-cardinality
      Native OTLP
    
Jaeger
      Open-source traces
      Native OTLP
    
Datadog
      Enterprise, AI-powered
      OTLP + agent
    
Signoz
      Open-source full stack
      Native OTLP
    
Clickhouse
      DIY, cheap at scale
      Via collector
    
The open-source Grafana stack (Loki + Tempo + Mimir + Grafana) is the dominant self-hosted choice. For managed observability without vendor lock-in, Honeycomb has the best developer experience.
Conclusion
OpenTelemetry is no longer optional for distributed systems. It’s the foundation of modern observability.
The adoption path is clear: start with auto-instrumentation (zero code changes), add manual spans for business context, deploy the Collector for vendor flexibility, implement tail-based sampling at scale.
The promise of OTel — instrument once, choose your backend freely — is fully delivered in 2026. The only question is how quickly you adopt it.
The alternative is vendor lock-in and re-instrumentation every time you want to evaluate a new observability tool. The industry made its choice. Now it’s yours.

OpenTelemetry in 2026: The Observability Standard That Finally Unified the Industry

댓글