Dev Note

The Observability Dark Ages
Not long ago, observability was vendor chaos. You instrumented with Datadog’s SDK, then migrated to New Relic, and your instrumentation was worthless. You adopted Jaeger for tracing but Zipkin for another service. Metrics lived in Prometheus, logs in ELK, traces somewhere else entirely. Three pillars, three silos, three times the toil.
OpenTelemetry changed everything. In 2026, it’s not just a CNCF project — it’s the universal language of observability. Every major vendor supports it. Every major framework auto-instruments with it. It’s the TCP/IP of distributed system visibility.
Photo by Luke Chesser on Unsplash
What OpenTelemetry Actually Is
OpenTelemetry (OTel) is an open-source framework for generating, collecting, and exporting telemetry data — traces, metrics, and logs — in a vendor-neutral way.
The key components:
┌─────────────────────────────────────────────────────┐
│                  Your Application                    │
│                                                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │
│  │    Traces   │  │   Metrics   │  │    Logs     │ │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘ │
│         └────────────────┴─────────────────┘        │
│                          │                          │
│              ┌────────────▼────────────┐            │
│              │   OTel SDK / API        │            │
│              └────────────┬────────────┘            │
└───────────────────────────┼─────────────────────────┘
                            │ OTLP
              ┌─────────────▼─────────────┐
              │   OTel Collector          │
              │  (receive/process/export) │
              └─────────────┬─────────────┘
                            │
           ┌────────────────┼────────────────┐
           ▼                ▼                ▼
      Datadog          Grafana           Honeycomb
      New Relic         Stack            Jaeger
      (any vendor)     (self-hosted)     (any backend)
Instrument once. Export anywhere.
The Three Pillars, Unified
Traces: The Whole Journey
A trace follows a request through your entire system:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Setup (usually done once at startup)
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)
# Usage
tracer = trace.get_tracer(__name__)
async def process_order(order_id: str):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.source", "api")
        
        # Child spans created in called functions automatically 
        # attach to this parent span
        inventory = await check_inventory(order_id)
        payment = await process_payment(order_id)
        
        span.set_attribute("order.status", "completed")
        return {"status": "ok", "order_id": order_id}
Metrics: The Numbers That Matter
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
# Counter: monotonically increasing
request_counter = meter.create_counter(
    name="http.server.requests",
    description="Total HTTP requests",
    unit="1"
)
# Histogram: distribution of values
request_duration = meter.create_histogram(
    name="http.server.duration",
    description="HTTP request duration",
    unit="ms"
)
# Gauge: current value (can go up or down)
active_connections = meter.create_up_down_counter(
    name="http.server.active_connections",
    description="Active HTTP connections",
    unit="1"
)
# Usage in request handler
def handle_request(method: str, path: str):
    start = time.time()
    active_connections.add(1, {"method": method})
    
    try:
        result = process_request(method, path)
        request_counter.add(1, {"method": method, "status": "200"})
        return result
    finally:
        duration_ms = (time.time() - start) * 1000
        request_duration.record(duration_ms, {"method": method, "path": path})
        active_connections.add(-1, {"method": method})
Logs: Correlated with Everything Else
The 2026 breakthrough: logs that know their trace context:
import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor
# Auto-inject trace context into all log records
LoggingInstrumentor().instrument(set_logging_format=True)
logger = logging.getLogger(__name__)
async def process_payment(order_id: str, amount: float):
    # This log automatically includes trace_id and span_id
    logger.info(
        "Processing payment",
        extra={"order_id": order_id, "amount": amount}
    )
    # Output: [2026-04-26T12:00:00Z] INFO trace_id=abc123 span_id=def456 
    #         Processing payment order_id=ORD-789 amount=99.99
Now your logs, traces, and metrics are correlated by trace ID. When something breaks, you click on the log, jump to the trace, see the metrics spike, all in one flow.
Auto-Instrumentation: No Code Required
The best OTel feature for adopting teams: auto-instrumentation adds observability without touching your code:
# Python: zero-code instrumentation
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
# Start your app with auto-instrumentation
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_SERVICE_NAME=payment-service \
opentelemetry-instrument python app.py
This automatically instruments:
FastAPI / Flask / Django HTTP requests
SQLAlchemy database queries
Redis operations
gRPC calls
boto3 AWS calls
And 100+ other libraries
# Kubernetes: inject auto-instrumentation via admission webhook
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://otel-collector:4318
  propagators:
    - tracecontext
    - baggage
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
---
# Annotate your Pod to opt in
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-python: "true"
Zero code changes. Full traces for your entire service.
The OTel Collector: Your Telemetry Pipeline
The Collector is the unsung hero — a vendor-neutral proxy that receives, processes, and exports telemetry:
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  
  # Also scrape existing Prometheus metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
processors:
  # Add k8s metadata to all telemetry
  k8sattributes:
    auth_type: "serviceAccount"
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
  
  # Drop high-cardinality noisy spans
  filter:
    spans:
      exclude:
        match_type: strict
        attributes:
          - key: http.url
            value: "/healthz"
  
  # Batch for efficiency
  batch:
    timeout: 1s
    send_batch_size: 1024
exporters:
  # Send to multiple backends simultaneously
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${DD_API_KEY}
  
  otlp/grafana:
    endpoint: https://otlp-gateway.grafana.net/otlp
    auth:
      authenticator: basicauth/grafana
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, filter, batch]
      exporters: [otlp/datadog, otlp/grafana]
    metrics:
      receivers: [otlp, prometheus]
      processors: [k8sattributes, batch]
      exporters: [otlp/grafana]
    logs:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [otlp/datadog]
One collector config, full observability. Change backends without touching your application code.
OTel + AI: The Emerging Frontier
In 2026, GenAI Semantic Conventions are being standardized. Trace your LLM calls:
from opentelemetry.semconv._incubating.attributes import gen_ai_attributes
with tracer.start_as_current_span("llm.chat") as span:
    span.set_attribute(gen_ai_attributes.GEN_AI_SYSTEM, "anthropic")
    span.set_attribute(gen_ai_attributes.GEN_AI_REQUEST_MODEL, "claude-3-7-sonnet")
    span.set_attribute(gen_ai_attributes.GEN_AI_REQUEST_MAX_TOKENS, 1024)
    
    response = anthropic_client.messages.create(
        model="claude-3-7-sonnet",
        messages=[{"role": "user", "content": prompt}]
    )
    
    span.set_attribute(gen_ai_attributes.GEN_AI_RESPONSE_MODEL, response.model)
    span.set_attribute(gen_ai_attributes.GEN_AI_USAGE_INPUT_TOKENS, response.usage.input_tokens)
    span.set_attribute(gen_ai_attributes.GEN_AI_USAGE_OUTPUT_TOKENS, response.usage.output_tokens)
Track token usage, latency, model versions, and costs per request — all correlated with the business logic that triggered them.
Vendor Landscape in 2026
Vendor
      OTel Support
      Differentiator
    
Grafana Stack
      Native
      Open-source, self-hostable, best value
    
Datadog
      Full
      Unified UX, AI insights, alerting
    
Honeycomb
      Native (pioneers)
      Best query UX, high cardinality
    
New Relic
      Full
      Generous free tier, broad coverage
    
Jaeger
      Native
      OSS, trace-focused, no cost
    
SigNoz
      Native
      OTel-first OSS alternative to Datadog
    
The emerging recommendation: OTel → Grafana Stack for most teams (Tempo for traces, Mimir for metrics, Loki for logs). Full observability at near-zero cost at reasonable scale.
Conclusion
OpenTelemetry has solved observability’s fragmentation problem. The investment in instrumenting your systems with OTel today is future-proof — you’re never locked into a vendor again.
The ecosystem in 2026 is mature, the auto-instrumentation removes most of the toil, and the Collector gives you a flexible pipeline to route telemetry wherever you need it.
If you’re still using vendor-specific SDKs, migration to OTel should be on your roadmap. When you eventually switch backends (and you will), you’ll thank yourself.
The OpenTelemetry Demo is the best way to see all the pieces working together in a realistic microservices environment.

OpenTelemetry in 2026: Observability Has Finally Grown Up

댓글