Dev Note

OpenTelemetry in 2026: The De Facto Standard for Distributed Observability
If you’re building distributed systems in 2026 and not using OpenTelemetry (OTel), you’re leaving significant operational capability on the table. What started as a CNCF project to unify distributed tracing has evolved into the comprehensive observability standard covering traces, metrics, logs, and (increasingly) profiling — all under a single, vendor-neutral API.
This post is a practical guide to OTel in production: what it is, how to instrument your services properly, and how to get real value from the data you’re collecting.
Photo by Luke Chesser on Unsplash
What OpenTelemetry Actually Provides
OTel has three main components:
The API — language-specific interfaces for instrumentation (what you call in your code)
The SDK — implementation of the API with configurable exporters and processors
The Collector — a standalone agent/gateway for receiving, processing, and exporting telemetry data
The key value proposition: instrument once, export anywhere. Whether your organization uses Jaeger, Tempo, Honeycomb, Datadog, Dynatrace, or builds its own stack — the instrumentation code doesn’t change.
The Four Signals (Now Including Profiling)
OTel has historically focused on three signals:
Signal
      What it tells you
      Examples
    
Traces
      The journey of a request across services
      Latency per hop, errors, service dependencies
    
Metrics
      Aggregated measurements over time
      Request rate, error rate, latency percentiles
    
Logs
      Discrete events with context
      Error messages, audit events, debug output
    
Profiles
      Code-level CPU/memory breakdown
      Which function is consuming 80% of CPU
    
In 2026, profiling support in OTel has reached stable maturity. Continuous profiling (via Parca, Pyroscope, or Grafana Beyla) is increasingly integrated into the OTel pipeline, giving you the full picture from business metric down to specific line of code.
Instrumentation: Auto vs. Manual
Auto-Instrumentation (Start Here)
For common frameworks, OTel provides zero-code instrumentation via agents or middleware:
Python (FastAPI + SQLAlchemy):
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
# Run with auto-instrumentation
opentelemetry-instrument \
  --service-name my-api \
  --exporter-otlp-endpoint http://otel-collector:4317 \
  uvicorn main:app
Node.js:
// tracing.js - loaded before your app
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const sdk = new NodeSDK({
  serviceName: 'order-service',
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317'
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': { enabled: true },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true },
    })
  ],
});
sdk.start();
Java (Spring Boot):
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=payment-service \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -jar app.jar
Auto-instrumentation gets you 80% of the value with near-zero code changes.
Manual Instrumentation (The Important 20%)
For business logic, custom operations, and meaningful context, manual instrumentation is essential:
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from opentelemetry.semconv.trace import SpanAttributes
tracer = trace.get_tracer(__name__, "1.0.0")
async def process_payment(payment_id: str, amount: float, user_id: str):
    with tracer.start_as_current_span("process_payment") as span:
        # Add semantic attributes
        span.set_attribute("payment.id", payment_id)
        span.set_attribute("payment.amount", amount)
        span.set_attribute("user.id", user_id)
        span.set_attribute(SpanAttributes.DB_SYSTEM, "postgresql")
        
        try:
            # Add span events for key moments
            span.add_event("payment_validation_started")
            await validate_payment(payment_id, amount)
            span.add_event("payment_validation_completed")
            
            result = await charge_card(payment_id, amount)
            
            # Record the outcome
            span.set_attribute("payment.status", result.status)
            span.set_attribute("payment.processor", result.processor)
            span.set_status(Status(StatusCode.OK))
            
            return result
            
        except PaymentDeclinedException as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.record_exception(e)
            span.set_attribute("payment.failure_reason", e.reason)
            raise
The OTel Collector: Your Telemetry Pipeline
The Collector is the heart of a production OTel deployment. It decouples instrumented services from specific backends:
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  # Add resource attributes (env, cluster, region)
  resourcedetection:
    detectors: [k8snode, env]
    timeout: 2s
  
  # Batch for efficiency
  batch:
    timeout: 1s
    send_batch_size: 1024
  
  # Tail-based sampling: keep 100% of errors, 1% of success traces
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces-policy
        type: latency
        latency: { threshold_ms: 1000 }
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }
exporters:
  # Tempo for traces
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  
  # Prometheus for metrics
  prometheus:
    endpoint: 0.0.0.0:8889
  
  # Loki for logs
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resourcedetection, batch, tail_sampling]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [resourcedetection, batch]
      exporters: [loki]
Tail-Based Sampling: Critical for High-Volume Services
Head-based sampling (make the decision at trace start) is simple but ineffective — you can’t know if a trace will be interesting when it starts. Tail-based sampling keeps the complete trace, then decides whether to store it:
Keep 100% of error traces
Keep 100% of traces exceeding your SLO threshold
Keep a small percentage of “normal” traces for baseline visibility
This dramatically reduces storage costs while ensuring you never lose an important trace.
Exemplars: Connecting Metrics to Traces
One of the most powerful OTel features in 2026 is exemplars — attaching specific trace IDs to metric data points. This lets you jump directly from a latency spike in a dashboard to the actual slow traces:
from opentelemetry import metrics, trace
from opentelemetry.sdk.metrics.export import Exemplar
meter = metrics.get_meter(__name__)
request_latency = meter.create_histogram(
    name="http.server.request.duration",
    description="HTTP request latency",
    unit="s",
)
async def handle_request(request):
    start = time.time()
    with tracer.start_as_current_span("http_request") as span:
        response = await process_request(request)
        
        duration = time.time() - start
        # The SDK automatically attaches the current trace ID as an exemplar
        request_latency.record(
            duration,
            {"http.method": request.method, "http.status_code": response.status_code}
        )
        
    return response
In Grafana, clicking on a latency spike shows you the exemplar trace IDs, which you can jump to directly in Tempo. This closes the loop between metrics and traces.
Semantic Conventions: The Most Underused Feature
OTel defines semantic conventions — standardized attribute names for common operations. Using them makes your telemetry interoperable across tools:
from opentelemetry.semconv.trace import SpanAttributes
# Database spans
span.set_attribute(SpanAttributes.DB_SYSTEM, "postgresql")
span.set_attribute(SpanAttributes.DB_NAME, "orders")
span.set_attribute(SpanAttributes.DB_OPERATION, "SELECT")
span.set_attribute(SpanAttributes.DB_SQL_TABLE, "orders")
# HTTP spans  
span.set_attribute(SpanAttributes.HTTP_METHOD, "POST")
span.set_attribute(SpanAttributes.HTTP_URL, "https://api.example.com/orders")
span.set_attribute(SpanAttributes.HTTP_STATUS_CODE, 200)
# Messaging spans
span.set_attribute(SpanAttributes.MESSAGING_SYSTEM, "kafka")
span.set_attribute(SpanAttributes.MESSAGING_DESTINATION, "order-events")
span.set_attribute(SpanAttributes.MESSAGING_OPERATION, "publish")
When all your services use semantic conventions, you get automatic dashboards, alerts, and service maps without any custom configuration in your observability backend.
The Grafana LGTM Stack: Popular OTel Backend
The Loki + Grafana + Tempo + Mimir stack has emerged as the most popular self-hosted backend for OTel:
# Quick start with Docker Compose
docker compose up -d \
  grafana/tempo \
  grafana/loki \
  prom/prometheus \
  grafana/grafana
With Grafana Alloy (the evolution of the Grafana Agent), you get a single collector that handles OTel data and integrates tightly with the LGTM stack.
What to Instrument First
For teams starting their OTel journey:
HTTP servers and clients — auto-instrumented, highest value
Database calls — auto-instrumented for most ORMs/drivers
Message queue producers/consumers — auto-instrumented for Kafka, RabbitMQ
External API calls — spans for all outbound HTTP
Business operations — manual spans for order processing, payment, etc.
Background jobs — spans for scheduled tasks and batch processing
Conclusion
OpenTelemetry is no longer something to evaluate — it’s the standard. In 2026, vendor-specific agents and proprietary instrumentation libraries are a technical liability. The teams running the best production systems are:
Auto-instrumenting all services from day one
Using the Collector as a telemetry pipeline with tail-based sampling
Connecting metrics and traces via exemplars
Using semantic conventions for interoperability
Continuously profiling hot paths
The investment is modest. The payoff — being able to debug production issues in minutes instead of hours — compounds over time.
Related posts: eBPF Production Observability, Kubernetes 2026 Gateway API and Service Mesh

OpenTelemetry in 2026: The De Facto Standard for Distributed Observability

댓글