<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1200&amp;q=80"><h2 id="introduction">Introduction</h2>In 2026, OpenTelemetry (OTel) has emerged as the undisputed standard for cloud-native observability. What began as a CNCF incubating project has graduated to become the second-largest CNCF project by contributor count (behind only Kubernetes itself). Every major cloud vendor, APM tool, and service mesh now speaks OpenTelemetry natively.This guide covers practical OTel deployment — not just the theory, but what actually works in production at scale.<img src="https://images.unsplash.com/photo-1558494949-ef010cbdcc31?w=900&amp;q=80" alt="Server monitoring dashboard with metrics"> Photo by Luke Chesser on Unsplash<hr><h2 id="why-opentelemetry-won">Why OpenTelemetry Won</h2>Before OTel, instrumentation was a vendor lock-in nightmare:<ul><li>Datadog agent → Datadog only</li><li>Jaeger client → Jaeger only</li><li>CloudWatch agent → AWS only</li></ul>OpenTelemetry solved this with a vendor-neutral wire format (OTLP) and standardized SDKs across 11+ languages. You instrument once, export anywhere.The tipping point came when all major vendors converged:<ul><li>AWS X-Ray now accepts OTLP natively</li><li>Datadog, New Relic, Honeycomb all support OTLP ingest</li><li>Grafana’s LGTM stack (Loki, Grafana, Tempo, Mimir) is built around it</li></ul><hr><h2 id="the-otel-architecture-three-signals">The OTel Architecture: Three Signals</h2>OpenTelemetry handles three observability signals:<h3 id="1-traces-distributed-tracing">1. Traces (Distributed Tracing)</h3>Traces follow a request as it propagates across microservices. Each unit of work is a span; related spans form a trace.<div><pre><code>from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Instrumentation
tracer = trace.get_tracer(__name__)

def process_order(order_id: str):
 with tracer.start_as_current_span("process_order") as span:
 span.set_attribute("order.id", order_id)
 span.set_attribute("order.source", "api")
 
 result = fetch_order_from_db(order_id)
 span.set_attribute("order.value", result.total)
 return result
</code></pre></div><h3 id="2-metrics">2. Metrics</h3>Metrics are numerical measurements over time — counters, gauges, histograms.<div><pre><code>from opentelemetry import metrics

meter = metrics.get_meter(__name__)

# Counter: monotonically increasing value
request_counter = meter.create_counter(
 "http.requests.total",
 description="Total HTTP requests",
 unit="1"
)

# Histogram: distribution of values
request_duration = meter.create_histogram(
 "http.request.duration",
 description="HTTP request duration",
 unit="ms"
)

def handle_request(method: str, path: str):
 start = time.time()
 try:
 response = process_request()
 request_counter.add(1, {"method": method, "path": path, "status": "200"})
 return response
 finally:
 duration_ms = (time.time() - start) * 1000
 request_duration.record(duration_ms, {"method": method, "path": path})
</code></pre></div><h3 id="3-logs-structured-logging">3. Logs (Structured Logging)</h3>OTel’s log signal connects logs to traces via trace context propagation, giving you correlated logs without a separate log correlation system.<div><pre><code>import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor

LoggingInstrumentor().instrument(set_logging_format=True)

logger = logging.getLogger(__name__)

# Logs now automatically include trace_id and span_id
logger.info("Processing payment", extra={
 "payment.amount": 99.99,
 "payment.currency": "USD",
 "user.id": user_id
})
# Output: [trace_id=abc123 span_id=def456] Processing payment ...
</code></pre></div><hr><h2 id="the-otel-collector-your-observability-router">The OTel Collector: Your Observability Router</h2>The OTel Collector is the key infrastructure component — a vendor-agnostic proxy that receives telemetry, processes it, and routes it to one or more backends.<div><pre><code># otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 
 # Also scrape Prometheus metrics
 prometheus:
 config:
 scrape_configs:
 - job_name: 'kubernetes-pods'
 kubernetes_sd_configs:
 - role: pod

processors:
 # Drop high-cardinality labels
 attributes:
 actions:
 - key: http.user_agent
 action: delete
 
 # Sample traces intelligently
 tail_sampling:
 decision_wait: 10s
 policies:
 - name: errors-policy
 type: status_code
 status_code: { status_codes: [ERROR] }
 - name: slow-traces
 type: latency
 latency: { threshold_ms: 1000 }
 - name: probabilistic-10pct
 type: probabilistic
 probabilistic: { sampling_percentage: 10 }
 
 # Add resource attributes
 resource:
 attributes:
 - key: deployment.environment
 value: production
 action: upsert

exporters:
 # Send to Grafana Tempo
 otlp/tempo:
 endpoint: tempo:4317
 tls:
 insecure: true
 
 # Send metrics to Prometheus
 prometheus:
 endpoint: "0.0.0.0:8889"
 
 # Also send errors to Datadog
 datadog:
 api:
 key: ${env:DD_API_KEY}

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [attributes, tail_sampling, resource]
 exporters: [otlp/tempo, datadog]
 metrics:
 receivers: [otlp, prometheus]
 processors: [resource]
 exporters: [prometheus]
</code></pre></div><hr><h2 id="auto-instrumentation-the-easy-win">Auto-Instrumentation: The Easy Win</h2>OTel’s killer feature for production adoption is zero-code auto-instrumentation. For most frameworks, you get traces and metrics for free:<h3 id="python-fastapi-sqlalchemy-redis-etc">Python (FastAPI, SQLAlchemy, Redis, etc.)</h3><div><pre><code>pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install

# Run your app with auto-instrumentation
opentelemetry-instrument \
 --service_name=order-service \
 --exporter_otlp_endpoint=http://otel-collector:4317 \
 python app.py
</code></pre></div><h3 id="java-spring-boot-hibernate-grpc-etc">Java (Spring Boot, Hibernate, gRPC, etc.)</h3><div><pre><code># Dockerfile
FROM eclipse-temurin:21-jre
COPY --from=build /app/app.jar /app.jar
COPY otel-javaagent.jar /otel-javaagent.jar

ENV JAVA_TOOL_OPTIONS="-javaagent:/otel-javaagent.jar"
ENV OTEL_SERVICE_NAME="payment-service"
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"

CMD ["java", "-jar", "/app.jar"]
</code></pre></div><h3 id="kubernetes-wide-via-operator">Kubernetes-Wide (via Operator)</h3><div><pre><code># Install OTel Operator and enable auto-instrumentation per namespace
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
 name: auto-instrumentation
 namespace: production
spec:
 exporter:
 endpoint: http://otel-collector:4317
 propagators:
 - tracecontext
 - baggage
 - b3
 python:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
 java:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
 nodejs:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
</code></pre></div><hr><h2 id="common-production-pitfalls">Common Production Pitfalls</h2><h3 id="1-cardinality-explosion">1. Cardinality Explosion</h3>High-cardinality labels (user IDs, request IDs in metric labels) can destroy your metrics backend.<div><pre><code># ❌ Bad: unbounded cardinality
request_counter.add(1, {"user_id": user_id, "path": request.path})

# ✅ Good: bounded labels only
request_counter.add(1, {"path": normalize_path(request.path), "method": request.method})
</code></pre></div><h3 id="2-synchronous-exporters-in-hot-paths">2. Synchronous Exporters in Hot Paths</h3><div><pre><code># ❌ Bad: blocks your request handler
SimpleSpanProcessor(exporter) 

# ✅ Good: async batching
BatchSpanProcessor(exporter, max_export_batch_size=512, schedule_delay_millis=5000)
</code></pre></div><h3 id="3-missing-context-propagation">3. Missing Context Propagation</h3>If you use thread pools, async queues, or background jobs, you must propagate trace context manually:<div><pre><code>from opentelemetry import context, propagate

# Before putting work on a queue
carrier = {}
propagate.inject(carrier)
queue.put({"task": task_data, "otel_context": carrier})

# When processing from queue
carrier = message["otel_context"]
ctx = propagate.extract(carrier)
token = context.attach(ctx)
try:
 process_task(message["task"])
finally:
 context.detach(token)
</code></pre></div><hr><h2 id="the-lgtm-stack-open-source-observability">The LGTM Stack: Open Source Observability</h2>For teams wanting full control without vendor costs, the Grafana LGTM stack is now production-ready:<ul><li>Loki — Log aggregation (like Elasticsearch but cheaper)</li><li>Grafana — Dashboards and alerting</li><li>Tempo — Distributed tracing (OTLP native)</li><li>Mimir — Horizontally scalable Prometheus-compatible metrics</li></ul>All four are OTel-native and deployable on Kubernetes with official Helm charts. At moderate scale (&lt; 1000 RPS), a 3-node cluster handles all four services comfortably.<hr><h2 id="conclusion">Conclusion</h2>OpenTelemetry in 2026 is no longer an emerging technology — it’s infrastructure. If you’re not using it, you’re probably using something that will eventually migrate to it.The key takeaway: start with auto-instrumentation, add manual spans for business context, run a collector, and pick backends later. The investment in OTel is portable. Your instrumentation code won’t need to change when you switch from Jaeger to Tempo, or from Datadog to Grafana Cloud.Observability isn’t optional at scale. OTel makes it achievable without vendor lock-in.<hr>Related Posts:<ul><li>Kubernetes Observability: eBPF vs OpenTelemetry in 2026</li><li>Grafana LGTM Stack on Kubernetes: Production Setup Guide</li></ul> <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

Introduction
In 2026, OpenTelemetry (OTel) has emerged as the undisputed standard for cloud-native observability. What began as a CNCF incubating project has graduated to become the second-largest CNCF project by contributor count (behind only Kubernetes itself). Every major cloud vendor, APM tool, and service mesh now speaks OpenTelemetry natively.
This guide covers practical OTel deployment — not just the theory, but what actually works in production at scale.
Photo by Luke Chesser on Unsplash
Why OpenTelemetry Won
Before OTel, instrumentation was a vendor lock-in nightmare:
Datadog agent → Datadog only
Jaeger client → Jaeger only
CloudWatch agent → AWS only
OpenTelemetry solved this with a vendor-neutral wire format (OTLP) and standardized SDKs across 11+ languages. You instrument once, export anywhere.
The tipping point came when all major vendors converged:
AWS X-Ray now accepts OTLP natively
Datadog, New Relic, Honeycomb all support OTLP ingest
Grafana’s LGTM stack (Loki, Grafana, Tempo, Mimir) is built around it
The OTel Architecture: Three Signals
OpenTelemetry handles three observability signals:
1. Traces (Distributed Tracing)
Traces follow a request as it propagates across microservices. Each unit of work is a span; related spans form a trace.
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Setup
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
# Instrumentation
tracer = trace.get_tracer(__name__)
def process_order(order_id: str):
 with tracer.start_as_current_span("process_order") as span:
 span.set_attribute("order.id", order_id)
 span.set_attribute("order.source", "api")
 
 result = fetch_order_from_db(order_id)
 span.set_attribute("order.value", result.total)
 return result
2. Metrics
Metrics are numerical measurements over time — counters, gauges, histograms.
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
# Counter: monotonically increasing value
request_counter = meter.create_counter(
 "http.requests.total",
 description="Total HTTP requests",
 unit="1"
)
# Histogram: distribution of values
request_duration = meter.create_histogram(
 "http.request.duration",
 description="HTTP request duration",
 unit="ms"
)
def handle_request(method: str, path: str):
 start = time.time()
 try:
 response = process_request()
 request_counter.add(1, {"method": method, "path": path, "status": "200"})
 return response
 finally:
 duration_ms = (time.time() - start) * 1000
 request_duration.record(duration_ms, {"method": method, "path": path})
3. Logs (Structured Logging)
OTel’s log signal connects logs to traces via trace context propagation, giving you correlated logs without a separate log correlation system.
import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor
LoggingInstrumentor().instrument(set_logging_format=True)
logger = logging.getLogger(__name__)
# Logs now automatically include trace_id and span_id
logger.info("Processing payment", extra={
 "payment.amount": 99.99,
 "payment.currency": "USD",
 "user.id": user_id
})
# Output: [trace_id=abc123 span_id=def456] Processing payment ...
The OTel Collector: Your Observability Router
The OTel Collector is the key infrastructure component — a vendor-agnostic proxy that receives telemetry, processes it, and routes it to one or more backends.
# otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 
 # Also scrape Prometheus metrics
 prometheus:
 config:
 scrape_configs:
 - job_name: 'kubernetes-pods'
 kubernetes_sd_configs:
 - role: pod
processors:
 # Drop high-cardinality labels
 attributes:
 actions:
 - key: http.user_agent
 action: delete
 
 # Sample traces intelligently
 tail_sampling:
 decision_wait: 10s
 policies:
 - name: errors-policy
 type: status_code
 status_code: { status_codes: [ERROR] }
 - name: slow-traces
 type: latency
 latency: { threshold_ms: 1000 }
 - name: probabilistic-10pct
 type: probabilistic
 probabilistic: { sampling_percentage: 10 }
 
 # Add resource attributes
 resource:
 attributes:
 - key: deployment.environment
 value: production
 action: upsert
exporters:
 # Send to Grafana Tempo
 otlp/tempo:
 endpoint: tempo:4317
 tls:
 insecure: true
 
 # Send metrics to Prometheus
 prometheus:
 endpoint: "0.0.0.0:8889"
 
 # Also send errors to Datadog
 datadog:
 api:
 key: ${env:DD_API_KEY}
service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [attributes, tail_sampling, resource]
 exporters: [otlp/tempo, datadog]
 metrics:
 receivers: [otlp, prometheus]
 processors: [resource]
 exporters: [prometheus]
Auto-Instrumentation: The Easy Win
OTel’s killer feature for production adoption is zero-code auto-instrumentation. For most frameworks, you get traces and metrics for free:
Python (FastAPI, SQLAlchemy, Redis, etc.)
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
# Run your app with auto-instrumentation
opentelemetry-instrument \
 --service_name=order-service \
 --exporter_otlp_endpoint=http://otel-collector:4317 \
 python app.py
Java (Spring Boot, Hibernate, gRPC, etc.)
# Dockerfile
FROM eclipse-temurin:21-jre
COPY --from=build /app/app.jar /app.jar
COPY otel-javaagent.jar /otel-javaagent.jar
ENV JAVA_TOOL_OPTIONS="-javaagent:/otel-javaagent.jar"
ENV OTEL_SERVICE_NAME="payment-service"
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
CMD ["java", "-jar", "/app.jar"]
Kubernetes-Wide (via Operator)
# Install OTel Operator and enable auto-instrumentation per namespace
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
 name: auto-instrumentation
 namespace: production
spec:
 exporter:
 endpoint: http://otel-collector:4317
 propagators:
 - tracecontext
 - baggage
 - b3
 python:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
 java:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
 nodejs:
 image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
Common Production Pitfalls
1. Cardinality Explosion
High-cardinality labels (user IDs, request IDs in metric labels) can destroy your metrics backend.
# ❌ Bad: unbounded cardinality
request_counter.add(1, {"user_id": user_id, "path": request.path})
# ✅ Good: bounded labels only
request_counter.add(1, {"path": normalize_path(request.path), "method": request.method})
2. Synchronous Exporters in Hot Paths
# ❌ Bad: blocks your request handler
SimpleSpanProcessor(exporter) 
# ✅ Good: async batching
BatchSpanProcessor(exporter, max_export_batch_size=512, schedule_delay_millis=5000)
3. Missing Context Propagation
If you use thread pools, async queues, or background jobs, you must propagate trace context manually:
from opentelemetry import context, propagate
# Before putting work on a queue
carrier = {}
propagate.inject(carrier)
queue.put({"task": task_data, "otel_context": carrier})
# When processing from queue
carrier = message["otel_context"]
ctx = propagate.extract(carrier)
token = context.attach(ctx)
try:
 process_task(message["task"])
finally:
 context.detach(token)
The LGTM Stack: Open Source Observability
For teams wanting full control without vendor costs, the Grafana LGTM stack is now production-ready:
Loki — Log aggregation (like Elasticsearch but cheaper)
Grafana — Dashboards and alerting
Tempo — Distributed tracing (OTLP native)
Mimir — Horizontally scalable Prometheus-compatible metrics
All four are OTel-native and deployable on Kubernetes with official Helm charts. At moderate scale (< 1000 RPS), a 3-node cluster handles all four services comfortably.
Conclusion
OpenTelemetry in 2026 is no longer an emerging technology — it’s infrastructure. If you’re not using it, you’re probably using something that will eventually migrate to it.
The key takeaway: start with auto-instrumentation, add manual spans for business context, run a collector, and pick backends later. The investment in OTel is portable. Your instrumentation code won’t need to change when you switch from Jaeger to Tempo, or from Datadog to Grafana Cloud.
Observability isn’t optional at scale. OTel makes it achievable without vendor lock-in.
Related Posts:
Kubernetes Observability: eBPF vs OpenTelemetry in 2026
Grafana LGTM Stack on Kubernetes: Production Setup Guide

OpenTelemetry in Production: A Practical Guide to Observability in 2026

댓글