<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1200&amp;q=80">The three pillars of observability — logs, metrics, traces — were the consensus model for the better part of a decade. In 2026, a fourth pillar has emerged: continuous profiling. Combined with eBPF-powered automatic instrumentation and the now-ubiquitous OpenTelemetry standard, modern observability looks quite different from what we built just three years ago.<img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=900&amp;q=80" alt="Data visualization dashboard"> Photo by <a href="https://unsplash.com/@lukechesser">Luke Chesser</a> on Unsplash<h2 id="the-current-state-opentelemetry-has-won">The Current State: OpenTelemetry Has Won</h2>OpenTelemetry (OTel) has achieved what seemed impossible: vendor-neutral observability instrumentation that the entire industry has coalesced around. In 2026, if you’re not using OTel, you’re leaving interoperability on the table.The promise was always “instrument once, send anywhere.” It’s now delivering on that promise.<h3 id="auto-instrumentation-zero-touch-observability">Auto-Instrumentation: Zero-Touch Observability</h3>The biggest shift is zero-code auto-instrumentation. For most mainstream languages, you get distributed tracing, metrics, and structured logs without changing application code:<div><pre><code># Python: instrument at runtime, not code time
OTEL_SERVICE_NAME=payment-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
opentelemetry-instrument python app.py
</code></pre></div><div><pre><code># Java: JVM agent does everything
FROM eclipse-temurin:21
COPY otel-javaagent.jar /opt/
COPY app.jar /opt/

ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/otel-javaagent.jar"
ENV OTEL_SERVICE_NAME="order-service"
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"

CMD ["java", "-jar", "/opt/app.jar"]
</code></pre></div>The Java agent alone instruments: HTTP clients/servers, JDBC, Redis, Kafka, gRPC, messaging, and more — with zero code changes.<h3 id="the-otel-collector-your-observability-router">The OTel Collector: Your Observability Router</h3>The OTel Collector is the backbone of modern observability pipelines. It receives telemetry from your services, processes/enriches it, and routes it to any backend:<div><pre><code># otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318
 
 # Also collect infrastructure metrics
 hostmetrics:
 collection_interval: 30s
 scrapers:
 cpu:
 memory:
 disk:
 network:

processors:
 batch:
 timeout: 5s
 send_batch_size: 1000
 
 # Enrich all telemetry with k8s metadata
 k8sattributes:
 auth_type: serviceAccount
 extract:
 metadata:
 - k8s.namespace.name
 - k8s.pod.name
 - k8s.deployment.name
 - k8s.node.name
 
 # Sample high-volume traces
 tail_sampling:
 decision_wait: 10s
 policies:
 - name: errors-policy
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: slow-traces
 type: latency
 latency: {threshold_ms: 1000}
 - name: probabilistic-sampling
 type: probabilistic
 probabilistic: {sampling_percentage: 10}

exporters:
 otlp/tempo:
 endpoint: tempo:4317
 tls:
 insecure: true
 
 prometheusremotewrite:
 endpoint: http://prometheus:9090/api/v1/write
 
 loki:
 endpoint: http://loki:3100/loki/api/v1/push

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [k8sattributes, tail_sampling, batch]
 exporters: [otlp/tempo]
 metrics:
 receivers: [otlp, hostmetrics]
 processors: [k8sattributes, batch]
 exporters: [prometheusremotewrite]
 logs:
 receivers: [otlp]
 processors: [k8sattributes, batch]
 exporters: [loki]
</code></pre></div><h2 id="ebpf-observability-without-instrumentation">eBPF: Observability Without Instrumentation</h2>eBPF has fundamentally changed what’s possible for infrastructure observability. eBPF programs run in the Linux kernel, allowing you to observe everything happening on a node — network flows, syscalls, CPU scheduling — with near-zero overhead.<h3 id="what-ebpf-enables">What eBPF Enables</h3><ul><li>Automatic distributed tracing across services — without any instrumentation at all</li><li>Network performance visibility — packet-level latency, retransmits, connection state</li><li>Security observability — every syscall, every file access, every network connection</li><li>CPU flame graphs — continuous profiling of every process</li></ul>Tools built on eBPF:Cilium / Hubble: Network observability and policy enforcement at the kernel level<div><pre><code># Hubble shows L7 traffic without any application changes
hubble observe --namespace production --protocol http
# TIMESTAMP SOURCE DESTINATION TYPE VERDICT SUMMARY
# 10:30:01.234 payment/payment-0 database/postgres-0 HTTP FORWARDED POST /api/v1/charge 200ms
</code></pre></div>Pixie: Automatic distributed tracing for Kubernetes — add one DaemonSet, get full request traces across your cluster.Parca / Pyroscope: Continuous profiling via eBPF — always-on CPU and memory profiling with no performance overhead.<h2 id="the-fourth-pillar-continuous-profiling">The Fourth Pillar: Continuous Profiling</h2>This is the game-changer. Traditional profiling was something you turned on during incidents — because it had overhead. eBPF-based continuous profiling has near-zero overhead and runs constantly.What you get: Flame graphs for every service, always, going back weeks.<img src="https://images.unsplash.com/photo-1504868584819-f8e8b4b6d7e3?w=900&amp;q=80" alt="Performance monitoring visualization"> Photo by <a href="https://unsplash.com/@frankvex">Frank Vessia</a> on UnsplashWhen a service slows down, you don’t ask “what was it doing?” — you see what it was doing, at the exact time, with stack traces.Parca setup on Kubernetes:<div><pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: parca-agent
spec:
 template:
 spec:
 hostPID: true
 hostNetwork: true
 containers:
 - name: parca-agent
 image: ghcr.io/parca-dev/parca-agent:latest
 securityContext:
 privileged: true # Required for eBPF
 args:
 - /bin/parca-agent
 - --http-address=:7071
 - --remote-store-address=parca.monitoring:7070
 - --kubernetes
 volumeMounts:
 - name: host-root
 mountPath: /host
 readOnly: true
 volumes:
 - name: host-root
 hostPath:
 path: /
</code></pre></div><h2 id="connecting-the-four-pillars">Connecting the Four Pillars</h2>The real power comes from correlation. Modern observability platforms connect:<ul><li>A slow trace → what was the CPU profile at that exact moment?</li><li>A spike in error logs → what changed in deployment events?</li><li>High memory usage → which code path is allocating?</li></ul>Grafana’s unified platform (Loki + Tempo + Mimir + Pyroscope) now correlates all four pillars natively. You click on a trace span, and it shows you the flame graph for that service at that exact timestamp.<h2 id="slos-the-missing-layer">SLOs: The Missing Layer</h2>Observability without SLOs is just expensive log storage. Service Level Objectives define what “good” looks like, and your observability stack should alert on SLO burn rate — not raw metrics:<div><pre><code># Pyrra SLO definition
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
 name: payment-api-availability
spec:
 description: Payment API should be available 99.9% of the time
 target: "99.9"
 window: 30d
 serviceMonitorSelector:
 matchLabels:
 app: payment-service
 indicator:
 ratio:
 errors:
 metric: http_requests_total{status=~"5.."}
 total:
 metric: http_requests_total
</code></pre></div>Alert on burn rate, not threshold. If you’re burning your 30-day error budget at 10x the normal rate, page someone — even if you haven’t violated the SLO yet.<h2 id="the-practical-stack-in-2026">The Practical Stack in 2026</h2>For self-hosted teams:<ul><li>Collection: OTel Collector (universal ingestion)</li><li>Traces: Tempo (Grafana)</li><li>Metrics: Prometheus + Thanos (long-term storage)</li><li>Logs: Loki</li><li>Profiling: Parca or Pyroscope</li><li>Visualization: Grafana (connects all four)</li><li>SLOs: Pyrra or Sloth</li></ul>For managed:<ul><li>Grafana Cloud (full stack, generous free tier)</li><li>Datadog (expensive but best-in-class UX)</li><li>Honeycomb (trace-first, excellent for complex distributed systems)</li></ul><h2 id="conclusion">Conclusion</h2>Observability in 2026 has moved from “set up Prometheus and hope for the best” to a sophisticated discipline with clear standards (OTel), powerful kernel-level tooling (eBPF), and a fourth pillar that answers questions you couldn’t ask before. The cost of being blind to your production systems has never been lower.Instrument with OTel, observe with eBPF, profile continuously, and define your SLOs. The pager will thank you. <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

The three pillars of observability — logs, metrics, traces — were the consensus model for the better part of a decade. In 2026, a fourth pillar has emerged: continuous profiling. Combined with eBPF-powered automatic instrumentation and the now-ubiquitous OpenTelemetry standard, modern observability looks quite different from what we built just three years ago.
Photo by Luke Chesser on Unsplash
The Current State: OpenTelemetry Has Won
OpenTelemetry (OTel) has achieved what seemed impossible: vendor-neutral observability instrumentation that the entire industry has coalesced around. In 2026, if you’re not using OTel, you’re leaving interoperability on the table.
The promise was always “instrument once, send anywhere.” It’s now delivering on that promise.
Auto-Instrumentation: Zero-Touch Observability
The biggest shift is zero-code auto-instrumentation. For most mainstream languages, you get distributed tracing, metrics, and structured logs without changing application code:
# Python: instrument at runtime, not code time
OTEL_SERVICE_NAME=payment-service \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317 \
opentelemetry-instrument python app.py
# Java: JVM agent does everything
FROM eclipse-temurin:21
COPY otel-javaagent.jar /opt/
COPY app.jar /opt/
ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/otel-javaagent.jar"
ENV OTEL_SERVICE_NAME="order-service"
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
CMD ["java", "-jar", "/opt/app.jar"]
The Java agent alone instruments: HTTP clients/servers, JDBC, Redis, Kafka, gRPC, messaging, and more — with zero code changes.
The OTel Collector: Your Observability Router
The OTel Collector is the backbone of modern observability pipelines. It receives telemetry from your services, processes/enriches it, and routes it to any backend:
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  
  # Also collect infrastructure metrics
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      disk:
      network:
processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  
  # Enrich all telemetry with k8s metadata
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
  
  # Sample high-volume traces
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-traces
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic-sampling
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, tail_sampling, batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [k8sattributes, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [loki]
eBPF: Observability Without Instrumentation
eBPF has fundamentally changed what’s possible for infrastructure observability. eBPF programs run in the Linux kernel, allowing you to observe everything happening on a node — network flows, syscalls, CPU scheduling — with near-zero overhead.
What eBPF Enables
Automatic distributed tracing across services — without any instrumentation at all
Network performance visibility — packet-level latency, retransmits, connection state
Security observability — every syscall, every file access, every network connection
CPU flame graphs — continuous profiling of every process
Tools built on eBPF:
Cilium / Hubble: Network observability and policy enforcement at the kernel level
# Hubble shows L7 traffic without any application changes
hubble observe --namespace production --protocol http
# TIMESTAMP      SOURCE              DESTINATION         TYPE    VERDICT  SUMMARY
# 10:30:01.234   payment/payment-0   database/postgres-0 HTTP    FORWARDED POST /api/v1/charge 200ms
Pixie: Automatic distributed tracing for Kubernetes — add one DaemonSet, get full request traces across your cluster.
Parca / Pyroscope: Continuous profiling via eBPF — always-on CPU and memory profiling with no performance overhead.
The Fourth Pillar: Continuous Profiling
This is the game-changer. Traditional profiling was something you turned on during incidents — because it had overhead. eBPF-based continuous profiling has near-zero overhead and runs constantly.
What you get: Flame graphs for every service, always, going back weeks.
Photo by Frank Vessia on Unsplash
When a service slows down, you don’t ask “what was it doing?” — you see what it was doing, at the exact time, with stack traces.
Parca setup on Kubernetes:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: parca-agent
spec:
  template:
    spec:
      hostPID: true
      hostNetwork: true
      containers:
        - name: parca-agent
          image: ghcr.io/parca-dev/parca-agent:latest
          securityContext:
            privileged: true     # Required for eBPF
          args:
            - /bin/parca-agent
            - --http-address=:7071
            - --remote-store-address=parca.monitoring:7070
            - --kubernetes
          volumeMounts:
            - name: host-root
              mountPath: /host
              readOnly: true
      volumes:
        - name: host-root
          hostPath:
            path: /
Connecting the Four Pillars
The real power comes from correlation. Modern observability platforms connect:
A slow trace → what was the CPU profile at that exact moment?
A spike in error logs → what changed in deployment events?
High memory usage → which code path is allocating?
Grafana’s unified platform (Loki + Tempo + Mimir + Pyroscope) now correlates all four pillars natively. You click on a trace span, and it shows you the flame graph for that service at that exact timestamp.
SLOs: The Missing Layer
Observability without SLOs is just expensive log storage. Service Level Objectives define what “good” looks like, and your observability stack should alert on SLO burn rate — not raw metrics:
# Pyrra SLO definition
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: payment-api-availability
spec:
  description: Payment API should be available 99.9% of the time
  target: "99.9"
  window: 30d
  serviceMonitorSelector:
    matchLabels:
      app: payment-service
  indicator:
    ratio:
      errors:
        metric: http_requests_total{status=~"5.."}
      total:
        metric: http_requests_total
Alert on burn rate, not threshold. If you’re burning your 30-day error budget at 10x the normal rate, page someone — even if you haven’t violated the SLO yet.
The Practical Stack in 2026
For self-hosted teams:
Collection: OTel Collector (universal ingestion)
Traces: Tempo (Grafana)
Metrics: Prometheus + Thanos (long-term storage)
Logs: Loki
Profiling: Parca or Pyroscope
Visualization: Grafana (connects all four)
SLOs: Pyrra or Sloth
For managed:
Grafana Cloud (full stack, generous free tier)
Datadog (expensive but best-in-class UX)
Honeycomb (trace-first, excellent for complex distributed systems)
Conclusion
Observability in 2026 has moved from “set up Prometheus and hope for the best” to a sophisticated discipline with clear standards (OTel), powerful kernel-level tooling (eBPF), and a fourth pillar that answers questions you couldn’t ask before. The cost of being blind to your production systems has never been lower.
Instrument with OTel, observe with eBPF, profile continuously, and define your SLOs. The pager will thank you.

Observability in 2026: OpenTelemetry, eBPF, and the Rise of Continuous Profiling

댓글