Kubernetes Observability: Collecting Logs and Metrics from Pods with OTel

DaemonSet vs Sidecar: The Two Collection Patterns

There are two common ways to run the OpenTelemetry Collector in Kubernetes. A DaemonSet runs one Collector instance per node, collecting logs and metrics from every pod on that node — efficient because you're not running a Collector per pod, but it means all pods on a node share the same Collector configuration. A sidecar runs a Collector container inside each pod alongside your application, giving per-pod configuration flexibility at the cost of higher resource overhead across a large cluster. Most teams default to DaemonSet for node-level log and metric collection, and reserve sidecars for cases needing pod-specific processing.

Collecting Pod Logs Without Changing Application Code

Kubernetes already writes every container's stdout/stderr to a file on the node (/var/log/pods/...). A DaemonSet-deployed Collector can tail these files directly using the filelog receiver, meaning you get logs from every pod without any application-level OpenTelemetry SDK integration — useful for legacy services or third-party containers you can't instrument directly.

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    operators:
      - type: container
        format: containerd

processors:
  batch: {}
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false

exporters:
  otlphttp:
    endpoint: http://xd-api.observability.svc:4318

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [k8sattributes, batch]
      exporters: [otlphttp]

The k8sattributes processor is the important addition here — it enriches every log with pod name, namespace, and labels automatically, which is what makes filtering by service or deployment possible later in your observability platform's UI.

What Changes for Instrumented Services

If a service is already instrumented with an OpenTelemetry SDK (see our Python OTel tutorial for an example), it can either export directly to the node-local DaemonSet Collector's service endpoint, or use a sidecar for anything needing pod-specific processing before the DaemonSet forwards it onward. Either approach avoids the service needing to know the cluster-wide backend address directly — it always talks to a local Collector first.

Simplifying Deployment With the OpenTelemetry Operator

Hand-writing DaemonSet manifests and Collector configs for every cluster gets repetitive. The OpenTelemetry Operator for Kubernetes manages Collector instances as custom resources, handling rollout, config updates, and auto-instrumentation injection for supported languages — worth adopting once you're running OTel across more than a handful of services, rather than maintaining raw YAML by hand indefinitely.

Conclusion

Kubernetes doesn't fundamentally change what OpenTelemetry collects — logs, metrics, and traces are still the same three signals. It changes how you get the Collector positioned to receive them: DaemonSet for node-wide, no-code-change log collection, sidecars for pod-specific needs, and the Operator to keep it all manageable at scale. If you're deploying without Kubernetes, the same Collector concepts apply more simply — see our Docker Compose stack guide.