OpenTelemetry Profiling goes Alpha

For the past couple of years, we've participated in the OpenTelemetry Profiling Working Group, which has been working hard on specifying profiling as the fourth signal within the OpenTelemetry project. Today this protocol has entered the Alpha stage within the OpenTelemetry project.

In this post we won't focus on the details of the protocol, the official announcement blog post does a great job at that. Instead we want to show you two examples on how to use it with the opentelemetry-ebpf-profiler, which is also available via an opentelemetry collector distribution. Look for the "opentelemetry-collector-ebpf-profiler" distribution, for example a container image you may use could be:

ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-ebpf-profiler:0.148.0

Polar Signals Cloud

While we generally recommend our customers to use our agent (which also builds on the opentelemetr-ebpf-profiler), it may make sense to use the built-in profiler within the collector if people are already running it anyway.

An example config for an OpenTelemetry collector to produce profiling data and send it to Polar Signals Cloud.

receivers:
  profiling:

exporters:
  otlp/polarsignals:
    endpoint: https://grpc.polarsignals.com:443
    headers:
      Authorization: Bearer 
      projectID: 

service:
  telemetry:
    logs:
      level: info
  pipelines:
    profiles:
      receivers: [profiling]
      exporters: [otlp/polarsignals]

And you can see data!

Screenshot of Polar Signals Cloud of a flamegraph showing data produced by the OpenTelemetry eBPF Profiler

You may have noticed, none of the native stacks have been symbolized, currently only JIT, interpreted, or Go stacks are symbolized on-host, and no support for asynchronous symbolization has been specified yet, this is among other things one of the follow ups we want to specify as the group, a protocol for uploading symbol data to a backend.

Open Source Parca

Or if you want to try it with the Parca open source server. Here's the config for the OpenTelemetry collector.

receivers:
  profiling:

exporters:
  otlp/local:
    endpoint: http://localhost:7070

    tls:
      insecure: true
      insecure_skip_verify: true

service:
  telemetry:
    logs:
      level: info
  pipelines:
    profiles:
      receivers: [profiling]
      exporters: [otlp/local]

Putting it together with a docker-compose.yaml:

services:
  parca:
    image: ghcr.io/parca-dev/parca:v0.27.1
    network_mode: host

  otelcol:
    image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-ebpf-profiler:0.148.0
    network_mode: host
    pid: host
    privileged: true
    command:
      - --config=/etc/otelcol/config.yaml
      - --feature-gates=+service.profilesSupport
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /sys/fs/bpf:/sys/fs/bpf
      - /sys/kernel/debug:/sys/kernel/debug
      - /run:/run
      - /boot:/boot
      - /lib/modules:/lib/modules
    depends_on:
      - parca

Conclusion

This has already been a monumental effort, and while we had a part in it, we want to give a huge shout out to everyone involved! We're not done, for one there is still a decent amount of work left to eventually declare it stable.

If this sounds interesting to you to work on, join an OpenTelemetry Profiling Working Group call, which happens every 2 weeks!