A few months ago, we published Part 1 of the Introduction for our favorite open source continuous profiler, Parca. In Part 2, we explore Parca Agent- the eBPF sampling profiler that continuously collects raw profiling data to profile all the processes on a machine.
Parca Agent Quickstart Tutorial
This tutorial covers setting up Parca Agent to fetch stack information for all processes on the system.
Prerequisites:
- Currently, Parca Agent needs a host machine with
Linux Kernel v4.18+
to use eBPF functionalities. - If you are using a Mac,
lima-vm
can be used to run the Agent. Use the following command to startlima
:limactl start --name=default template://docker-rootful - A tutorial for using Parca Agent in Kubernetes can be found here
Download the latest server binary release for your architecture from the releases page.
For amd64
machines:
curl -sL https://github.com/parca-dev/parca-agent/releases/download/v0.11.0/parca-agent_0.11.0_`uname -s`_x86_64.tar.gz | tar xvfz - parca-agent && cd parca-agent
For arm64
machines:
curl -sL https://github.com/parca-dev/parca-agent/releases/download/v0.11.0/parca-agent_0.11.0_`uname -s`_arm64.tar.gz | tar xvfz - parca-agent && cd parca-agent
To run Parca Agent using docker
instead:
docker run --rm -it --privileged --pid host -p 7071:7071 -v /run:/run -v /boot:/boot -v /lib/modules:/lib/modules -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf -v /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket ghcr.io/parca-dev/parca-agent:v0.11.0 /bin/parca-agent
The Agent needs to be run as a root
user (or CAP_SYS_ADMIN
& CAP_SYS_RESOURCE
& CAP_BPF
) to leverage the eBPF
capabilities on the host Linux machine. Start parca agent
with:
sudo ./parca-agent
You should see the Parca agent logs appear so

By default, Parca Agent UI is exposed at host port 7071. The host port can be changed by using the --http-address
flag while starting the Agent.
Note:
To see the stack traces for the profiled processes using the Parca Server UI, the Agent must be started with the
--remote-store-address=7070
flag where7070
denoteslocalhost:7070
which exposes the Parca server. Detailed instructions to run the Parca server can be found here
We can see the Agent running if we navigate to the browser at http://localhost:7071/

Click on Process List
in the Processes
section to see all the processes profiled by the Agent.
If you take a closer look at the processes, you will notice several labels associated with each process PID. For example, let's look at a process that calls the zsh
shell:

These labels expose important metadata for processes, including the process name with comm
, the executable binary for the process with executable
, the kernel version, and whether or not the binary has debuginfo intact with the stripped
label, among others. Additional Kubernetes metadata such as pod
name is also added for processes running in containers.
The CPU profiler attaches various labels to each PID to not only make process discovery easier for the user but also enable Prometheus-style relabelling to allow the user to customize labels and filter profiles being sent from the Agent to the Parca Server accordingly.
Suppose we want to configure the Agent to send profiles for only processes that run the zsh
shell, we can use the following config:
relabel_configs:- source_labels: [comm]regex: zsh(.*)action: keep
The config can be stored in a YAML
file which can be passed to the Agent using the --config-file
flag. Expanding the Content
under the Configuration section will show us the config we passed earlier.

If you expand the Processes
section, you will notice only the processes with the label comm = zsh
are present.

More details and examples for relabelling are available in the Parca Agent documentation here
Inside Agent
Now that we have set up Parca Agent to discover all processes on our machine, let us dive deeper into its architecture and components.

Service/process discovery
The Agent discovers CPU samples for all processes across the host to have an always-on profiler. This system-wide profiling is enabled by attaching the CPU profiler to a single perf
event.
For 19x per second, the profiler looks to fetch new processes and update associated metadata. A profile is created for every process(or PID) discovered. The Agent also continuously polls for Kubernetes events that are running. Once a containerised process is discovered, a PID is associated with the containerised binary and available pod/cluster metadata attached to it.
eBPF in the agent
Under the hood, the CPU profiler is actually an eBPF program- this is attached to a perf_event
for each CPU, specifically the PERF_COUNT_SW_CPU_CLOCK
event. This enables us to instruct the kernel to call the BPF program numerous times every second without a very high overhead.
eBPF programs collect the raw data in kernel space. BPF maps are used to communicate this data to user space. These maps are created and populated by the profiler and further processed to give us stack traces.
These are the two BPF maps used by the Parca Agent:

Stack traces
- The stack traces map comprises the stack trace ID as the key and the memory addresses that point to the code executed - i.e. the stack trace address - as the value.
Counts
- The counts map comprises the triple of PID, user-space stack ID, and kernel-space stack ID as the key and the frequency with which the same stack trace ID is observed is the value.
Once the Agent accesses the data in user space, the maps are flushed every 10 seconds to enable the collection of new data.
Symbolisation
Stack traces collected by the profiler are memory location addresses- these need to be resolved to human readable symbols for us to utilise them. We obtain two types of symbols:
Kernel symbols
- Kernel stack traces are immediately symbolized by the Parca Agent since the Kernel can have a dynamic memory layout (for example, loaded eBPF programs in addition to the static kernel pieces). This is done by reading symbols from
/proc/kallsyms
and resolving the memory addresses accordingly.
- Kernel stack traces are immediately symbolized by the Parca Agent since the Kernel can have a dynamic memory layout (for example, loaded eBPF programs in addition to the static kernel pieces). This is done by reading symbols from
Application symbols
- Binaries or shared libraries/objects that contain debug symbols can have their symbols extracted and uploaded to the remote server. The remote Parca server can then use it to symbolize the stack traces at read time rather than in the agent. This also allows debug symbols to be uploaded separately if they are stripped in a CI process or retrieved from upstream public symbol servers such as debuginfod.
An issue that frequently arises while symbolising applications is the lack of debug information in the application binary - often the case when binaries are compiled without frame pointers. To fix this, we extended our symbolizer to generate unwind tables from DWARF debug information using the .eh_frame
section of a binary. Although still experimental, the DWARF-based stack walking can be used by enabling the --experimental-enable-dwarf-unwinding -debug-process-names="node"
flag.
Supported language profiles
Parca agent currently supports CPU profiling for all compiled languages like C/C++, Rust, Go, etc.
Symbolisation for some JIT compilers like JVM, Erlang/Elixir, and NodeJS can be achieved by using perf-maps
. Our parca-demo repo has some great examples for showcasing profiling with Parca using perf-maps
. Out-of-the-box support for dynamic languages symbolisation is on our current roadmap and related issues are being tracked here.
Additional resources
Worried about giving privileged permissions to an eBPF based, in-kernel binary? We have got you covered.
There are several other flags to make running the agent more configurable. Check them out here.
Want to contribute to Parca Agent? Start here or say hi on Discord!