Offline Mode for the Parca Agent

Introduction

CPU profiling with Parca involves two main components: the Parca agent, which runs as a process called parca-agent on the host being profiled, and the Parca backend, which runs as a process called parca on any host; possibly not the same one. The agent continuously collects stack traces of the code scheduled on the CPU. It then periodically sends the collected stack traces to the backend, where they are stored in a database for future retrieval.

The design of Polar Signals Cloud is exactly analogous: it features a more performant and scalable backend, but it communicates with the Parca agent in the same way. Thus, in this post, "the backend" should be taken to mean either the Parca backend or the Polar Signals Cloud backend.

The Motivation for Offline Mode

In the past, the agent has typically communicated with the backend over the network, and a lost network connection usually implies that data collected while the network was down will not be reported. In a typical modern server workload, this is acceptable: a host losing network connectivity is a rare scenario that means the host is basically useless anyway.

But the world of computing is broader than just servers, and the Parca team would like our software to be useful in other kinds of deployments as well. In the modern world, many computerized devices are either never connected to the internet or only unreliably connected: this includes everything from smartphones to autonomous vehicles.

Thus, we decided to develop Offline Mode: a new feature for the Parca agent allowing it to save data locally and upload it for further processing later.

How It Works

Recording the Data

In traditional operation ("online" mode), the agent communicates with the backend via the following stateful protocol: first, it uploads a list of stack IDs (computed by hashing the stacks themselves) along with a count of how many times each stack ID occurred. The backend then responds with a list of IDs that it needs the full stack trace for, and finally, the agent responds with these. This allows the backend to cache stacks it's already seen, decreasing network traffic.

In offline mode, every five seconds, rather than sending anything to the backend, we record two records in a file (each prefixed with their size in bytes): first, the stack IDs and their counts; second, the full stacks for any IDs that have not yet been recorded in the same file. We then call fsync to ensure data persistence, and finally, update the count of batches in the header of the file.

This format is self-describing and resistent to crashes: since the batch count is not updated until after the batch is synced to disk, an attempt to read a partially-written file will only see atomically written batches (though it might miss an entire final batch if it was in the process of being written when the agent process terminated).

Every ten minutes, the storage file is rotated: it is compressed using ZSTD to reduce storage cost, and a new file is started. The files are saved with the scheme {timestamp}-{pid}.padata so that later they can be read in timestamp order.

Uploading the Data

Later, data may be uploaded whenever and from wherever the user chooses to do so; this does not necessarily have to be done on the same machine where the data was recorded, as long as the uploading machine has access to the storage directory where the files were written.

The uploader reads files from the storage in the order they were written (sorting using the timestamp in the filename). It uploads samples to the backend using the same protocol as the agent does during normal operation, using the full stacks (the second record in each batch) to respond to responses from the backend requesting them. After each file is successfully uploaded, the uploader removes it from the storage directory, so it can pick up where it left off if it's interrupted.

Try It Out

If you want to profile an x86-64 or aarch64 Linux installation that has reliable access to storage but not to the network, the Parca agent's offline mode might be just what you're looking for. To try it out, run parca-agent with --offline-mode-storage-path=/path/to/storage to begin collecting profiling data locally. The agent will create .padata files in the specified directory, rotating and compressing them every 10 minutes by default.

When you're ready to upload the collected data, run parca-agent with both --offline-mode-storage-path=/path/to/storage and --offline-mode-upload, along with your usual backend configuration flags (like --remote-store-address). The uploader will process all files in timestamp order and remove them after successful upload. This doesn't have to be done on the same machine as collection: nothing stops you from copying the /path/to/storage directory to anywhere that is capable of maintaining a network connection and running parca-agent.

We hope this is useful. Happy profiling!