Towards the end of last year, we open sourced our continuous profiler Parca and wrote about it in our announcement blog post. We have had a great response from the community, and tons of improvements have been added, many of them contributed by the community. Caught up in all of these exciting changes, however, we overlooked that we are missing a canonical "Introduction to Parca" blog post. We want to take this space to fix that now.
- Outlines the core concepts behind various components in the Parca project
- Provides a hands-on tutorial for setting Parca up from its binary and using it to profile your system
We will discuss Parca Agent in part II of this blog post series.
What is Parca?
We have plenty of tools to capture logs, traces, and metrics- often called the three pillars of observability. However, there is a need for more correlation between these three pillars to enable us to cut through noisy signals. Parca was born out of this need.
Measuring the state of your application by capturing CPU/memory profiles or snapshots of the application is known as sampling profiling. But instead of using discrete profiles, Parca monitors the state continuously over time. Continuous sampling profiling carries very little overhead, making it suitable for production environments.
These profiles can then be visualised and queried to drill down on resource-heavy areas of your code.
Parca comprises two components:
- Parca Server
- Parca Agent
Parca agent implements the sampling profiler which leverages eBPF to continuously collect raw profiling data for both user-space and kernel-space applications. It discovers targets to profile such as Kubernetes containers or systemd units.
The Parca server stores and symbolises the profiles collected by the agent, and allows for querying over time.
Parca Quickstart Tutorial
This tutorial covers:
- setting up the Parca server to profile itself
Now, let's try to profile our systems and see Parca in action. Download the latest server binary release for your architecture.
curl -sL https://github.com/parca-dev/parca/releases/download/v0.11.1/parca_0.11.1_`uname -s`_`uname -m`.tar.gz | tar xvfz - parca
Download the example configuration from our repo:
curl -sL https://raw.githubusercontent.com/parca-dev/parca/release-0.11/parca.yaml > parca.yaml
Start `parca` with :
You should see Parca start up with the server on port 7070. As specified in the yaml configuration, Parca scrapes its own pprof endpoints and is configured to profile itself every few seconds.
Navigate to the browser at localhost:7070. Click on `Select profile` on the top right- a dropdown menu should appear. Selecting any profile type and clicking `Search` will retrieve the trace for the profile for the last one hour. This duration is configurable from the dashboard.
Clicking at any point in the profile will render an icicle graph for the profile at the corresponding point in time.
For a more aggregate view of the function names, Parca also offers a `Table view`.
Clicking on the `Both` button displays the icicle graph and table view adjacently for better correlation.
If you go to the top left of the screen, next to the `Profile` button, you will also notice a `Targets` button. Clicking on it redirects us to a table with the health status and labels for targets discovered by Parca.
This is not all, however. Often, to understand how code changes and environments affect performance, it is necessary to be able to compare two profiles over time.
This is exactly what Parca's `Compare` feature does.
Clicking on the `Compare` button splits the dashboard and lets you select two different points in time to be compared and generates an icicle graph of the difference between the selected profiles.
You should see an icicle graph color code to highlight relative changes in performance. The lavender bits signify no change in performance. Performance improvements are represented by the green parts while the red bits signify parts where the performance got worse. If you select the `Table View` while comparing profiles, you will also notice a new `Diff` column which gives us the relative difference between stack trace calls of both profiles.
Parca also has the functionality to combine all profile samples for a query into a single report with the `Merge` feature.
Profiles before merging
Here, we can see symbolised stack traces for all services over time!
Profiles after merging
Parca under the hood
Now that we have played with the Parca UI, let's dive into some of the concepts behind its features.
Parca can collect profiles using either a push mode or a pull mode or both.
- `Push Mode`: Profiles can be pushed to the Parca server by the parca agent through the server's gRPC API.
- `Pull Mode`: The server can pull pprof profiles in by scraping HTTP endpoints that are then exposed by targets.
Ingested profiles are stored in Parca's metastore. These ingested profiles comprise stack traces that contain machine addresses. We symbolise these machine addresses into human-readable symbols, i.e. functions, and variable names, enabling us to see locations and resource usage for every part of our applications’ code. Symbolisation is performed asynchronously with respect to profile ingestion.
The stack traces sent over by Parca agent contain memory address locations that are not meaningful to us as humans. We need to have additional debug information to translate the memory addresses to human-readable symbols.
Symbolisation is enabled by fetching debug information for binaries and shared objects. Debug information, also referred to as debuginfos, can be ELF object files, DWARF debug data and source code.
Often, application packages distributed by various Linux distros strip away debug information to minimise the size of the binaries. There are publicly accessible debuginfod servers, distributing debug information for various Linux package managers and distributions. Debuginfos can be obtained from either Parca's symbol store, upstream debuginfod servers, or even private debuginfod servers added by the user.
Users can add private debuginfod servers to be queried through the `--debuginfod-upstream-servers` flag in Parca.
Sometimes, CI pipelines strip away debug information in releases to minimise the size of the binaries. The Parca agent also provides a `debug-info` CLI tool for users to extract or upload their debug information separately for their binaries.
Usage: parca-debug-info Flags: -h, --help Show context-sensitive help. --log-level="info" Log level. --temp-dir="tmp" Temporary directory path to use for object files. Commands: upload --store-address=STRING <path> ... Upload debug information files. extract <path> ... Extract debug information. Run "parca-debug-info --help" for more information on a command.
An in-depth usage guide for the `debug-info` CLI can be found here.
Storing and Querying Profiles
Parca has two storage components: A metastore and a sample store.
Stack traces in Parca are represented by an ID associated with their locations. A location references the combination of mapping and function(multiple functions in case of inlined functions) with a concrete line number, and memory address if the location came from a compiled binary.
Profile samples, represented by these IDs are stored in the samplestore. The profile series ingested by Parca are identified by their unique labels which are stored against the respective IDs in the samplestore. The samplestore is implemented through an embedded columnar database, FrostDB, which was originally created keeping Parca’s specific Observability needs in mind. You can read more about FrostDB here.
The metastore stores the metadata associated with stacktrace IDs, which are- function names, object file mappings, and locations. Parca uses the embedded key-value store, badger, for the metastore.
Parca currently supports CPU profiling using the Parca agent for all compiled languages like C/C++, Rust, Go etc. All languages with pprof compatible profilers (such as those built into the Go runtime) work with Parca using a pull-based approach to scrape the pprof HTTP endpoints to collect profile snapshots.
Symbolisation for some JIT compilers like JVM, Erlang/Elixir, and NodeJS can be achieved with Parca by using perf-maps. Out-of-the-box support for dynamic languages symbolisation is on our current roadmap and the issues are being tracked here.
We understand that being able to share profile snapshots through a URL is often the key to better collaboration, whether within your organisation or with the Parca team to get the best out of our observability tools. So we built pprof.me for you to share your profiles.
Have more questions? Or something you would like us to improve? Come say hi on the Parca discord!
- Want to try Parca without setting it up? Check out our demo instance!
- More notes on configuring Parca.
- Want to contribute to Parca? Start here!
We will explore Parca Agent in the next part of this blog series, stay tuned!