Perfessor - Continuous Profiling Without Instrumentation

Continuously profiling your applications without having to make modifications!

Mar 31, 2021 | Thor Hansen

Today we're introducing an open source continuous profiling utility called Perfessor!

Perfessor is a sidecar utililty that uses perf to continuously profile applications, converts them into pprof format, and ships them to a storage backend.

How it works

It uses perf to sample CPU functions of the given process, which generates a perf.data report. Using perf data converter and pprof we convert the perf data into a pprof profile. Then we can ship that profile to conprof or to any other supported storage system like Polar Signals. Perfessor has actually very little code, and it leverages these powerful utilities.

How to get started

Next we'll walk through how to setup profiling a live Kubernetes application with Perfessor.

First we'll start by creating an API token that we can use to have Perfessor ship it's profiles to Polar Signals.

Navigate to the settings page of your project, name a token, and click create. Copy that token out to be used later.

Now that we have an API token, it's time to add Perfessor as a Kubernetes sidecar. Below you'll see the relevant section of a Kubernetes manifest that adds the sidecar container.

There are a few interesting things you can do with perfessor. You can add custom labels to profiles shipped by the sidecar. Below you'll see the --ship.label=app=perfessor-demo which will ship profiles with the label pair of app=perfessor-demo

You can set the duration of the profiles using the --perf.duration flag. You will need to tell it the process name to take profiles by using the --perf.pname flag. In our example we are targeting the api process in this manifest.

Now from the step before we can copy in our API token into the --ship.bearer-token flag.

To allow perfessor to see the processes in the other container in the pod we will need the shareProcessNamespace: true setting in our manifest. The last thing of note in this sidecar is the securityContext. Because Perfessor uses the perf utility, which makes low-level kernel calls, we'll need to add the SYS_ADMIN capability and run the pod as root.

spec:
shareProcessNamespace: true
securityContext:
capabilities:
add:
- SYS_ADMIM
runAsUser: 0
containers:
- command:
- /bin/perfessor
- --log.level=debug
- --ship.store=grpc.polarsignals.com:443
- --ship.bearer-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwcm9qZWN0SWQiOiIzODI4ODdjNy02YjYxLTRiNDUtODYxNS03ZTUwOTFkMjgyMzMiLCJ3cml0ZVByb2ZpbGVzIjp0cnVlLCJqdGkiOiI4ZWE4MjM3ZC05NzNjLTRiNDUtYWE2ZC03ZGM4ZjM0YjY4ZDQiLCJpc3MiOiJodHRwczovL2FwaS5wb2xhcnNpZ25hbHMuY29tLyJ9.hIRy8K4NiTQnxC3mbkAyXeiI95pxJc-uw2TCRrOJ990
- --ship.label=app=perfessor-demo
- --ship.label=duration=10s
- --perf.pname=api
- --perf.duration=10s
image: quay.io/conprof/perfessor:d561d3ba-linux-amd64
name: perfessor

Once we've applied our manifest Perfessor should start taking profiles and shipping them to storage! If you're using Polar Signals you can navigate to the project and start browsing the profiles.

You can see in the video below, we find our profiles in Polar Signals, with the labels specified in our manifest, as well as the process name as a label. You can see the perf command that was used to generate the profile in the graph view.

With Perfessor you can now start profiling your applications even if they don't support pprof.

What's next?

Perfessor is being open sourced under the Conprof Github organization. We invite the community to help contribute to this product to bring continuous profiling to eveyrone.

If you'd like to try this out for yourself, sign up for Polar Signals private beta.


Sign up for the latest Polar Signals news


More posts like this one

Continuous Profiler Cumulative Metrics

Our first major feature since our product launch: cumulative profiling metrics!

Mar 23, 2021 | Frederic Branczyk

Read more →

Demystifying Pprof Labels with Go

April 13, 2021 | Frederic Branczyk

Read more →

DIY pprof profiles using Go

August 3, 2021 | Matthias Loibl

Read more →