Today we're introducing an open source continuous profiling utility called Perfessor!
How it works
It uses perf to sample CPU functions of the given process, which generates a perf.data report. Using perf data converter and pprof we convert the perf data into a pprof profile. Then we can ship that profile to conprof or to any other supported storage system like Polar Signals. Perfessor has actually very little code, and it leverages these powerful utilities.
How to get started
Next we'll walk through how to setup profiling a live Kubernetes application with Perfessor.
First we'll start by creating an API token that we can use to have Perfessor ship it's profiles to Polar Signals.
Navigate to the settings page of your project, name a token, and click create. Copy that token out to be used later.
Now that we have an API token, it's time to add Perfessor as a Kubernetes sidecar. Below you'll see the relevant section of a Kubernetes manifest that adds the sidecar container.
There are a few interesting things you can do with perfessor. You can add custom labels to profiles shipped by the sidecar.
Below you'll see the
--ship.label=app=perfessor-demo which will ship profiles with the label pair of
You can set the duration of the profiles using the
--perf.duration flag. You will need to tell it the process name to take profiles by
--perf.pname flag. In our example we are targeting the
api process in this manifest.
Now from the step before we can copy in our API token into the
To allow perfessor to see the processes in the other container in the pod we will need the
shareProcessNamespace: true setting in our manifest.
The last thing of note in this sidecar is the
securityContext. Because Perfessor uses the perf utility, which makes low-level kernel calls,
we'll need to add the SYS_ADMIN capability and run the pod as root.
spec: shareProcessNamespace: true securityContext: capabilities: add: - SYS_ADMIM runAsUser: 0 containers: - command: - /bin/perfessor - --log.level=debug - --ship.store=grpc.polarsignals.com:443 - --ship.bearer-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwcm9qZWN0SWQiOiIzODI4ODdjNy02YjYxLTRiNDUtODYxNS03ZTUwOTFkMjgyMzMiLCJ3cml0ZVByb2ZpbGVzIjp0cnVlLCJqdGkiOiI4ZWE4MjM3ZC05NzNjLTRiNDUtYWE2ZC03ZGM4ZjM0YjY4ZDQiLCJpc3MiOiJodHRwczovL2FwaS5wb2xhcnNpZ25hbHMuY29tLyJ9.hIRy8K4NiTQnxC3mbkAyXeiI95pxJc-uw2TCRrOJ990 - --ship.label=app=perfessor-demo - --ship.label=duration=10s - --perf.pname=api - --perf.duration=10s image: quay.io/conprof/perfessor:d561d3ba-linux-amd64 name: perfessor
Once we've applied our manifest Perfessor should start taking profiles and shipping them to storage! If you're using Polar Signals you can navigate to the project and start browsing the profiles.
You can see in the video below, we find our profiles in Polar Signals, with the labels specified in our manifest, as well as the process name as a label. You can see the perf command that was used to generate the profile in the graph view.
With Perfessor you can now start profiling your applications even if they don't support pprof.
Perfessor is being open sourced under the Conprof Github organization. We invite the community to help contribute to this product to bring continuous profiling to eveyrone.
If you'd like to try this out for yourself, sign up for Polar Signals private beta.