Our new and faster profiler dashboard
Learn more about our new and improved profiler dashboard.
We've been working hard at a revamp of our profiling dashboard and we wanted to share some of our learnings and improvements we've observed along the way, backed by numbers.
This is not the first time we have written about making flame graphs fast. Last year we parallelised the SVG render and took the initial render of a large query from around eight seconds to about 91 milliseconds.
But SVG has a ceiling, and that ceiling is made up of DOM nodes. Every frame in a flame graph is a <rect> and a <text>. Therefore a large profile is tens of thousands of frames, which is tens of thousands of DOM nodes and for each node the browser has to style, lay out, paint and keep in a tree it can later garbage collect. To stop that from freezing the tab, the old renderer drew nodes in batches of roughly 500 per frame. This worked, but batching the DOM is only just treating a symptom. The fundamental problem here was that we were asking the browser to manage an enormous amount of structure.
To be honest, we more or less said it at the end of that post: we suspected the next real gain was to stop using the DOM for the flame graph, even though we were unsure as to how much Canvas would actually buy us. This post is us answering our own open question.
tl:dr Why is the new profiler dashboard faster?
The old renderer was drawn with SVG, one DOM element and a small JavaScript object for every one of its tens of thousands of frames, all of which the browser then had to style, lay out, paint, and garbage-collect.
The new one draws the same data on a Canvas: it reads the numbers straight from their columns and paints pixels, with no per-frame element or object at all. This makes zooming, resetting, and scrolling feel instant, because none of them touch the DOM any more.
A flame graph is made up of frames
Let's go through the differences between the SVG and Canvas HTML elements with respect to rendering flame graphs.
The SVG and the DOM are retained-mode APIs in the browser while the Canvas is an immediate-mode API. What does this mean?
In retained-mode, the browser is handed a tree of elements and it owns them. It is responsible for handling layout changes and painting for every element, and because they live in a tree, any change forces a re-computation of styles and layout even for other elements in the tree i.e. the browser keeps a model of what you drew and re-renders it for you.
This works well for static documents but can be a lot of bookkeeping for a complex component (like a flame graph) that is essentially thousands of coloured frames with text on them.
(If you write React, this is the world you already live in: you re-describe the UI, and React keeps a DOM tree in sync with it for you. That is retained mode with a friendly face on top.)
In immediate-mode API, there is no tree. Instead, there is a drawing context, and you can issue draw commands like fillRect or fillText, and the browser turns them into pixels and forgets about it. With no nodes to style, or boxes to lay out, there is nothing left behind for the garbage collector and the browser's background chore of reclaiming memory nobody is using any more is eliminated.
For us especially, the cost of rendering the flame graph in Canvas changes from scaling with the number of frames the browser has to remember to scaling with how many you actually draw.
The flame graph is now a SQL query
When we rewrote our querier and database in Rust, the flame graph became the output of a composable function you can call in SQL. The querier walks the samples, builds the tree, computes every frame's position and width, and streams the finished thing to us as render-ready, columnar Arrow.
If you're not familiar with columnar Arrow, it essentially means the data is sent back to us as a table, where instead of multiple objects each containing a frame's x, width, and depth together, it keeps each field in its own array. All the x-positions in one array, all the widths in another, all the depths in another.
Now rendering the flame graph means doing the same small job thousands of times over: take one frame's left edge, its width, and its row, draw it, move on to the next. Because all the left edges already sit together in a single array, reading them is just going straight down a list instead of opening an object for every frame to pull out the one number it needs.
The columnar Arrow records come over Arrow Flight (Arrow's own network transport) in its IPC format. The cool thing about this is, Arrow's layout on the wire is identical to its layout in memory, so once the data arrives, a tableFromIPC function easily gives us a usable table without parsing them or copying them into anything new.
This is "zero-copying", and it is a strategy we always did our best to adhere to when getting data over the wire and rendering it. By doing this, we were setting ourselves up for fast canvas draws, because the flame graph tree is built and the layout is done before we get the data back.
Drawing on Canvas
With positioning precomputed, the draw loop is quite trivial. For each frame we need to paint, we read its fields straight out of the Arrow columns and map them to canvas coordinates:
This is where zero-copy earns its keep. By simply reading a column, we see everything we need to know about that frame. We never have to spend additional time building JavaScript objects and SVG elements for every single frame, which then had to be tracked and reclaimed by the browser and the garbage collector.
We also retained the viewport culling strategy we had in our previous flame graph implementation i.e. only drawing what can be seen on screen. The difference here is, the culling strategy decides how many fillRect calls are made rather than how many DOM nodes we mount. If a profile has 50,000 frames and only 800 of them fall inside the visible region, we issue roughly 800 draw calls, not 50,000.
What about scrolling and zooming
Drawing the flame graph quickly is only half the job. The interactions that used to hurt most were the ones after that first paint: scrolling a deep stack, zooming into a hot path, resetting back out. They are cheap now for two reasons, and both come back to Canvas being a better fit for the job.
First, we never draw the whole graph at once. A flame graph can be far taller than your screen, and a canvas can't be made endlessly tall anyway, so we size it to just a bit more than the visible window. As you scroll, most of the time we do not redraw at all: we slide those already-painted pixels the way any web page scrolls, and only repaint a fresh slice when you move past that margin.
Second, zooming does not rebuild anything. Every frame's position and width was worked out once already, back in the query, so zooming just re-projects a narrower slice of those fixed numbers across the canvas, and resetting projects the full range back out. There is no tree to walk a second time, and nothing to measure again.
Before we move on, here is the canvas renderer running live.
What we measured
Internally, the feedback we kept getting was that the new profiler "feels faster." That is the best kind of feedback to receive and the worst kind to put in a blog post: it does not tell you what got faster, whether the win is in the query or the render, or whether the flame graph is simply smaller this time. So we measured it three ways: browser automation for what a user actually feels, fixture replay to isolate the renderer, and Chrome DevTools traces to see where the browser spends its time.
The number we trust most comes from fixture replay. We save the exact Arrow response for one profile (a two-day on-CPU query, deep and large), then mount the old SVG renderer and the new Canvas renderer against those same bytes and time only the render, with no server and no network. The two renderers build slightly different trees (17,889 frames versus 16,515, because they group stacks a little differently), so we hold the total sample count identical across the pair and compare per-frame cost as well as total. The browser runs told the same story with more noise, and the traces showed why: the old path spends its time in style, layout, and paint work that the canvas renderer simply never does.
Show me the numbers
The isolated render
On the same profile of roughly 17,000 frames, isolated in fixture replay, the SVG renderer takes a median of 164 milliseconds to draw the flame graph. The Canvas renderer takes 46 milliseconds. That is about 3.6× faster, and about 3.3× less time per frame, 9.2 microseconds (µs) down to 2.8µs.
The per-frame figure is just render time divided by frame count, which is a way to compare two trees that differ by 8% in size.
On a real page
For the live browser run, it was worth asking what happens in the actual product, where the query, the network, and the rest of the app are all in the mix and what the differences are between the old and new profiler page.
So we ran the same before-and-after in a real browser, on the live old and new pages, and timed the things one actually does in a flame graph.
Zooming into a hot path went from about 121 ms to 29 ms, roughly four times quicker. Scrolling redraws dropped from around 30 ms to 6 ms, four to five times quicker. And resetting the zoom, the worst offender on the old page, went from about 528 ms to 47 ms.
One caveat about the render and first-render numbers: isolated in the fixture replay they came out at about 3.6×, not 14×.
The difference is not the renderer improving between the two tests; it is the old renderer under contention. On a live page it competes with everything else for the single main thread, so its render time balloons and swings from run to run, anywhere from about 450 ms to over a second. The 3.6× is the fair, repeatable figure; the live number is what a user feels on a busy page, not a stable benchmark.
What about interactions?
Flamegraphs are rich in interactions too, so we measured those as well. We ran the same scripted hover sweep over both renderers with the CPU throttled 15× (DevTools throttling on an M4 Mac) to simulate a slower, busier machine, and measured the frames the page produced.
The canvas renderer's hover tooltip never stutters, even on the heavily throttled environment. It holds 118.5 fps, with a single dropped frame and a p95 of 9 ms.
The video shows one of these 15×-throttled runs side by side. Chrome's own FPS meter sits top-left of each pane.
Trade offs
Canvas is faster because the browser does less for you. Which means that everything the DOM did for free is now your problem. This is the part we were genuinely unsure about last year, so let's touch on them a little bit.
Hit-testing. With no elements, there is nothing to receive a click or a hover. So when the pointer moves on the flame graph, we invert the same maths we drew with: the row falls out of the mouse y divided by the row height, and then a quick left-to-right scan across that row's frames finds the one under the cursor. Essentially the same function responsible for computing a frame's rectangle in order to draw it is the one hit-testing calls to decide what sits under the cursor. So what you see and what you can click are the same.
Hover and tooltips. There is no :hover, so we track the hovered frame ourselves and render the tooltip on top of the canvas. React still owns the tooltip's contents; the canvas just tells it which frame and where.
Text. There is no text-overflow: ellipsis. We measure every label and truncate it ourselves before drawing, so a long symbol does not bleed out of its frame.
High-DPI screens. A canvas is a bitmap, so we scale its backing store by devicePixelRatio and draw at that scale, or the whole graph renders soft and blurry on a retina display.
Accessibility and text selection. This one was a loss we had to bear as a canvas is opaque to a screen reader, and you cannot select text off it the way you could off SVG. We don't fully have a solution for this yet but we'll continue to look for one.
Conclusion
If you have a profile that used to feel heavy, we would love for you to open it again in the Polar Signals dashboard and tell us how it feels now.
Happy profiling!
Read more
Parallel Flame Graph Rendering
How we improved our Flame Graph rendering by moving computation to the backend.
Ice and Fire: How to read icicle and flame graphs
Flame graphs and icicle graphs are a great way to visualize performance profiles. In this post, we will learn how to read and interpret them.
Keep up with Polar Signals
Receive new posts, product updates, and insights on performance engineering straight to your inbox.