Kubectl flame is a plugin for kubectl that allows you to profile applications in production with low overhead by generating FlameGraphs.

It is a Yahoo project.

It is installed via krew, the plugin manager for kubectl, and allows you to generate FlameGraphs for applications in Go, Java (all JVM languages), Python, Ruby, and NodeJS.

After installing krew, you can install flame via: kubectl krew install flame.

For each supported language, it will use a different profiler to generate a FlameGraph. These profilers are of sampling type, and based on this sampling information (how many times a method has been called), it will generate a FlameGraph.

FlameGraph

A FlameGraph is a way to visualize the profile of an application allowing to instantly (or very quickly) detect the most frequent code path.

It will display, on the x-axis, the population (usually the method) whose size is proportional to the number of samples in the profile,
and on the y-axis, the depth in the call stack.

More information on Brendan Gregg’s website: FlameGraphs.

Here is an example of a FlameGraph from Brendan Gregg’s site.

kubetcl without and with the flame plugin

To generate a FlameGraph of a Java application deployed in a pod-my-app pod using async-profiler, and assuming that the Java process has a pid of 1, we would need to do something close to this:

wget https://github.com/jvm-profiling-tools/async-profiler/releases/download/v2.7/async-profiler-2.7-linux-x64.tar.gz
kubectl cp async-profiler-2.7-linux-x64.tar.gz pod-my-app:/tmp
kubectl exec -ti pod-my-app -- bash
$ tar xvf /tmp/async-profiler-2.7-linux-x64.tar.gz
$ /tmp/async-profiler-2.7-linux-x64/profiler.sh -d 60 -e cpu -f /tmp/profile-cpu.html 1
$ exit
kubectl cp pod-my-app:/tmp/profile-cpu.html ./profile-cpu.html

Lots of steps, some requiring elevated privileges on the Kubernetes cluster.

With kubectl’s flame plugin, a single command is enough!

kubectl flame -t  30s --lang java -e cpu -f profile-cpu.svg pod-my-app

We can note here a small difference: the Java profiler used by kubectl flame is async-profiler in version 1 which generates FlameGraphs in SVG format, while version 2 generates them in HTML format. Kubectl flame being an open source project, I opened a PR to upgrade to async-profiler 2 but haven’t had time to finish it yet.

kubetcl flame in details

Given the following command:

kubectl flame -t  30s --lang java -e cpu -f profile-cpu.svg pod-my-app

• -t 30s : will profile the application for 30s.
• –lang java : for Java / JVM applications.
• -e cpu : profile the CPU event, all events supported by async-profiler are supported.
• -f profile-cpu.svg : will save the profile in the local file profile-cpu.svg.

The most interesting thing about kubectl flame is that we will use the same command line with the same options for the different languages it supports.

Example, for an application in Go started in the hello-go pod:

kubectl flame -t 30s --lang go -f profile-cpu.svg hello-go

When no event is passed to the command line, the default event (usually the CPU) will be profiled.

The way kubectl flame work is as follows:

• A Kubernetes Job is launched, this allows us to ensure that the profiling is running smoothly and to restart it if necessary.
• The Job will launch a Pod.
• The Pod will use a profiler image corresponding to the language, then locate the application process to be profiled (the --pgrep option allows you to specify the name of the command to search for).
• The resulting profile file will be copied locally.

For more information about kubectl flame you can check out its GitHub repo: https://github.com/yahoo/kubectl-flame.