Perf: Profiling on machine A and investigating on machine B

Perf setup

Apt (Debian, Ubuntu)

• On newer systems:

sudo apt install linux_perf

• On older systems:

sudo apt install linux-tools-$(uname -r)

DNF (Fedora)

dnf install perf

Ubuntu perf-archive setup

On Ubuntu 'perf-archive' script is not installed by default.

• Get perf-archive.sh from linux sources https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/perf/perf-archive.sh
• Save it as perf-archive (note no .sh) to $(perf --exec-path)/perf-archive. Typically /usr/libexec/perf-core/perf-archive.
• Make it executable chmod +x /usr/libexec/perf-core/perf-archive

Profile

Machine A - Profile

- Gather perf record

perf record ...

Creates perf.data

- Archive, to get all the binaries with the symbols

perf archive

Additionally creates perf.data.tar.bz2 archive.

- Distribute

Copy perf.data and perf.data.tar.bz2 to machine B

Machine B - Analyze

- Unpack archive

Create ~/.debug directory

mkdir ~/.debug

And unpack the archive there

perf archive --unpack perf.data.tar.bz2

This will unpack all the gatherd binaries to ~/.debug directory. In case you have several reports from the same machine A you can call unpack on all of the perf archives at once. In case you have a report from a different machine C unpacking archive from different machines into ~/.debug at the same time could still work. Otherwise cleanup ~/.debug directory before unpacking archive from a different machine.

- Analyze the report

No you can analyze the report from machine A on machine B.

perf report

Debug symbols for stripped binaries

In case the binary you analyze is stripped from debug symbols, you won't see symbol names in perf report, only the addresses.

• Get unstripped version of a binary
• Place unstripped version of a binary to the same location on machine B that it was on machine A.
• Now running perf report should show you debug symbols.

Troubleshooting

• Sometimes cleaning up of ~/.debug is necessary
• perf archive --unpack could fail if archive has a long name. Rename to perf.data.tar.bz2 before unpacking.

Useful commands for perf

• Report function SELF time (time without children), use Intel assembly syntax:

perf report -M intel --no-children

• See difference between 2 perf reports:

perf diff perf_1.data perf_2.data