kernel/tools/perf/builtin-ftrace.c, branch linux-rolling-stable

perf ftrace: Fix hashmap__new() error checking

2026-03-19T15:14:55Z

[ Upstream commit be34705aa527872e5ce83927b7bc9307ba8095ca ] The hashmap__new() function never returns NULL, it returns error pointers. Fix the error checking to match. Additionally, set ftrace->profile_hash to NULL on error, and return the exact error code from hashmap__new(). Fixes: 0f223813edd051a5 ("perf ftrace: Add 'profile' command") Suggested-by: Ian Rogers Signed-off-by: Chen Ni Cc: Adrian Hunter Cc: Alexander Shishkin Cc: Ingo Molnar Cc: James Clark Cc: Jiri Olsa Cc: Mark Rutland Cc: Namhyung Kim Cc: Peter Zijlstra Signed-off-by: Arnaldo Carvalho de Melo Signed-off-by: Sasha Levin

perf: ftrace: add graph tracer options args/retval/retval-hex/retaddr

2025-07-23T00:47:22Z

This change adds support for new funcgraph tracer options funcgraph-args, funcgraph-retval, funcgraph-retval-hex and funcgraph-retaddr. The new added options are: - args : Show function arguments. - retval : Show function return value. - retval-hex : Show function return value in hexadecimal format. - retaddr : Show function return address. # ./perf ftrace -G vfs_write --graph-opts retval,retaddr # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 5) | mutex_unlock() { /* <-rb_simple_write+0xda/0x150 */ 5) 0.188 us | local_clock(); /* <-lock_release+0x2ad/0x440 ret=0x3bf2a3cf90e */ 5) | rt_mutex_slowunlock() { /* <-rb_simple_write+0xda/0x150 */ 5) | _raw_spin_lock_irqsave() { /* <-rt_mutex_slowunlock+0x4f/0x200 */ 5) 0.123 us | preempt_count_add(); /* <-_raw_spin_lock_irqsave+0x23/0x90 ret=0x0 */ 5) 0.128 us | local_clock(); /* <-__lock_acquire.isra.0+0x17a/0x740 ret=0x3bf2a3cfc8b */ 5) 0.086 us | do_raw_spin_trylock(); /* <-_raw_spin_lock_irqsave+0x4a/0x90 ret=0x1 */ 5) 0.845 us | } /* _raw_spin_lock_irqsave ret=0x292 */ 5) | _raw_spin_unlock_irqrestore() { /* <-rt_mutex_slowunlock+0x191/0x200 */ 5) 0.097 us | local_clock(); /* <-lock_release+0x2ad/0x440 ret=0x3bf2a3cff1f */ 5) 0.086 us | do_raw_spin_unlock(); /* <-_raw_spin_unlock_irqrestore+0x23/0x60 ret=0x1 */ 5) 0.104 us | preempt_count_sub(); /* <-_raw_spin_unlock_irqrestore+0x35/0x60 ret=0x0 */ 5) 0.726 us | } /* _raw_spin_unlock_irqrestore ret=0x80000000 */ 5) 1.881 us | } /* rt_mutex_slowunlock ret=0x0 */ 5) 2.931 us | } /* mutex_unlock ret=0x0 */ Signed-off-by: Changbin Du Reviewed-by: Ian Rogers Link: https://lore.kernel.org/r/20250613114048.132336-1-changbin.du@huawei.com Signed-off-by: Namhyung Kim

perf ftrace latency: Add -e option to measure time between two events

2025-07-15T05:51:58Z

In addition to the function latency, it can measure events latencies. Some kernel tracepoints are paired and it's menningful to measure how long it takes between the two events. The latency is tracked for the same thread. Currently it only uses BPF to do the work but it can be lifted later. Instead of having separate a BPF program for each tracepoint, it only uses generic 'event_begin' and 'event_end' programs to attach to any (raw) tracepoints. $ sudo perf ftrace latency -a -b --hide-empty \ -e i915_request_wait_begin,i915_request_wait_end -- sleep 1 # DURATION | COUNT | GRAPH | 256 - 512 us | 4 | ###### | 2 - 4 ms | 2 | ### | 4 - 8 ms | 12 | ################### | 8 - 16 ms | 10 | ################ | # statistics (in usec) total time: 194915 avg time: 6961 max time: 12855 min time: 373 count: 28 Reviewed-by: Ian Rogers Link: https://lore.kernel.org/r/20250714052143.342851-1-namhyung@kernel.org Signed-off-by: Namhyung Kim

perf target: Remove uid from target

2025-06-09T18:18:18Z

Gathering threads with a uid by scanning /proc is inherently racy leading to perf_event_open failures that quit perf. All users of the functionality now use BPF filters, so remove uid and uid_str from target. Signed-off-by: Ian Rogers Link: https://lore.kernel.org/r/20250604174545.2853620-10-irogers@google.com Signed-off-by: Namhyung Kim

perf ftrace: Use process/session specific trace settings

2025-05-20T16:11:09Z

Executing 'perf ftrace' commands 'ftrace', 'profile' and 'latency' leave tracing disabled as can seen in this output: # echo 1 > /sys/kernel/debug/tracing/tracing_on # cat /sys/kernel/debug/tracing/tracing_on 1 # perf ftrace trace --graph-opts depth=5 sleep 0.1 > /dev/null # cat /sys/kernel/debug/tracing/tracing_on 0 # The 'tracing_on' file is not restored to its value before the command. To fix that this patch uses the .../tracing/instances/XXX subdirectory feature. Each 'perf ftrace' invocation creates its own session/process specific subdirectory and does not change the global state in the .../tracing directory itself. Use rmdir(../tracing/instances/dir) to stop process/session specific tracing and delete all process/session specific setings. Reported-by: Alexander Egorenkov Suggested-by: Arnaldo Carvalho de Melo Signed-off-by: Thomas Richter Cc: Alexander Gordeev Cc: Heiko Carstens Cc: Namhyung Kim Cc: Sumanth Korikkar Cc: Vasily Gorbik Link: https://lore.kernel.org/r/20250520093726.2009696-1-tmricht@linux.ibm.com Signed-off-by: Arnaldo Carvalho de Melo

perf ftrace latency: allow to hide empty buckets

2025-02-26T21:48:02Z

Especially while using several buckets, it isn't uncommon to have some of them empty and reading the histogram may be a bit more complex: # perf ftrace latency -a -T mutex_lock --bucket-range 5 --max-latency 200 # DURATION | COUNT | GRAPH | 0 - 5 us | 14816 | ###################################### | 5 - 10 us | 1228 | ### | 10 - 15 us | 438 | # | 15 - 20 us | 106 | | 20 - 25 us | 21 | | 25 - 30 us | 11 | | 30 - 35 us | 1 | | 35 - 40 us | 2 | | 40 - 45 us | 4 | | 45 - 50 us | 0 | | 50 - 55 us | 1 | | 55 - 60 us | 0 | | 60 - 65 us | 1 | | 65 - 70 us | 1 | | 70 - 75 us | 1 | | 75 - 80 us | 2 | | 80 - 85 us | 0 | | 85 - 90 us | 1 | | 90 - 95 us | 0 | | 95 - 100 us | 1 | | 100 - 105 us | 0 | | 105 - 110 us | 0 | | 110 - 115 us | 0 | | 115 - 120 us | 0 | | 120 - 125 us | 1 | | 125 - 130 us | 0 | | 130 - 135 us | 0 | | 135 - 140 us | 1 | | 140 - 145 us | 0 | | 145 - 150 us | 0 | | 150 - 155 us | 0 | | 155 - 160 us | 0 | | 160 - 165 us | 0 | | 165 - 170 us | 0 | | 170 - 175 us | 0 | | 175 - 180 us | 0 | | 180 - 185 us | 0 | | 185 - 190 us | 0 | | 190 - 195 us | 0 | | 195 - 200 us | 0 | | 200 - ... us | 2 | | Allow the optional flag --hide-empty to remove buckets with no element and produce a more compact graph. This feature could be misleading since there is no clear indication for missing buckets, for this reason it's disabled by default. # perf ftrace latency -a -T mutex_lock --bucket-range 5 --max-latency --hide-empty 200 # DURATION | COUNT | GRAPH | 0 - 5 us | 14816 | ###################################### | 5 - 10 us | 1228 | ### | 10 - 15 us | 438 | # | 15 - 20 us | 106 | | 20 - 25 us | 21 | | 25 - 30 us | 11 | | 30 - 35 us | 1 | | 35 - 40 us | 2 | | 40 - 45 us | 4 | | 50 - 55 us | 1 | | 60 - 65 us | 1 | | 65 - 70 us | 1 | | 70 - 75 us | 1 | | 75 - 80 us | 2 | | 85 - 90 us | 1 | | 95 - 100 us | 1 | | 120 - 125 us | 1 | | 135 - 140 us | 1 | | 200 - ... us | 2 | | Signed-off-by: Gabriele Monaco Link: https://lore.kernel.org/r/20250207080446.77630-2-gmonaco@redhat.com Signed-off-by: Namhyung Kim

perf ftrace latency: variable histogram buckets

2025-02-26T21:48:02Z

The max-latency value can make the histogram smaller, but not larger, we have a maximum of 22 buckets and specifying a max-latency that would require more buckets has no effect. Dynamically allocate the buckets and compute the bucket number from the max latency as (max-min) / range + 2 If the maximum is not specified, we still set the bucket number to 22 and compute the maximum accordingly. Fail if the maximum is smaller than min+range, this way we make sure we always have 3 buckets: those below min, those above max and one in the middle. Since max-latency is not available in log2 mode, always use 22 buckets. Signed-off-by: Gabriele Monaco Link: https://lore.kernel.org/r/20250207080446.77630-1-gmonaco@redhat.com Signed-off-by: Namhyung Kim

perf ftrace: Fix display for range of the first bucket

2025-01-10T17:45:43Z

When min_latency is not given, it prints 0 - 0. It should be 0 - 1. Before: $ sudo ./perf ftrace latency -a -T do_futex sleep 1 # DURATION | COUNT | GRAPH | 0 - 0 us | 321 | ########### | ... After: $ sudo ./perf ftrace latency -a -T do_futex sleep 1 # DURATION | COUNT | GRAPH | 0 - 1 us | 699 | ############ | ... Fixes: 08b875b6bf608589 ("perf ftrace latency: Introduce --min-latency to narrow down into a latency range") Reviewed-by: Gabriele Monaco Signed-off-by: Namhyung Kim Tested-by: Arnaldo Carvalho de Melo Tested-by: Gabriele Monaco Cc: Ian Rogers Cc: Ingo Molnar Cc: Jiri Olsa Cc: Kan Liang Cc: Peter Zijlstra Link: https://lore.kernel.org/r/20250108210015.1188531-2-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf ftrace: Check min/max latency only with bucket range

2025-01-10T17:43:55Z

It's an optional feature and remains 0 when bucket range is not given. And it makes the histogram goes to the last entry always because any latency (num) is greater than or equal to 0. Before: $ sudo ./perf ftrace latency -a -T do_futex sleep 1 # DURATION | COUNT | GRAPH | 0 - 0 us | 0 | | 1 - 2 us | 0 | | 2 - 4 us | 0 | | 4 - 8 us | 0 | | 8 - 16 us | 0 | | 16 - 32 us | 0 | | 32 - 64 us | 0 | | 64 - 128 us | 0 | | 128 - 256 us | 0 | | 256 - 512 us | 0 | | 512 - 1024 us | 0 | | 1 - 2 ms | 0 | | 2 - 4 ms | 0 | | 4 - 8 ms | 0 | | 8 - 16 ms | 0 | | 16 - 32 ms | 0 | | 32 - 64 ms | 0 | | 64 - 128 ms | 0 | | 128 - 256 ms | 0 | | 256 - 512 ms | 0 | | 512 - 1024 ms | 0 | | 1 - ... s | 1353 | ############################################## | After: $ sudo ./perf ftrace latency -a -T do_futex sleep 1 # DURATION | COUNT | GRAPH | 0 - 0 us | 321 | ########### | 1 - 2 us | 132 | #### | 2 - 4 us | 202 | ####### | 4 - 8 us | 188 | ###### | 8 - 16 us | 16 | | 16 - 32 us | 12 | | 32 - 64 us | 30 | # | 64 - 128 us | 98 | ### | 128 - 256 us | 53 | # | 256 - 512 us | 57 | ## | 512 - 1024 us | 9 | | 1 - 2 ms | 9 | | 2 - 4 ms | 1 | | 4 - 8 ms | 98 | ### | 8 - 16 ms | 5 | | 16 - 32 ms | 7 | | 32 - 64 ms | 32 | # | 64 - 128 ms | 10 | | 128 - 256 ms | 10 | | 256 - 512 ms | 2 | | 512 - 1024 ms | 0 | | 1 - ... s | 0 | | Fixes: 690a052a6d85c530 ("perf ftrace latency: Add --max-latency option") Reviewed-by: Gabriele Monaco Signed-off-by: Namhyung Kim Tested-by: Arnaldo Carvalho de Melo Tested-by: Gabriele Monaco Cc: Ian Rogers Cc: Ingo Molnar Cc: Jiri Olsa Cc: Kan Liang Cc: Peter Zijlstra Link: https://lore.kernel.org/r/20250108210015.1188531-1-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf ftrace profile: Add --graph-opts option

2025-01-08T20:20:42Z

Like trace subcommand, it should be able to pass some options to control the tracing behavior for the function graph tracer. But some options are limited in order to maintain the internal behavior. For example, it can limit the function call depth like below: # perf ftrace profile --graph-opts depth=5 -- myprog Committer testing: root@number:~# perf ftrace profile --graph-opts thresh=1000 -- sleep 1 # Total (us) Avg (us) Max (us) Count Function 1001419.301 500709.650 1000032.000 2 x64_sys_call 1000032.000 1000032.000 1000032.000 1 __x64_sys_clock_nanosleep 1000032.000 1000032.000 1000032.000 1 common_nsleep 1000031.000 1000031.000 1000031.000 1 do_nanosleep 1000031.000 1000031.000 1000031.000 1 hrtimer_nanosleep 1000024.000 1000024.000 1000024.000 1 schedule 1387.208 1387.208 1387.208 1 __x64_sys_execve 1386.691 1386.691 1386.691 1 do_execveat_common.isra.0 1334.170 1334.170 1334.170 1 bprm_execve 1258.413 1258.413 1258.413 1 load_elf_binary 1123.068 1123.068 1123.068 1 begin_new_exec 1113.550 1113.550 1113.550 1 mmput 1109.237 1109.237 1109.237 1 exit_mmap root@number:~# perf ftrace profile --graph-opts thresh=1200 -- sleep 1 # Total (us) Avg (us) Max (us) Count Function 1001448.204 500724.102 1000018.000 2 x64_sys_call 1000017.000 1000017.000 1000017.000 1 __x64_sys_clock_nanosleep 1000017.000 1000017.000 1000017.000 1 common_nsleep 1000017.000 1000017.000 1000017.000 1 hrtimer_nanosleep 1000016.000 1000016.000 1000016.000 1 do_nanosleep 1000012.000 1000012.000 1000012.000 1 schedule 1430.112 1430.112 1430.112 1 __x64_sys_execve 1429.581 1429.581 1429.581 1 do_execveat_common.isra.0 1376.289 1376.289 1376.289 1 bprm_execve 1301.743 1301.743 1301.743 1 load_elf_binary root@number:~# Reviewed-by: James Clark Signed-off-by: Namhyung Kim Tested-by: Arnaldo Carvalho de Melo Cc: Adrian Hunter Cc: Ian Rogers Cc: Ingo Molnar Cc: Jiri Olsa Cc: Kan Liang Cc: Peter Zijlstra Link: https://lore.kernel.org/r/20250107224352.1128669-2-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo