redonkable
/
alistair23-linux
1
0
Fork 0
Code Releases Activity
alistair23-linux/tools/perf/util
History
Joseph Schuchart c0268e8d1f perf script python: Fix mem leak due to missing Py_DECREFs on dict entries 
We are using the Python scripting interface in perf to extract kernel
events relevant for performance analysis of HPC codes. We noticed that
the "perf script" call allocates a significant amount of memory (in the
order of several 100 MiB) during it's run, e.g. 125 MiB for a 25 MiB
input file:

  $> perf record -o perf.data -a -R -g fp \
       -e power:cpu_frequency -e sched:sched_switch \
       -e sched:sched_migrate_task -e sched:sched_process_exit \
       -e sched:sched_process_fork -e sched:sched_process_exec \
       -e cycles  -m 4096 --freq 4000
  $> /usr/bin/time perf script -i perf.data -s dummy_script.py
  0.84user 0.13system 0:01.92elapsed 51%CPU (0avgtext+0avgdata
  125532maxresident)k
  73072inputs+0outputs (57major+33086minor)pagefaults 0swaps

Upon further investigation using the valgrind massif tool, we noticed
that Python objects that are created in trace-event-python.c via
PyString_FromString*() (and their Integer and Long counterparts) are
never free'd.

The reason for this seem to be missing Py_DECREF calls on the objects
that are returned by these functions and stored in the Python
dictionaries. The Python dictionaries do not steal references (as
opposed to Python tuples and lists) but instead add their own reference.

Hence, the reference that is returned by these object creation functions
is never released and the memory is leaked. (see [1,2])

The attached patch fixes this by wrapping all relevant calls to
PyDict_SetItemString() and decrementing the reference counter
immediately after the Python function call.

This reduces the allocated memory to a reasonable amount:

  $> /usr/bin/time perf script -i perf.data -s dummy_script.py
  0.73user 0.05system 0:00.79elapsed 99%CPU (0avgtext+0avgdata
  49132maxresident)k
  0inputs+0outputs (0major+14045minor)pagefaults 0swaps

For comparison, with a 120 MiB input file the memory consumption
reported by time drops from almost 600 MiB to 146 MiB.

The patch has been tested using Linux 3.8.2 with Python 2.7.4 and Linux
3.11.6 with Python 2.7.5.

Please let me know if you need any further information.

[1] http://docs.python.org/2/c-api/tuple.html#PyTuple_SetItem
[2] http://docs.python.org/2/c-api/dict.html#PyDict_SetItemString

Signed-off-by: Joseph Schuchart <joseph.schuchart@tu-dresden.de>
Reviewed-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Tom Zanussi <tom.zanussi@linux.intel.com>
Link: http://lkml.kernel.org/r/1381468543-25334-4-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
 		2013-10-24 10:16:54 -03:00
..
include perf tools: Default to cpu// for events v5 2013-07-12 13:53:52 -03:00
scripting-engines perf script python: Fix mem leak due to missing Py_DECREFs on dict entries 2013-10-24 10:16:54 -03:00
PERF-VERSION-GEN perf tools: Fix perf version generation 2013-07-08 18:09:52 -03:00
abspath.c perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
alias.c perf tools: Use __maybe_used for unused variables 2012-09-11 12:19:15 -03:00
annotate.c perf annotate: Fix objdump line parsing offset validation 2013-09-19 11:32:17 -03:00
annotate.h perf tools: Remove dependency on libnewt 2013-04-01 12:23:21 -03:00
bitmap.c perf tools: Add bitmap_or function into bitmap object 2012-02-13 23:28:10 -02:00
build-id.c perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
build-id.h perf tools: Move BUILD_ID_SIZE into build-id object 2012-10-29 11:35:32 -02:00
cache.h perf ui: Always compile browser setup code 2012-11-14 16:53:03 -03:00
callchain.c perf callchain: Remove unnecessary validation 2013-08-29 15:11:29 -03:00
callchain.h perf callchain: Remove unnecessary validation 2013-08-29 15:11:29 -03:00
cgroup.c perf tools: Use __maybe_used for unused variables 2012-09-11 12:19:15 -03:00
cgroup.h perf tool: Add cgroup support 2011-02-16 13:30:48 +01:00
color.c perf tools: Use scnprintf where applicable 2012-03-14 12:36:19 -03:00
color.h perf hist: Replace ->print() routines by ->snprintf() equivalents 2010-04-02 16:28:15 -03:00
config.c perf tools: Use __maybe_used for unused variables 2012-09-11 12:19:15 -03:00
cpumap.c perf stat: Add per-core aggregation 2013-03-25 16:13:26 -03:00
cpumap.h perf tools: Rename cpu_map__all() to cpu_map__empty() 2013-07-12 13:45:44 -03:00
ctype.c perf tools: ctype.c only wants util.h 2012-02-13 23:17:40 -02:00
debug.c perf tools: Fix TUI helpline output 2012-12-09 08:46:07 -03:00
debug.h perf evsel: Introduce perf_evsel__open_strerror method 2013-01-24 16:40:09 -03:00
dso.c perf symbols: Add support for reading from /proc/kcore 2013-08-07 17:35:33 -03:00
dso.h perf symbols: Add support for reading from /proc/kcore 2013-08-07 17:35:33 -03:00
dwarf-aux.c perf probe: Fix to find line information for probe list 2013-10-04 15:16:05 -03:00
dwarf-aux.h perf probe: Fix to find line information for probe list 2013-10-04 15:16:05 -03:00
environment.c
…
event.c perf: Disable PERF_RECORD_MMAP2 support 2013-10-17 16:27:14 -03:00
event.h perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
evlist.c perf evlist: Fix id pos in perf_evlist__open() 2013-09-09 15:51:44 -03:00
evlist.h perf tests: Add 'keep tracking' test 2013-09-02 14:58:20 -03:00
evsel.c perf: Disable PERF_RECORD_MMAP2 support 2013-10-17 16:27:14 -03:00
evsel.h perf tools: Add support for PERF_SAMPLE_IDENTIFIER 2013-08-29 16:09:31 -03:00
exec_cmd.c perf tools: Makefile: Remove various and sundry cruft 2011-02-18 07:43:06 -02:00
exec_cmd.h perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
generate-cmdlist.sh perf tools: Convert to LIBELF_SUPPORT 2012-09-28 21:07:36 -03:00
header.c perf session: Fix infinite loop on invalid perf.data file 2013-10-04 15:17:46 -03:00
header.h perf header: Recognize version number for perf data file 2013-07-17 17:04:00 -03:00
help.c perf tools: Use __maybe_used for unused variables 2012-09-11 12:19:15 -03:00
help.h perf tools: Protect header files with a consistent style 2009-09-24 21:27:51 +02:00
hist.c perf session: Check for SIGINT in more loops 2013-09-19 11:32:17 -03:00
hist.h perf tools: Move hist_entry__period_snprintf into stdio code 2013-07-12 13:54:18 -03:00
hweight.c perf tools: Don't use code surrounded by __KERNEL__ 2010-05-02 12:00:44 -03:00
intlist.c perf tools: Allow passing a list to intlist__new 2013-01-24 16:40:53 -03:00
intlist.h perf tools: Allow passing a list to intlist__new 2013-01-24 16:40:53 -03:00
levenshtein.c
…
levenshtein.h perf tools: Protect header files with a consistent style 2009-09-24 21:27:51 +02:00
machine.c perf machine: Fix path unpopulated in machine__create_modules() 2013-09-25 12:48:24 -03:00
machine.h perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
map.c perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
map.h perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
pager.c perf tools: Fix pager on minimal-install embedded systems 2012-05-30 15:10:39 -03:00
parse-events.c perf list: Skip unsupported events 2013-09-02 14:58:21 -03:00
parse-events.h perf parse events: Demystify memory allocations 2013-07-12 13:52:05 -03:00
parse-events.l perf tools: Add support for PERF_COUNT_SW_DUMMY 2013-09-02 14:58:20 -03:00
parse-events.y perf parse events: Demystify memory allocations 2013-07-12 13:52:05 -03:00
parse-options.c perf tools: Long option completion support for each subcommands 2012-10-02 18:36:44 -03:00
parse-options.h perf tools: Long option completion support for each subcommands 2012-10-02 18:36:44 -03:00
path.c perf tools: Convert to HAVE_STRLCPY 2012-10-02 18:36:24 -03:00
perf_regs.h perf tools: Convert to LIBUNWIND_SUPPORT 2012-10-02 17:56:01 -03:00
pmu.c perf list: List kernel supplied event aliases 2013-07-12 13:53:53 -03:00
pmu.h perf list: List kernel supplied event aliases 2013-07-12 13:53:53 -03:00
pmu.l perf tools: Add perf pmu object to access pmu format definition 2012-03-16 14:29:35 -03:00
pmu.y perf tools: Fix build with bison 2.3 and older. 2013-02-14 16:12:34 -03:00
probe-event.c perf tools: Introduce tools/lib/lk library 2013-03-15 13:06:00 -03:00
probe-event.h perf probe: Provide perf interface for uprobes 2012-05-11 13:58:17 -03:00
probe-finder.c perf probe: Fix to initialize fname always before use it 2013-10-15 14:37:38 -03:00
probe-finder.h perf probe: Fix probing symbols with optimization suffix 2013-09-25 12:27:31 -03:00
pstack.c perf tools: Stop using 'self' in pstack 2012-11-14 16:50:38 -03:00
pstack.h perf ui: Move hists browser to util/ui/browsers/ 2010-08-10 16:11:08 -03:00
python-ext-sources perf tools: Introduce tools/lib/lk library 2013-03-15 13:06:00 -03:00
python.c perf tools: Add support for PERF_COUNT_SW_DUMMY 2013-09-02 14:58:20 -03:00
quote.c perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
quote.h perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
rblist.c Account the nr_entries in rblist properly 2012-10-24 14:20:11 -02:00
rblist.h perf tools: Introducing rblist 2012-08-03 10:37:21 -03:00
record.c perf tools: Add support for PERF_SAMPLE_IDENTIFIER 2013-08-29 16:09:31 -03:00
run-command.c perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
run-command.h perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
session.c perf session: Fix infinite loop on invalid perf.data file 2013-10-04 15:17:46 -03:00
session.h perf session: Check for SIGINT in more loops 2013-09-19 11:32:17 -03:00
setup.py perf tools: Merge all *CFLAGS* make variable into CFLAGS 2013-05-30 16:12:03 +03:00
sigchain.c perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
sigchain.h perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
sort.c perf tools: Move weight back to common sort keys 2013-07-22 16:58:28 -03:00
sort.h perf tools: Move weight back to common sort keys 2013-07-22 16:58:28 -03:00
stat.c perf stats: Add max and min stats 2013-08-07 17:35:26 -03:00
stat.h perf stats: Add max and min stats 2013-08-07 17:35:26 -03:00
strbuf.c perf tools: Fix strbuf_addf() when the buffer needs to grow 2012-10-30 10:32:56 -02:00
strbuf.h perf tools: Remove some unused functions 2010-05-18 23:03:35 -03:00
strfilter.c perf: Fix missing strndup declaration 2011-03-04 01:17:18 +01:00
strfilter.h perf tools: Add strfilter for general purpose string filter 2011-01-28 09:19:38 -02:00
string.c perf tools: Default to cpu// for events v5 2013-07-12 13:53:52 -03:00
strlist.c perf probe: Fix segfault 2013-03-13 17:00:33 -03:00
strlist.h perf tools: Stop using 'self' in strlist 2013-01-25 12:49:28 -03:00
svghelper.c perf timechart: Fix black idle boxes in the title 2011-02-28 08:56:14 +01:00
svghelper.h perf tools: Protect header files with a consistent style 2009-09-24 21:27:51 +02:00
symbol-elf.c Revert "perf symbols: Demangle cloned functions" 2013-09-29 16:12:54 +02:00
symbol-minimal.c perf symbols: Add support for reading from /proc/kcore 2013-08-07 17:35:33 -03:00
symbol.c perf tools: Fix symbol offset computation for some dsos 2013-09-02 14:58:21 -03:00
symbol.h perf symbols: Add support for reading from /proc/kcore 2013-08-07 17:35:33 -03:00
sysfs.c perf tools: Fix possible (unlikely) buffer overflow 2013-01-24 16:40:18 -03:00
sysfs.h perf tools: Add sysfs mountpoint interface 2012-02-13 23:27:15 -02:00
target.c perf tools: Replace mempcpy with memcpy 2012-09-05 19:35:21 -03:00
target.h perf target: Add uses_mmap field 2012-05-17 12:32:54 -03:00
thread.c perf tools: Add pid to struct thread 2013-08-27 11:05:53 -03:00
thread.h perf tools: Add pid to struct thread 2013-08-27 11:05:53 -03:00
thread_map.c perf tools: fix thread_map__new_by_pid_str() memory leak in error path 2012-05-25 11:40:59 -03:00
thread_map.h perf evlist: Add thread_map__nr() helper 2013-03-15 13:06:02 -03:00
tool.h perf tools: Add attr->mmap2 support 2013-09-11 10:09:32 -03:00
top.c perf top: Fix percent output when no samples collected 2013-05-28 16:23:59 +03:00
top.h perf top: move CONSOLE_CLEAR to header file 2013-08-07 17:35:26 -03:00
trace-event-info.c perf util: No need to call read_trace_init() in tracing_data_header() 2013-07-12 13:52:31 -03:00
trace-event-parse.c perf tools: Fix old GCC build error in trace-event-parse.c:parse_proc_kallsyms() 2013-09-19 15:08:47 -03:00
trace-event-read.c perf util: Parse header_page to get proper long size 2013-07-12 13:52:24 -03:00
trace-event-scripting.c perf script: Fix named threads support 2013-07-22 11:55:53 -03:00
trace-event.h perf script: Fix named threads support 2013-07-22 11:55:53 -03:00
types.h perf tools: Add union u64_swap type for swapping u64 data 2012-05-22 12:50:25 -03:00
unwind.c perf tools: Remove filter parameter of thread__find_addr_map() 2013-08-12 10:31:12 -03:00
unwind.h perf tools: Convert to LIBUNWIND_SUPPORT 2012-10-02 17:56:01 -03:00
usage.c perf target: Introduce perf_target__parse_uid() 2012-05-07 16:46:48 -03:00
util.c perf util: Add parse_nsec_time() function 2013-08-07 17:35:26 -03:00
util.h perf util: Add parse_nsec_time() function 2013-08-07 17:35:26 -03:00
values.c perf tools: Fix a memory leak on perf_read_values_destroy 2011-12-20 13:41:34 -02:00
values.h perf tools: Protect header files with a consistent style 2009-09-24 21:27:51 +02:00
vdso.c perf symbols: Fix vdso list searching 2013-07-08 17:59:07 -03:00
vdso.h perf tools: Back [vdso] DSO with real data 2012-09-11 12:08:30 -03:00
wrapper.c perf tools: Use __maybe_used for unused variables 2012-09-11 12:19:15 -03:00
xyarray.c perf tools: Introduce event selectors 2011-01-03 16:39:04 -02:00
xyarray.h perf tools: Introduce event selectors 2011-01-03 16:39:04 -02:00