Bug #1283
closedProblem with lttng load on 2.12
0%
Description
Hi, I just upgraded from 2.10 -> 2.12 and have encountered this issue while running my usual script.
Console output :
---------------
+ lttng_msg=-q + session_cfg_path=/tmp/lttng_session_cfg + lttng -q save --force --output-path=/tmp/lttng_session_cfg session_trace + exit_status=0 + '[' 0 -eq 0 ']' + lttng -q stop session_trace + lttng -q destroy session_trace + sleep 1 + rm -rf '/var/log/lttng/shm/save/*' /var/log/lttng/shm/current + df /pramfs -h Filesystem Size Used Avail Use% Mounted on /dev/pmem0 488M 2.7M 456M 1% /pramfs + lttng -vvv load --input-path=/tmp/lttng_session_cfg DEBUG3 - 13:51:54.740354470 [3561/3561]: URI string: file:///var/volatile/tmp/lttng_session_cfg (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:298) DEBUG3 - 13:51:54.740476330 [3561/3561]: URI file destination: /var/volatile/tmp/lttng_session_cfg (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:335) DEBUG3 - 13:51:54.740498090 [3561/3561]: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:512) DEBUG1 - 13:51:54.745829510 [3561/3561]: LSM cmd type : 49 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143) DEBUG1 - 13:51:54.746218790 [3561/3561]: LSM cmd type : 25 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143) DEBUG1 - 13:51:54.746497020 [3561/3561]: LSM cmd type : 40 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143) *Error: Failed to load session session_trace: Not enough memory* DEBUG1 - 13:51:54.746995900 [3561/3561]: LSM cmd type : 17 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143) DEBUG1 - 13:51:54.747147820 [3561/3561]: LSM cmd type : 24 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143) DEBUG1 - 13:51:54.747353630 [3561/3561]: *epoll set max size is 1024860* (in compat_epoll_set_max_size() at lttng-tools/2.12.0-r0/git/src/common/compat/compat-epoll.c:346) *Error: Not enough memory* DEBUG1 - 13:51:54.748448160 [3561/3561]: Clean exit (in clean_exit() at lttng-tools/2.12.0-r0/git/src/bin/lttng/lttng.c:152)
--------------
Has anyone encountered a similar issue? I know there is lttng clear in 2.12, but would still like to understand why the OOM crash is there...
/Filip
Updated by Jonathan Rajotte Julien about 4 years ago
- Description updated (diff)
Hi,
Please share the session configuration? (xml file)
Please provide the output from the following command?
df -h free -h
Also please provide the verbose output from lttng-sessiond:
lttng-sessiond -vvv --verbose-consumer
I do not think lttng clear have anything to do with the current problem.
Is the problem always reproducible? Are you able to create a similar configuration manually after the load fails?
Thanks
Updated by Jonathan Rajotte Julien about 4 years ago
- Status changed from New to Feedback
Updated by Filip Pudak about 4 years ago
Hi Jonathan,
I worked around this by using lttng_clear in the end.
The issue was reproducible.
We can close this case.
Cheers,
Filip
Updated by Jonathan Rajotte Julien about 4 years ago
Hi Filip,
I understand you found a work around here. Keep in mind that here the clear option probably work only because we do not allocate new buffers for the session, we simply clear them.
Still it seems that there is a problem/quirk regarding the lifetime of the allocated memory for the buffers. Which is probably why on the "load" you are seeing allocation failure.
It would really help us to get the output from the following command on error:
lttng-sessiond -vvv --verbose-consumer
Cheers
Updated by Jonathan Rajotte Julien over 3 years ago
- Status changed from Feedback to Won't fix