Project

General

Profile

Bug #1283

Problem with lttng load on 2.12

Added by Filip Pudak 7 months ago. Updated about 1 month ago.

Status:
Won't fix
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
09/17/2020
Due date:
% Done:

0%

Estimated time:

Description

Hi, I just upgraded from 2.10 -> 2.12 and have encountered this issue while running my usual script.
Console output :

---------------


+ lttng_msg=-q
+ session_cfg_path=/tmp/lttng_session_cfg
+ lttng -q save --force --output-path=/tmp/lttng_session_cfg session_trace
+ exit_status=0
+ '[' 0 -eq 0 ']'
+ lttng -q stop session_trace
+ lttng -q destroy session_trace
+ sleep 1
+ rm -rf '/var/log/lttng/shm/save/*' /var/log/lttng/shm/current
+ df /pramfs -h
Filesystem Size Used Avail Use% Mounted on
/dev/pmem0 488M 2.7M 456M 1% /pramfs
+ lttng -vvv load --input-path=/tmp/lttng_session_cfg
DEBUG3 - 13:51:54.740354470 [3561/3561]: URI string: file:///var/volatile/tmp/lttng_session_cfg (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:298)
DEBUG3 - 13:51:54.740476330 [3561/3561]: URI file destination: /var/volatile/tmp/lttng_session_cfg (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:335)
DEBUG3 - 13:51:54.740498090 [3561/3561]: URI dtype: 3, proto: 0, host: , subdir: , ctrl: 0, data: 0 (in uri_parse() at lttng-tools/2.12.0-r0/git/src/common/uri.c:512)
DEBUG1 - 13:51:54.745829510 [3561/3561]: LSM cmd type : 49 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143)
DEBUG1 - 13:51:54.746218790 [3561/3561]: LSM cmd type : 25 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143)
DEBUG1 - 13:51:54.746497020 [3561/3561]: LSM cmd type : 40 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143)
*Error: Failed to load session session_trace: Not enough memory*
DEBUG1 - 13:51:54.746995900 [3561/3561]: LSM cmd type : 17 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143)
DEBUG1 - 13:51:54.747147820 [3561/3561]: LSM cmd type : 24 (in send_session_msg() at lttng-tools/2.12.0-r0/git/src/lib/lttng-ctl/lttng-ctl.c:143)
DEBUG1 - 13:51:54.747353630 [3561/3561]: *epoll set max size is 1024860*  (in compat_epoll_set_max_size() at lttng-tools/2.12.0-r0/git/src/common/compat/compat-epoll.c:346)
*Error: Not enough memory*
DEBUG1 - 13:51:54.748448160 [3561/3561]: Clean exit (in clean_exit() at lttng-tools/2.12.0-r0/git/src/bin/lttng/lttng.c:152)


--------------
Has anyone encountered a similar issue? I know there is lttng clear in 2.12, but would still like to understand why the OOM crash is there...

/Filip

#1

Updated by Jonathan Rajotte Julien 7 months ago

  • Description updated (diff)

Hi,

Please share the session configuration? (xml file)

Please provide the output from the following command?

df -h
free -h

Also please provide the verbose output from lttng-sessiond:

lttng-sessiond -vvv --verbose-consumer

I do not think lttng clear have anything to do with the current problem.

Is the problem always reproducible? Are you able to create a similar configuration manually after the load fails?

Thanks

#2

Updated by Jonathan Rajotte Julien 7 months ago

  • Status changed from New to Feedback
#3

Updated by Filip Pudak 6 months ago

Hi Jonathan,

I worked around this by using lttng_clear in the end.
The issue was reproducible.
We can close this case.

Cheers,
Filip

#4

Updated by Jonathan Rajotte Julien 6 months ago

Hi Filip,

I understand you found a work around here. Keep in mind that here the clear option probably work only because we do not allocate new buffers for the session, we simply clear them.

Still it seems that there is a problem/quirk regarding the lifetime of the allocated memory for the buffers. Which is probably why on the "load" you are seeing allocation failure.

It would really help us to get the output from the following command on error:

lttng-sessiond -vvv --verbose-consumer

Cheers

#5

Updated by Jonathan Rajotte Julien about 1 month ago

  • Status changed from Feedback to Won't fix

Also available in: Atom PDF