Actions
Bug #1032
closedBug #1031: Cannot create buffers of size 4 GiB or greater
LTTng session daemon may hang under certain circumstances
Start date:
06/16/2016
Due date:
% Done:
0%
Estimated time:
Description
(LTTng 2.8.0 lttng-tools a0f518a on Ubuntu 14.04.4, kernel 4.2.0-27-generic)
I ran into this while working with bug 1031.
After trying to load a session with apparently too greedy buffer memory demands (11 channels of twice 64 MiB each), I stopped the session daemon completely (sudo service lttng-sessiond stop), stop confirmed by the System Monitor) and restarted it. I then tried to load a less demanding session (which normally succeeds on a freshly booted system) and the LTTng command line client hanged, apparently because the session daemon hanged as well.
~$ lttng load demanding-session Error: Failed to load session demanding-session: Kernel create channel failed Error: Kernel create channel failed $ sudo service lttng-sessiond stop lttng-sessiond stop/waiting $ sudo service lttng-sessiond start lttng-sessiond start/running, process 2693 $ lttng load less-demanding-session ^C $ sudo service lttng-sessiond stop lttng-sessiond stop/waiting
The verbose session daemon log is attached. The last part (in response to the lttng load less-demanding-session) is:
DEBUG1 - 10:20:53.405425 [2842/2938]: Wait for client response (in thread_manage_clients() at main.c:4489) DEBUG1 - 10:20:53.405520 [2842/2938]: Receiving data from client ... (in thread_manage_clients() at main.c:4534) DEBUG1 - 10:20:53.405612 [2842/2938]: Nothing recv() from client... continuing (in thread_manage_clients() at main.c:4538) DEBUG1 - 10:20:53.405630 [2842/2938]: Clean command context structure (in clean_command_ctx() at main.c:831) DEBUG1 - 10:20:53.405638 [2842/2938]: Accepting client command ... (in thread_manage_clients() at main.c:4437) DEBUG1 - 10:20:53.405714 [2842/2938]: Wait for client response (in thread_manage_clients() at main.c:4489) DEBUG1 - 10:20:53.405732 [2842/2938]: Receiving data from client ... (in thread_manage_clients() at main.c:4534) DEBUG1 - 10:20:53.405819 [2842/2938]: Processing client command 8 (in process_client_msg() at main.c:3020) DEBUG1 - 10:20:53.405849 [2842/2938]: Waiting for 1 URIs from client ... (in process_client_msg() at main.c:3795) DEBUG2 - 10:20:53.405911 [2842/2938]: Trying to find session by name less-demanding-session (in session_find_by_name() at session.c:276) DEBUG3 - 10:20:53.405961 [2842/2938]: Created hashtable size 4 at 0x7f5048006f00 of type 1 (in lttng_ht_new() at hashtable.c:145) DEBUG1 - 10:20:53.405973 [2842/2938]: Allocating ltt_sessions_ht_by_id (in ltt_sessions_ht_alloc() at session.c:153) DEBUG3 - 10:20:53.405978 [2842/2938]: Created hashtable size 4 at 0x7f50480072c0 of type 2 (in lttng_ht_new() at hashtable.c:145) DEBUG1 - 10:20:53.405998 [2842/2938]: Tracing session less-demanding-session created with ID 0 by UID 1000 GID 1000 (in session_create() at session.c:420) DEBUG2 - 10:20:53.406003 [2842/2938]: Trying to find session by name less-demanding-session (in session_find_by_name() at session.c:276) DEBUG3 - 10:20:53.406032 [2842/2938]: Created hashtable size 4 at 0x7f504800c6f0 of type 1 (in lttng_ht_new() at hashtable.c:145) DEBUG2 - 10:20:53.406039 [2842/2938]: Setting trace directory path from URI to /media/sf_MD1200bas/Ubuntu 14.04.4/ltraces/less-demanding-session (in add_uri_to_consumer() at cmd.c:761) DEBUG1 - 10:20:53.406049 [2842/2938]: Sending response (size: 20, retcode: Success (10)) (in thread_manage_clients() at main.c:4584) DEBUG1 - 10:20:53.406065 [2842/2938]: Clean command context structure (in clean_command_ctx() at main.c:831) DEBUG1 - 10:20:53.406071 [2842/2938]: Accepting client command ... (in thread_manage_clients() at main.c:4437) DEBUG1 - 10:20:53.406343 [2842/2938]: Wait for client response (in thread_manage_clients() at main.c:4489) DEBUG1 - 10:20:53.406364 [2842/2938]: Receiving data from client ... (in thread_manage_clients() at main.c:4534) DEBUG1 - 10:20:53.406425 [2842/2938]: Nothing recv() from client... continuing (in thread_manage_clients() at main.c:4538) DEBUG1 - 10:20:53.406439 [2842/2938]: Clean command context structure (in clean_command_ctx() at main.c:831) DEBUG1 - 10:20:53.406444 [2842/2938]: Accepting client command ... (in thread_manage_clients() at main.c:4437) DEBUG1 - 10:20:53.406507 [2842/2938]: Wait for client response (in thread_manage_clients() at main.c:4489) DEBUG1 - 10:20:53.406524 [2842/2938]: Receiving data from client ... (in thread_manage_clients() at main.c:4534) DEBUG1 - 10:20:53.406586 [2842/2938]: Processing client command 5 (in process_client_msg() at main.c:3020) DEBUG1 - 10:20:53.406597 [2842/2938]: Getting session less-demanding-session by name (in process_client_msg() at main.c:3108) DEBUG2 - 10:20:53.406601 [2842/2938]: Trying to find session by name less-demanding-session (in session_find_by_name() at session.c:276) DEBUG1 - 10:20:53.406606 [2842/2938]: Creating kernel session (in create_kernel_session() at main.c:2936) DEBUG3 - 10:20:53.406635 [2842/2938]: Created hashtable size 4 at 0x7f5048003d30 of type 1 (in lttng_ht_new() at hashtable.c:145)
Somehow the session daemon hangs shortly after lttng_ht_new() at hashtable.c:145
This is weird because it has to come from either persistent (file-based) LTTng data or from messing up the kernel itself.
Files
Actions