Bug #747: Assertion failure in thread_ht_cleanup in lttng-sessiond with --disable-epoll - LTTng-tools - LTTng bugs repository

Actions

Copy link

Bug #747

closed

Assertion failure in thread_ht_cleanup in lttng-sessiond with --disable-epoll

Added by Anonymous about 10 years ago. Updated over 9 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Philippe Proulx

Target version:

Start date:

03/03/2014

Due date:

% Done:

100%

Estimated time:

Description

With lttng-ust 2.3.2, lttng-tools 2.3.2 and userspace-rcu 0.8.3 on CentOS 6 built using "gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)", if I ./configure lttng-tools with the flags --disable-static --disable-epoll, then lttng-sessiond -v produces some error messages at startup:

DEBUG1 [31030/31036]: Woken up but nothing in the UST command queue (in thread_dispatch_ust_registration() at main.c:1527)
Error: Unknown poll events 0 for sock 3
DEBUG1 [31030/31036]: Futex n to 1 wait done (in futex_nto1_wait() at futex.c:90)
DEBUG1 [31030/31038]: fd 19 of 3 added to pollfd (in compat_poll_add() at compat-poll.c:188)
DEBUG1 [31030/31038]: Apps with sock 19 added to poll set (in thread_manage_apps() at main.c:1311)
DEBUG1 [31030/31038]: Apps thread polling on 2 fds (in thread_manage_apps() at main.c:1248)
DEBUG1 [31030/31038]: fd 21 of 4 added to pollfd (in compat_poll_add() at compat-poll.c:188)
DEBUG1 [31030/31038]: Apps with sock 21 added to poll set (in thread_manage_apps() at main.c:1311)
DEBUG1 [31030/31038]: Apps thread polling on 3 fds (in thread_manage_apps() at main.c:1248)
DEBUG1 [31030/31036]: Futex n to 1 prepare done (in futex_nto1_prepare() at futex.c:75)
DEBUG1 [31030/31036]: Woken up but nothing in the UST command queue (in thread_dispatch_ust_registration() at main.c:1527)
DEBUG1 [31030/31039]: fd 20 of 3 added to pollfd (in compat_poll_add() at compat-poll.c:188)
DEBUG3 [31030/31039]: UST thread notify added sock 20 to pollset (in ust_thread_manage_notify() at ust-thread.c:131)
DEBUG3 [31030/31039]: [ust-thread] Manage notify polling on 2 fds (in ust_thread_manage_notify() at ust-thread.c:61)
Error: Unknown poll events 0 for sock 3
DEBUG1 [31030/31039]: fd 22 of 4 added to pollfd (in compat_poll_add() at compat-poll.c:188)
DEBUG3 [31030/31039]: UST thread notify added sock 22 to pollset (in ust_thread_manage_notify() at ust-thread.c:131)
Error: Unknown poll events 0 for sock 20
DEBUG3 [31030/31039]: [ust-thread] Manage notify polling on 3 fds (in ust_thread_manage_notify() at ust-thread.c:61)

and then, when issuing a command such as lttng enable-event --userspace --all or lttng add-context --userspace -t vpid -t vtid -t procname, an assertion failure is seen in the session daemon's output:

DEBUG1 [31030/31035]: Creating UST session (in create_ust_session() at main.c:2454)
DEBUG3 [31030/31035]: Created hashtable size 4 at 0x7f01cc003d80 of type 0 (in lttng_ht_new() at hashtable.c:112)
DEBUG3 [31030/31035]: Created hashtable size 4 at 0x7f01cc0041c0 of type 1 (in lttng_ht_new() at hashtable.c:112)
DEBUG3 [31030/31035]: Copying tracing session consumer output in UST session (in copy_session_consumer() at main.c:2404)
DEBUG3 [31030/31035]: Created hashtable size 4 at 0x7f01cc004600 of type 1 (in lttng_ht_new() at hashtable.c:112)
DEBUG3 [31030/31035]: Copy session consumer subdir /ust (in copy_session_consumer() at main.c:2422)
DEBUG1 [31030/31035]: Spawning consumerd (in spawn_consumerd() at main.c:2085)
lttng-sessiond: ht-cleanup.c:92: thread_ht_cleanup: Assertion `pollfd == ht_cleanup_pipe[0]' failed.
DEBUG1 [3397/3397]: Using 64-bit UST consumer at: /opt/lttng-tools-2.3.2-no-epoll/lib/lttng/libexec/lttng-consumerd (in spawn_consumerd() at main.c:2161)
Aborted (core dumped)

If I remove --disable-epoll from the configure command line, then no such errors or assertion failures are seen.

I had used --disable-epoll previously for CentOS 5 compatibility, as CentOS 5 does not have epoll_pwait().

Actions

Copy link

Updated by Anonymous about 10 years ago

From lttng-sessiond core dump:

(gdb) bt full
#0  0x0000003bcbc32925 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000003bcbc34105 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000003bcbc2ba4e in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x0000003bcbc2bb10 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x0000000000414ac5 in thread_ht_cleanup (data=<optimized out>)
    at ht-cleanup.c:92
        ht = 0x3bcc007962 <start_thread+98>
        ret = 0
        i = <optimized out>
        pollfd = <optimized out>
        err = -1
        revents = <optimized out>
        nb_fd = <optimized out>
        events = {wait = {nb_fd = 2, alloc_size = 2, init_size = 2,
            events = 0x7f01dc0008c0}, current = {nb_fd = 2, alloc_size = 2,
            init_size = 2, events = 0x7f01dc0008e0}, need_realloc = 0,
          need_update = 0}
        __func__ = "thread_ht_cleanup" 
        __PRETTY_FUNCTION__ = "thread_ht_cleanup" 
#5  0x0000003bcc0079d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6  0x0000003bcbce8b6d in clone () from /lib64/libc.so.6
No symbol table info available.

Actions

Copy link

Updated by Anonymous about 10 years ago

When this occurs, the lttng command's output indicates that there is a session daemon problem, e.g.:

UST context vpid added to all channels
Error: vtid: No session daemon is available
Error: procname: No session daemon is available
Warning: Some command(s) went wrong

Actions

Copy link

Updated by Mathieu Desnoyers about 10 years ago

It appears that there is an issue in thread_ht_cleanup: it expects the for() loop to only iterate on file descriptors returned by epoll() (hence the assert to make sure of this), but in the poll() case, it iterates on all file descriptors. We should add a check to see if the revent is 0 at the beginning of the loop, and continue to the next loop iteration if it is the case.

Actions

Copy link