Bug #626
closed
assertion in libust cause application to quit
Added by Yannick Brosseau about 11 years ago.
Updated about 8 years ago.
Description
I'm instrumenting RSyslog and I sometime get the following output and assert:
libust[4357/4358]: Error: sendmsg: Socket operation on non-socket (in ustcomm_send_unix_sock() at lttng-ust-comm.c:332)
libust[4357/4358]: Error: Socket shutdown error (in ustcomm_send_unix_sock() at lttng-ust-comm.c:339)
libust[4357/4358]: Error: Error handling message for global socket (in ust_listener_thread() at lttng-ust-comm.c:1087)
libust[4357/4358]: Error: recvmsg: Socket operation on non-socket (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:285)
libust[4357/4358]: Error: Socket shutdown error (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:292)
libringbuffer[4357/4358]: Error: close: Bad file descriptor (in shmp_object_destroy() at shm.c:414)
rsyslogd: shm.c:415: shmp_object_destroy: Assertion `0' failed.
This happens when I issue a lttng stop command.
It also shows sometime when I try to start the application.
I run the app with the following command:
LD_PRELOAD=liblttng-ust-fork.so /usr/local/sbin/rsyslogd
I did not see the problem when I run in rsyslog non fork mode.
Currently running both 2.3 for lttng-ust and lttng-tools. Same problem was happening on 2.2
Yes it close all the FDs
I tried to do a manual "reinit" function, but it does not seem to help. I might be missing something.
(It still get event in my trace)
- Status changed from New to Feedback
It seems that a LD_PRELOADed .so that overrides close() to make sure the application does not close ust's file descriptors seems to fix the issue, is it right ? Any progress on this front ?
Thanks,
Mathieu
Yes, it fixes the problem. But it would also be nice to not it the assert.
For the close preload, it's coming well. Should post a cleaner version to the mailing list today or tomorrow.
- Assignee set to Mathieu Desnoyers
- Status changed from Feedback to Confirmed
- Target version set to 2.7
I've started integrating the patch, and did most of the cleanup needed to get it in shape, but it ends up that this approach is racy.
We have no guarantee that the application won't close the FD between our "open()" and the moment we set it in one of the FDs part of ust_is_my_fd(). So the application can close our FD during that time. We would need some synchronization there. Moreover, there are more FDs that just the global/local apps notify_socket and socket: we also have FDs for ring buffer mappings for instance (per session). We would therefore need to keep track of a dynamic list of FDs within a hash table.
another approach I discussed with a glibc maintainer a while ago would be to override clone() used by pthread_create() from liblttng-ust, and make it allocate our own set of FDs.
- Target version changed from 2.7 to 2.8
- Target version changed from 2.8 to 2.9
I've pushed the following commit in lttng-ust. It allows applications to use getenv("LTTNG_UST_LOADED") to check if lttng-ust is loaded, and skip the "close all fd" work-around in that case.
commit 2315088b93f925c5cccee865ccb8f6bc542e52dc
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Tue Jul 12 15:11:06 2016 -0400
Introduce LTTNG_UST_LOADED environment variable
Set internally by liblttng-ust's constructor. Can be used by
applications to detect if lttng-ust is loaded, even if liblttng-ust is
not directly linked by the application.
The main use-case is to allow applications to detect that they should
not try to close file descriptors that do not belong to them (e.g.
BSD closefrom). This is a common pattern with applications invoking
daemon(3).
Note that this environment variable is passed to children of a traced
process, and through exec calls. Therefore, an application might think
that lttng-ust is loaded even though it's not loaded in its own address
space if it was loaded by one of its parent processes.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Previous commit reverted, replaced by:
commit 07b57e5eb9db5d4219cab7cb3ad61edbc32eb9e5
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Tue Jul 12 16:28:45 2016 -0400
Introduce lttng_ust_loaded weak symbol
This variable can be tested by applications to check whether lttng-ust
is loaded. They simply have to define their own "lttng_ust_loaded" weak
symbol, and test it. It is set to 1 by the library constructor.
The main use-case is to allow applications to detect that they should
not try to close file descriptors that do not belong to them (e.g. BSD
closefrom). This is a common pattern with applications invoking
daemon(3).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
- Status changed from Confirmed to Resolved
Also available in: Atom
PDF