Bug #626
closedassertion in libust cause application to quit
0%
Description
I'm instrumenting RSyslog and I sometime get the following output and assert:
libust[4357/4358]: Error: sendmsg: Socket operation on non-socket (in ustcomm_send_unix_sock() at lttng-ust-comm.c:332) libust[4357/4358]: Error: Socket shutdown error (in ustcomm_send_unix_sock() at lttng-ust-comm.c:339) libust[4357/4358]: Error: Error handling message for global socket (in ust_listener_thread() at lttng-ust-comm.c:1087) libust[4357/4358]: Error: recvmsg: Socket operation on non-socket (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:285) libust[4357/4358]: Error: Socket shutdown error (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:292) libringbuffer[4357/4358]: Error: close: Bad file descriptor (in shmp_object_destroy() at shm.c:414) rsyslogd: shm.c:415: shmp_object_destroy: Assertion `0' failed.
This happens when I issue a lttng stop command.
It also shows sometime when I try to start the application.
I run the app with the following command:
LD_PRELOAD=liblttng-ust-fork.so /usr/local/sbin/rsyslogd
I did not see the problem when I run in rsyslog non fork mode.
Currently running both 2.3 for lttng-ust and lttng-tools. Same problem was happening on 2.2
Updated by Mathieu Desnoyers over 11 years ago
Does rsyslog close all file descriptors (e.g. all FD from 0 to 99)
without caring if they are opened or not upon fork ?
See http://bugs.lttng.org/issues/253
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Updated by Yannick Brosseau over 11 years ago
Yes it close all the FDs
I tried to do a manual "reinit" function, but it does not seem to help. I might be missing something.
(It still get event in my trace)
Updated by Mathieu Desnoyers about 11 years ago
- Status changed from New to Feedback
It seems that a LD_PRELOADed .so that overrides close() to make sure the application does not close ust's file descriptors seems to fix the issue, is it right ? Any progress on this front ?
Thanks,
Mathieu
Updated by Yannick Brosseau about 11 years ago
Yes, it fixes the problem. But it would also be nice to not it the assert.
For the close preload, it's coming well. Should post a cleaner version to the mailing list today or tomorrow.
Updated by Christian Babeux over 10 years ago
- Assignee set to Mathieu Desnoyers
A patch has been posted on the mailing list at : http://lists.lttng.org/pipermail/lttng-dev/2013-October/021533.html
Mathieu will review it and decide the appropriate time to merge it.
Updated by David Goulet over 10 years ago
- Status changed from Feedback to Confirmed
Updated by Mathieu Desnoyers over 9 years ago
I've started integrating the patch, and did most of the cleanup needed to get it in shape, but it ends up that this approach is racy.
We have no guarantee that the application won't close the FD between our "open()" and the moment we set it in one of the FDs part of ust_is_my_fd(). So the application can close our FD during that time. We would need some synchronization there. Moreover, there are more FDs that just the global/local apps notify_socket and socket: we also have FDs for ring buffer mappings for instance (per session). We would therefore need to keep track of a dynamic list of FDs within a hash table.
Updated by Mathieu Desnoyers over 9 years ago
another approach I discussed with a glibc maintainer a while ago would be to override clone() used by pthread_create() from liblttng-ust, and make it allocate our own set of FDs.
Updated by Mathieu Desnoyers over 9 years ago
- Target version changed from 2.7 to 2.8
Updated by Jérémie Galarneau over 8 years ago
- Target version changed from 2.8 to 2.9
Updated by Mathieu Desnoyers over 8 years ago
I've pushed the following commit in lttng-ust. It allows applications to use getenv("LTTNG_UST_LOADED") to check if lttng-ust is loaded, and skip the "close all fd" work-around in that case.
commit 2315088b93f925c5cccee865ccb8f6bc542e52dc Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Date: Tue Jul 12 15:11:06 2016 -0400 Introduce LTTNG_UST_LOADED environment variable Set internally by liblttng-ust's constructor. Can be used by applications to detect if lttng-ust is loaded, even if liblttng-ust is not directly linked by the application. The main use-case is to allow applications to detect that they should not try to close file descriptors that do not belong to them (e.g. BSD closefrom). This is a common pattern with applications invoking daemon(3). Note that this environment variable is passed to children of a traced process, and through exec calls. Therefore, an application might think that lttng-ust is loaded even though it's not loaded in its own address space if it was loaded by one of its parent processes. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Updated by Mathieu Desnoyers over 8 years ago
Previous commit reverted, replaced by:
commit 07b57e5eb9db5d4219cab7cb3ad61edbc32eb9e5 Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Date: Tue Jul 12 16:28:45 2016 -0400 Introduce lttng_ust_loaded weak symbol This variable can be tested by applications to check whether lttng-ust is loaded. They simply have to define their own "lttng_ust_loaded" weak symbol, and test it. It is set to 1 by the library constructor. The main use-case is to allow applications to detect that they should not try to close file descriptors that do not belong to them (e.g. BSD closefrom). This is a common pattern with applications invoking daemon(3). Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Updated by Mathieu Desnoyers about 8 years ago
- Status changed from Confirmed to Resolved
Applied in changeset lttng-ust|6548fca464f68be5ded73be6b9a479daf08e62d6.