Project

General

Profile

Bug #626

assertion in libust cause application to quit

Added by Yannick Brosseau over 6 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Target version:
Start date:
09/05/2013
Due date:
% Done:

0%

Estimated time:

Description

I'm instrumenting RSyslog and I sometime get the following output and assert:

libust[4357/4358]: Error: sendmsg: Socket operation on non-socket (in ustcomm_send_unix_sock() at lttng-ust-comm.c:332)
libust[4357/4358]: Error: Socket shutdown error (in ustcomm_send_unix_sock() at lttng-ust-comm.c:339)
libust[4357/4358]: Error: Error handling message for global socket (in ust_listener_thread() at lttng-ust-comm.c:1087)
libust[4357/4358]: Error: recvmsg: Socket operation on non-socket (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:285)
libust[4357/4358]: Error: Socket shutdown error (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:292)
libringbuffer[4357/4358]: Error: close: Bad file descriptor (in shmp_object_destroy() at shm.c:414)
rsyslogd: shm.c:415: shmp_object_destroy: Assertion `0' failed.


This happens when I issue a lttng stop command.
It also shows sometime when I try to start the application.

I run the app with the following command:
LD_PRELOAD=liblttng-ust-fork.so /usr/local/sbin/rsyslogd

I did not see the problem when I run in rsyslog non fork mode.

Currently running both 2.3 for lttng-ust and lttng-tools. Same problem was happening on 2.2


Related issues

Is duplicate of LTTng-UST - Bug #253: lttng-ust file descriptors can be closed by "misbehaving" daemon appsResolved06/05/2012

Actions
#1

Updated by Mathieu Desnoyers over 6 years ago

Does rsyslog close all file descriptors (e.g. all FD from 0 to 99)
without caring if they are opened or not upon fork ?

See http://bugs.lttng.org/issues/253

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

#2

Updated by Yannick Brosseau over 6 years ago

Yes it close all the FDs

I tried to do a manual "reinit" function, but it does not seem to help. I might be missing something.

(It still get event in my trace)

#3

Updated by Mathieu Desnoyers over 6 years ago

  • Status changed from New to Feedback

It seems that a LD_PRELOADed .so that overrides close() to make sure the application does not close ust's file descriptors seems to fix the issue, is it right ? Any progress on this front ?

Thanks,

Mathieu

#4

Updated by Yannick Brosseau over 6 years ago

Yes, it fixes the problem. But it would also be nice to not it the assert.

For the close preload, it's coming well. Should post a cleaner version to the mailing list today or tomorrow.

#5

Updated by Christian Babeux about 6 years ago

  • Assignee set to Mathieu Desnoyers

A patch has been posted on the mailing list at : http://lists.lttng.org/pipermail/lttng-dev/2013-October/021533.html
Mathieu will review it and decide the appropriate time to merge it.

#6

Updated by David Goulet almost 6 years ago

  • Status changed from Feedback to Confirmed
#7

Updated by Mathieu Desnoyers over 5 years ago

  • Target version set to 2.7
#8

Updated by Mathieu Desnoyers almost 5 years ago

I've started integrating the patch, and did most of the cleanup needed to get it in shape, but it ends up that this approach is racy.

We have no guarantee that the application won't close the FD between our "open()" and the moment we set it in one of the FDs part of ust_is_my_fd(). So the application can close our FD during that time. We would need some synchronization there. Moreover, there are more FDs that just the global/local apps notify_socket and socket: we also have FDs for ring buffer mappings for instance (per session). We would therefore need to keep track of a dynamic list of FDs within a hash table.

#9

Updated by Mathieu Desnoyers almost 5 years ago

another approach I discussed with a glibc maintainer a while ago would be to override clone() used by pthread_create() from liblttng-ust, and make it allocate our own set of FDs.

#10

Updated by Mathieu Desnoyers over 4 years ago

  • Target version changed from 2.7 to 2.8
#11

Updated by Jérémie Galarneau about 4 years ago

  • Target version changed from 2.8 to 2.9
#12

Updated by Mathieu Desnoyers almost 4 years ago

I've pushed the following commit in lttng-ust. It allows applications to use getenv("LTTNG_UST_LOADED") to check if lttng-ust is loaded, and skip the "close all fd" work-around in that case.

commit 2315088b93f925c5cccee865ccb8f6bc542e52dc
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Jul 12 15:11:06 2016 -0400

    Introduce LTTNG_UST_LOADED environment variable

    Set internally by liblttng-ust's constructor. Can be used by
    applications to detect if lttng-ust is loaded, even if liblttng-ust is
    not directly linked by the application.

    The main use-case is to allow applications to detect that they should
    not try to close file descriptors that do not belong to them (e.g.
    BSD closefrom). This is a common pattern with applications invoking
    daemon(3).

    Note that this environment variable is passed to children of a traced
    process, and through exec calls. Therefore, an application might think
    that lttng-ust is loaded even though it's not loaded in its own address
    space if it was loaded by one of its parent processes.

    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
#13

Updated by Mathieu Desnoyers almost 4 years ago

Previous commit reverted, replaced by:

commit 07b57e5eb9db5d4219cab7cb3ad61edbc32eb9e5
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Jul 12 16:28:45 2016 -0400

    Introduce lttng_ust_loaded weak symbol

    This variable can be tested by applications to check whether lttng-ust
    is loaded. They simply have to define their own "lttng_ust_loaded" weak
    symbol, and test it. It is set to 1 by the library constructor.

    The main use-case is to allow applications to detect that they should
    not try to close file descriptors that do not belong to them (e.g.  BSD
    closefrom). This is a common pattern with applications invoking
    daemon(3).

    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
#14

Updated by Mathieu Desnoyers over 3 years ago

  • Status changed from Confirmed to Resolved

Also available in: Atom PDF