LTTng » LTTng-UST

Just to understand a little more about the use-case, I have 2 specific questions:

- Is the library call to daemon(3) used to daemonize the application ?

- How exactly are the file descriptor closed ? In a previous email, you mentioned that all FD are closed up to SC_OPEN_MAX. I just want to confirm that this is the case, and know when this is performed (before/after daemon(3) call, in parent or child ?).

I'm trying to come up with an elegant solution to this. I think daemon(3) should be overridden like we do for fork() and clone() (we currently have a bug open in UST about this), but closing arbitrary file descriptor is another issue we also have to deal with.

Thanks!

Mathieu

the daemon(3) is not used. No fd's are closed, only fork() is called and the parent do exit().
The child continues to run and no exec() are called. The close of fd's up to SC_OPEN_MAX is used later
to fork/exec application programs, I guess that should not be a problem, as the exec starts a new program
and the "constructors" are run. Perhaps pthread_atfork() can be used to do the "handover" to the child,
i.e. close mutexes, etc.. /Thanks Hans

Ah, then I think all you need is to LD_PRELOAD liblttng-ust-fork.so for your application. This wrapper around fork, clone, and BSD rfork perform something similar to pthread_atfork, but the advantage of liblttng-ust-fork over pthread_atfork is that if a traced application uses pthread_atfork, this API does not provide any guarantee about the order in the callback chain. Overriding the symbol lets us ensure that we'll be the last "callback" called.

Let me know how it works out for you!

Thanks,

Mathieu

FYI, I added documentation to the manpage:

USING LTTNG UST WITH DAEMONS
Some extra care is needed when using liblttng-ust with daemon
applications that call fork(), clone(), or BSD rfork() without a
following exec() family system call. The library "liblttng-ust-fork.so"
needs to be preloaded for the application (launch with e.g.
LD_PRELOAD=liblttng-ust-fork.so appname).

I have some problem to get this to work, I start the daemon application with the LD_PRELOAD set, then
the lttng-sessiond prints a message and Linux reboots. I run Ubuntu 12.04 with the latest updates.
The Ubuntu is run in VirtualBox. The message from lttng-sessiond is something like
ust-app.c:1405 Assertion `!ret´ failed., and then Linux reboots.

(Adding Christian and David as watchers to this bug, because it seems to be related to an interaction between liblttng-ust-fork and lttng-sessiond)

I suspect that something wrong is happening in your VirtualBox, and that it's not LTTng-related. But let's try to narrow down the cause anyway. The fact that the Linux in your VM reboots is just not normal (from a Linux standpoint). Do you have any lttng-modules installed, or just lttng-ust and lttng-tools ?

The assert is in ust_app_unregister(), and expects that a removal from an application from the application hash table should never fail. So it looks like we get a double unregister from the same application.

Please start

lttng-sessiond -vvv

after making sure all lttng-sessiond were killed.

Please provide the complete output of lttng-sessiond when you trigger this issue,

Also, please try to run the fork demo program provided with the lttng-ust source tree, under tests/fork. Run by doing "./run". Try to repeat in the same conditions as your application (although the fork test program is already linked on liblttng-ust-fork).

For completeness of bug description, I get from the email thread that you "have installed lttng-ust 2.0.5 and lttng-tools 2.0.4".

Thanks,

Mathieu

I rebuilt and re-installed lttng-ust 2.0.5 and lttng-tools 2.0.4, and now it works. I have not installed any
lttng-modules explicitly and lsmod shows that several lttng modules are installed. /Thanks Hans