Project

General

Profile

Bug #341

Daemon applications do not work with lttng

Added by Hans Nordebäck about 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
High
Target version:
Start date:
09/18/2012
Due date:
% Done:

0%

Estimated time:

Description

lttng threads are created before main via attribute(constructor). After daemonizing an application
the lttng threads "disappears" after "daemonizing" the application.
The daemon application is a common pattern and frequently used so it is important to get this working.
Solutions are discussed in the dev mail list referenced below,
and an additional suggestion to solve this issue is to use the pthread_atfork() function to do the controlled "handover" to
the child process.

reference, http://lists.lttng.org/pipermail/lttng-dev/2012-September/018652.html


Related issues

Related to LTTng-tools - Bug #340: recvmsg in ustcomm_recv_unix_sock to hangsInvalid09/18/2012

Actions
#1

Updated by Mathieu Desnoyers almost 8 years ago

Just to understand a little more about the use-case, I have 2 specific questions:

- Is the library call to daemon(3) used to daemonize the application ?

- How exactly are the file descriptor closed ? In a previous email, you mentioned that all FD are closed up to SC_OPEN_MAX. I just want to confirm that this is the case, and know when this is performed (before/after daemon(3) call, in parent or child ?).

I'm trying to come up with an elegant solution to this. I think daemon(3) should be overridden like we do for fork() and clone() (we currently have a bug open in UST about this), but closing arbitrary file descriptor is another issue we also have to deal with.

Thanks!

Mathieu

#2

Updated by Mathieu Desnoyers almost 8 years ago

  • Status changed from New to Confirmed
  • Assignee set to Mathieu Desnoyers
  • Target version set to 2.1 stable
#3

Updated by Hans Nordebäck almost 8 years ago

the daemon(3) is not used. No fd's are closed, only fork() is called and the parent do exit().
The child continues to run and no exec() are called. The close of fd's up to SC_OPEN_MAX is used later
to fork/exec application programs, I guess that should not be a problem, as the exec starts a new program
and the "constructors" are run. Perhaps pthread_atfork() can be used to do the "handover" to the child,
i.e. close mutexes, etc.. /Thanks Hans

#4

Updated by Mathieu Desnoyers almost 8 years ago

Ah, then I think all you need is to LD_PRELOAD liblttng-ust-fork.so for your application. This wrapper around fork, clone, and BSD rfork perform something similar to pthread_atfork, but the advantage of liblttng-ust-fork over pthread_atfork is that if a traced application uses pthread_atfork, this API does not provide any guarantee about the order in the callback chain. Overriding the symbol lets us ensure that we'll be the last "callback" called.

Let me know how it works out for you!

Thanks,

Mathieu

#5

Updated by Mathieu Desnoyers almost 8 years ago

FYI, I added documentation to the manpage:

USING LTTNG UST WITH DAEMONS
Some extra care is needed when using liblttng-ust with daemon
applications that call fork(), clone(), or BSD rfork() without a
following exec() family system call. The library "liblttng-ust-fork.so"
needs to be preloaded for the application (launch with e.g.
LD_PRELOAD=liblttng-ust-fork.so appname).

#6

Updated by Hans Nordebäck almost 8 years ago

I have some problem to get this to work, I start the daemon application with the LD_PRELOAD set, then
the lttng-sessiond prints a message and Linux reboots. I run Ubuntu 12.04 with the latest updates.
The Ubuntu is run in VirtualBox. The message from lttng-sessiond is something like
ust-app.c:1405 Assertion `!ret´ failed., and then Linux reboots.

#7

Updated by Mathieu Desnoyers almost 8 years ago

(Adding Christian and David as watchers to this bug, because it seems to be related to an interaction between liblttng-ust-fork and lttng-sessiond)

I suspect that something wrong is happening in your VirtualBox, and that it's not LTTng-related. But let's try to narrow down the cause anyway. The fact that the Linux in your VM reboots is just not normal (from a Linux standpoint). Do you have any lttng-modules installed, or just lttng-ust and lttng-tools ?

The assert is in ust_app_unregister(), and expects that a removal from an application from the application hash table should never fail. So it looks like we get a double unregister from the same application.

Please start

lttng-sessiond -vvv

after making sure all lttng-sessiond were killed.

Please provide the complete output of lttng-sessiond when you trigger this issue,

Also, please try to run the fork demo program provided with the lttng-ust source tree, under tests/fork. Run by doing "./run". Try to repeat in the same conditions as your application (although the fork test program is already linked on liblttng-ust-fork).

For completeness of bug description, I get from the email thread that you "have installed lttng-ust 2.0.5 and lttng-tools 2.0.4".

Thanks,

Mathieu

#8

Updated by Hans Nordebäck almost 8 years ago

I rebuilt and re-installed lttng-ust 2.0.5 and lttng-tools 2.0.4, and now it works. I have not installed any
lttng-modules explicitly and lsmod shows that several lttng modules are installed. /Thanks Hans

#9

Updated by Mathieu Desnoyers almost 8 years ago

  • Status changed from Confirmed to Resolved

OK, great to know that it works now!

about lttng-modules automatically installed, I suppose you installed a ubuntu lttng meta-package that installs it too. So you can use kernel tracing if you want (with "lttng enable-event -a -k" for instance) to trace the Linux kernel and your application on a common time-line, but you can also just use UST, it's up to you.

I will therefore close this bug. Don't hesitate to reopen if something pops up, or if you have a different issue, to open a new bug. It will be our pleasure to assist you.

Thank you!

Mathieu

Also available in: Atom PDF