LTTng » LTTng-tools

Adding a timeout to recvmsg() and sendmsg() to the application socket can possibly have some "unknown" repercussions over time...

There is now way of detecting a SIGSTOP on the app socket so I wonder if we should either timeout or make the tracer sends a notification of a sigstop for which I'm not sure is that simple....?

Thoughts?

So here is what I'm going to do.

For each application socket, I'll add a default timeout, that can be specified by a ENV var to the session daemon, and if the recvmsg() do timeout, the command will be aborted but the application will still be registered.

For now, it's the only possible solution we ended up with for 2.1-stable.

Just to clarify:
1)_ does it mean that after the timeout, the "hanging" app will still be registered but it will not participate in that session and therefore there should be no <pid> directory under this session directory for that hanging app, right ?

2)_ Also, let's say the above hanging app comes out from its hanging state, if a new session is created, will it be able to participate then ?

Hmmm, somehow not all my comments got submitted .

2)_ Also, let's say the above hanging app comes out of its hanging state, if another session got created, will that app be able to participate at this point ?

Just to clarify:
1)_ does it mean that after the timeout, the "hanging" app will still be registered but it will not participate in that session and therefore there should be no <pid> directory under this session directory for that hanging app, right ?

I just spoke with Mathieu about this issue and there is actually more
problems.

When sending a command to a stopped application, the sendmsg() succeeds
but the recvmsg() times out so the command is aborted but the
application stays registered (on the session daemon side). However, once
the app. comes back to life, the command previously sent is handled and
a reply is sent back to the session daemon which is not waiting for
this (and for now is creating an infinite loop in the epoll thread).

And to be honest, we don't have a way to fix that without changing a
large part of the application and command subsystems.

So the conclusion was to unregister the application if detected so the
session daemon can at least continue to operate.

Now the question is can we do that on the trace side or do we have to
dot it on the session daemon.

We will sleep the night on this one and figure out the best way to
handle the issue here.

Thanks!
David

Hi David,

It looks like we have a couple of things to consider now:

1) How can we bring to the user attention that the app has been
inresponsive during event enabling ?
I am guessing that if it fail to reply during the event enabling, it will
not participate in the sesson and therefore it has no PID directory under
the session directory. But is there another better way ?

2) How to perform the "unregistration" of the app .
(Can we assume that "unregister" means the app is no longer traceable ?)

3) How to register the app again (without restarting the app of course).

Regards,
Tan

So here are the two scenarios we can envision. Either keep the app registered or unregister it. Here is what we'll have to do in terms of dev.

• We have to make significant changes to the command and application sub systems to handle this possible asynchronous command.
• Detect the stopped state by a socket timeout so other subsequent commands will be queued (not trivial).
• Flag the app "stopped" in the application hash table which will also tell the user on "lttng list" (new flag, new state).

• We have to notify the app thread of this possible error so the thread unregisters the app. It is crucial for synchronization.
• Some minor changes to libustctl to send back the errno code on recv/send error so the sessiond is able to know what went wrong.
• App is no longer traceable and needs to re-register again. (This should probably be done automatically just after the libust detects that the previous sockets died and that it should return in a wait mode).

Also, the SIGSTOP signal can't be caught by the application (see signal(7)) so this needs to be handled on the session daemon side.

Btw, this also affects 2.0.x-stable.

Thanks!

UST and tools now got a fixed that shutdown the socket when a socket failure is detected making the session daemon automatically delete the application.

The branch 'bug386' of my personal repository adds the timeout so can you please test it and see if it fixes your use case(s)? On my side, we have the desired effect which is a timeout on the recvmsg() of the UST command on the session daemon side followed by the application unregistering.

http://git.dorsal.polymtl.ca/~dgoulet?p=lttng-tools-next.git;a=shortlog;h=refs/heads/bug386

Thanks!

I pushed this one upstream (commit: ae9e45b342636e7b42eafc15cd105bccfbbbe373).

Any issue you come across or show stopper, we'll revert it or try to fix it as quickly as possible.

Thanks!