LTTng » LTTng-UST

Hi Tan,

I would need the following additional info:

- build the instrumented application without optimisations (O0 instead
of -O2).
output, with gdb on core file:

"thread apply all bt full"

So I get the backtrace from all threads (there appears to be 3 threads).

print lttng_ust_comm_should_quit

You might want to try the patch I am attaching to this bug report, I
think it should help.

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Hi Mathieu,

We are pretty busy at the moment with our release activities.
I will give a try by early next week .
Will keep you posted.
/Tan

No problem, it is pushed into master/stable-2.1 now.

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Can you try with the following stable-2.1 commit ?

commit 164931d23d430b1770611ec0300720350261b346
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Wed Mar 6 10:07:12 2013 -0500

Fix: refcount issue in lttng-ust-abi.c

The following scenario could lead to a segmentation fault in
    applications when the sessiond disappears or when the application try to
    exit. This is caused by incorrect handling of reference counts.

This can happen in very particular race scenario, described as follows:

1) The sessiond asks for "release" of one or more objects (e.g. a
       session object), but without asking for release of all objects
       referencing the session.
    2) The application exits, thus calling objd_table_destroy(). It walks on
       all objects, decrementing their reference count, freeing memory when
       their reference count reaches "1".

However, here is the issue: since "release" has already been performed
    by sessiond on the session object, this extra reference count unref
    performed by objd_table_destroy() can make the session object disappear
    while it is still needed by either the channel object or the enabler
    object. Therefore, we can experience a segmentation fault when we try to
    unref and free the channel or enabler objects within
    objd_table_destroy().

Fix this issue by adding the concept of an "owner reference". Only
    objd_table_destroy(), lttng_ust_objd_table_owner_cleanup(), "release" 
    commands, and failure paths of object creation are allowed to decrement
    the owner reference. We restrict objd_table_destroy() and
    lttng_ust_objd_table_owner_cleanup() to only decrement refcount of
    objects if their owner reference is still held.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Hi Mathieu,

We have re-run our stress test scenario last weekend.
Unfortunately, the whole trace traffic got stuck in "deactivating" state on early morning of
saturday due to some issue where sessionD which keeps reporting to have more data pending (currently, it is still reporting more data pending even if "lttng list" shows that all sessions are in "inactive" state).
Therefore the run lasted only about 12hrs. So far, it looks very promissing. No segfault from testApp_Mini has been detected after that 12hrs run (usually we would have at least a couple).

We will try again with longer run (at least 48hrs) and let you know (hopefully, it will not got hit by the above deactivating issue). We will look into the log data to see what causes the "deactivating" issue.
Thanks,
Tan

Our stress test has been ran for more than 60hrs and the issue reported in this bug report is no longer seen. The fix works well.