Project

General

Profile

Actions

Bug #624

closed

sessiond crash at system shutdown

Added by Stanislav Vovk about 8 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
08/30/2013
Due date:
% Done:

100%

Estimated time:

Description

Commit id:
UST: 0ff43c44ea52d0d833ea17f15777622ec64d1b1a
tools: 95f3c88c5b889bb3d16bb6857f137c84e06b84c2
urcu: b2f20dff3e0ce8db5f4f0067e2d8810420f16346

Problem:
Sessiond can sometimes crash at system shutdown. Sessiond is started as root at system boot.
Session is active when shutdown starts i.e. 'lttng stop' or 'destroy' never run.

Configured with:

su_sirpa()
{
    su - sirpa -c "$1" 2> /dev/null > /dev/kmsg
}
   lttng create $LTTNG_SES_NAME --snapshot -o $LTTNG_TELOG_PATH/ > /dev/kmsg
    lttng enable-channel $LTTNG_TRI_CHN_NAME -u -s $LTTNG_SES_NAME > /dev/kmsg
    lttng start $LTTNG_SES_NAME > /dev/kmsg

    ## Create session as sirpa, sirpa then can control this session
    cmd="lttng create $LTTNG_DEF_SES_NAME --snapshot -o $LTTNG_DEF_OUTPUT" 
    su_sirpa "$cmd" 

    lttng enable-channel $LTTNG_DEF_CHN_NAME -u -s $LTTNG_DEF_SES_NAME > /dev/kmsg
    lttng start $LTTNG_DEF_SES_NAME > /dev/kmsg

Backtrace:

Core was generated by `/usr/bin/lttng-sessiond -d --no-kernel'.
(gdb) bt
#0  0x0fcb0aa0 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0fcb5930 in abort () at abort.c:91
#2  0x0fca899c in __assert_fail_base (
    fmt=0xfdc2218 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=<value optimized out>, file=0x1004ae94 "main.c", 
    line=<value optimized out>, function=<value optimized out>) at assert.c:94
#3  0x0fca8a54 in __assert_fail (assertion=0x1003e480 "fd >= 0", 
    file=0x1004ae94 "main.c", line=1372, 
    function=0x1004a870 "send_socket_to_thread") at assert.c:103
#4  0x1002941c in send_socket_to_thread (fd=-1, sock=30) at main.c:1372
#5  0x1002b4e0 in thread_dispatch_ust_registration (data=<value optimized out>)
    at main.c:1657
#6  0x0fe0a84c in start_thread (arg=0x32026060) at pthread_create.c:311
#7  0x0fd616f4 in clone ()
   from /repo/ebennik/tmp/dumps/dbg/RCS/DBG/ROOTFS_CXC1733870_1/dbg/lib/libc.so.6
(gdb) 

Following is done to running programs at system shutdown:

# Kill all processes.
echo "Sending all processes the TERM signal..." 
killall5 -15
sleep 5
echo "Sending all processes the KILL signal..." 
killall5 -9

A more complete bt will be provided when debug binaries/libs are in place.

Actions #1

Updated by Mathieu Desnoyers about 8 years ago

Hi Stanislav,

Commit id:
UST: 0ff43c44ea52d0d833ea17f15777622ec64d1b1a
tools: 95f3c88c5b889bb3d16bb6857f137c84e06b84c2
urcu: b2f20dff3e0ce8db5f4f0067e2d8810420f16346

I notice that these commits are old (2.3-rc2). Can you try to reproduce
against tools and ust master branches ? They are currently at commits:

tools: 8a251cb8ade9db2f560604fa53b3515ca0d60c22
ust: 256175cffb00b15543bd49a4268e6c528bd51fce

Thanks!

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Actions #2

Updated by David Goulet about 8 years ago

  • Status changed from New to Confirmed
  • Assignee set to David Goulet
  • Target version set to 2.2

So yes, there is a race condition that can trigger this assert.

I'm about to push a fix in master branch and stable-2.2 as well. The commit will close this issue automatically but again I'll would recommend for you to use the latest git head (that will include a fix).

Thanks!

Actions #3

Updated by David Goulet about 8 years ago

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF