Project

General

Profile

Actions
Copy link

Bug #386

closed

A hanging instrumented application can cause consummerD to hang, which in turn will cause sessiond to hang

Added by Tan le tran about 12 years ago. Updated about 12 years ago.

Status:
Resolved
Priority:
Critical
Assignee:
David Goulet
Target version:
LTTng - 2.1 stable
Start date:
10/26/2012
Due date:
% Done:

100%

Estimated time:

Description

Description: 
============ 
  After a session has been activated, if an instrumented app (that is currently
  being traced on by the session) hang, then it will cause consumerD to hang
  as well. Any lttng command towards sessiond that needs interaction with 
  consumerD will also cause sessionD to hang and that will lead to unavailability
  of tracing.

  Killing sessonD will not help (see bug-367) .

Commit used: 
============ 
  userspace:   e7e6ff7 rculfhash test: fix trivial memleak and return node leak and errors
  lttng-ust:   1c7b4a9 Fix: memcpy of string is larger than source
  lttng-tools: dda67f6  Fix: Error handling when sending relayd sockets to consumer
  babeltrace : 6ca30a4 Cleanup: fix cppcheck warning

Scenario: 
========= 
# lttng create ses1
# lttng enable-event com_ericsson_cba_trace_testapp_lowtraf:OnePerSecB -u
# lttng start
# lttng list
    Available tracing sessions:
      1) ses1 (/root/lttng-traces/ses1-20121026-153048) [active]

# lttng list -u
    UST events:
    -------------
    None

# /home/test_apps/TestApp_type1 5555 np &
# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:00 TestApp_type1
     5469 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31730 ?        00:00:00 lttng-sessiond
    31823 ?        00:00:00 lttng-consumerd
# /home/test_apps/TestApp_Thread1 44444 np &
# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:01 TestApp_type1
      690 pts/2    00:00:00 TestApp_Thread1
     5469 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31730 ?        00:00:00 lttng-sessiond
    31823 ?        00:00:00 lttng-consumerd
# pkill -STOP TestApp_type1
    [1]+  Stopped                 /home/test_apps/TestApp_type1 5555 np
# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:02 TestApp_type1
      690 pts/2    00:00:00 TestApp_Thread1
     5469 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31730 ?        00:00:00 lttng-sessiond
    31823 ?        00:00:00 lttng-consumerd
# lttng create ses2
     Session ses2 created.
     Traces will be written in /root/lttng-traces/ses2-20121026-153239
# lttng enable-event com_ericsson_cba_trace_testapp_thread_main:OnePerSecA -u
     Command hang !!!
# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:02 TestApp_type1
      690 pts/2    00:00:05 TestApp_Thread1
     1176 pts/2    00:00:00 lttng   <-------------------
     5469 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31730 ?        00:00:00 lttng-sessiond
    31823 ?        00:00:00 lttng-consumerd

# pkill -9 sessiond
# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:02 TestApp_type1
      690 pts/2    00:00:08 TestApp_Thread1
     1233 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31823 ?        00:00:00 lttng-consumerd

# lttng-sessiond -vvv &
    DEBUG3: Creating LTTng run directory: /var/run/lttng [in create_lttng_rundir() at main.c:3525]
    DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in main() at main.c:3773]
    DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command [in main() at main.c:3775]
    DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in main() at main.c:3828]
    DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in main() at main.c:3829]
    DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:3830]
    DEBUG2: UST consumer 32 bits err path: /var/run/lttng/ustconsumerd32/error [in main() at main.c:3839]
    DEBUG2: UST consumer 32 bits cmd path: /var/run/lttng/ustconsumerd32/command [in main() at main.c:3841]
    DEBUG2: UST consumer 64 bits err path: /var/run/lttng/ustconsumerd64/error [in main() at main.c:3850]
    DEBUG2: UST consumer 64 bits cmd path: /var/run/lttng/ustconsumerd64/command [in main() at main.c:3852]
    Error: Already running daemon.

    [3]-  Exit 1                  lttng-sessiond -vvv

# ps -e |egrep -i 'lttng|TestApp|trace'
      646 pts/2    00:00:02 TestApp_type1
      690 pts/2    00:00:08 TestApp_Thread1
     1233 ?        00:00:00 TraceEa
     9961 ?        00:00:20 TraceEa
    31823 ?        00:00:00 lttng-consumerd
# pkill -9 lttng-consumerd      (on the background, there is a process continously attempt to launch a new sessiond)
# ps -e |egrep -i 'lttng|TestApp|trace'
  646 pts/2    00:00:02 TestApp_type1
  690 pts/2    00:00:09 TestApp_Thread1
 1233 ?        00:00:00 TraceEa
 1775 ?        00:00:00 lttng-sessiond    (sessiond can finally be launched)
 9961 ?        00:00:20 TraceEa
Actions
Copy link

Also available in: Atom PDF