Project

General

Profile

Bug #1181

Deadlock in session daemon and consumer for short lived apps in live per-pid

Added by Francis Deslauriers 3 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
04/01/2019
Due date:
% Done:

0%

Estimated time:

Description

I encountered a deadlock of the session daemon when tracing short lived UST app in burst. I was not able to reproduce it.
Here is the session configuration:

lttng create allo --live
lttng enable-channel --buffers-pid -s allo --tracefile-size=5M --tracefile-count=4 -u chan1
lttng enable-event -u -a -c chan1
lttng start allo

After configured this session, I ran the following for loop followed by a lttng destroy command. As such:
for i in `seq 1 10`; do
   ../../../lttng-emojis/main & ;
done;
lttng destroy -a

These apps run and exit very quickly. The lttng destroy command never completes .

I attached the source code of the app as well as the backtraces of both lttng session daemon and the lttng consumer daemon.

We gathered so far:
sessiond thread 8: waiting on consumer socket (recv), while holding the consumer socket lock
sessiond thread 11: waiting for socket lock, while holding ust registry lock
sessiond thread 15: waiting for ust registry lock

consumerd thread 4: waiting for stream lock
consumerd thread 7: waiting on consumer metadata socket(recv), while holding stream lock
consumerd thread 8: waiting for stream lock

I was running this lttng-tools branch: https://github.com/PSRCode/lttng-tools/tree/live-per-pid (commit f0e3b9ebe)


Files

consumer-bt.txt (6.66 KB) consumer-bt.txt Francis Deslauriers, 04/01/2019 03:20 PM
sessiond-bt.txt (11.6 KB) sessiond-bt.txt Francis Deslauriers, 04/01/2019 03:20 PM
emoji.tgz (1.06 KB) emoji.tgz Francis Deslauriers, 04/01/2019 03:28 PM
relayd-bt-2.txt (6.47 KB) relayd-bt-2.txt Francis Deslauriers, 04/01/2019 04:08 PM
consumerd-bt-2.txt (6.66 KB) consumerd-bt-2.txt Francis Deslauriers, 04/01/2019 04:08 PM
sessiond-bt-2.txt (11.7 KB) sessiond-bt-2.txt Francis Deslauriers, 04/01/2019 04:08 PM

History

#1

Updated by Francis Deslauriers 3 months ago

  • Description updated (diff)
#2

Updated by Francis Deslauriers 3 months ago

  • Description updated (diff)
#3

Updated by Francis Deslauriers 3 months ago

I was able to reproduce it by using the following loop:

for i in `seq 1 40`; do 
  ./setup-trace.sh ; 
  ../../../lttng-emojis/main &; 
  lttng destroy -a ;
done;

I attached the backtrace of this run.

#4

Updated by Francis Deslauriers 3 months ago

  • Description updated (diff)

Also available in: Atom PDF