Project

General

Profile

Bug #882

destroying one session stops tracing on all other sessions

Added by Anand Neeli over 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Target version:
-
Start date:
03/02/2015
Due date:
% Done:

0%

Estimated time:

Description

With lttng 2.6.0 with 0.8.6 liburcu see following issue:
On a multisession setup with relayd, Destroying one session is showing errors on the console and tracing of all other sessions is stopped.

steps to recreate are as follows
1) Create multiple sessions with relayd. (in below logs i have created 2 sessions)
2) Destroy one session and then tracing on all the sessions stops

(Have not check this with single session, could be happening with single session also)

logs
---------
node-a # lttng list
Available tracing sessions:
1) mys5 (tcp4://128.0.0.4:5342/ [data: 5343]) [active]
Trace path: tcp4://128.0.0.4:5342/ [data: 5343]
Live timer interval (usec): 2000000

2) mysession (tcp4://128.0.0.4:5342/ [data: 5343]) [active]
Trace path: tcp4://128.0.0.4:5342/ [data: 5343]
Live timer interval (usec): 2000000

Use lttng list <session_name> for more details

node-a # lttng list mys5
Tracing session mys5: [active]
Trace path: tcp4://128.0.0.4:5342/ [data: 5343]

=== Domain: UST global ===

Buffer type: per PID

Channels:
-------------
- myc5: [enabled]

Attributes:
overwrite mode: 0
subbufers size: 4096
number of subbufers: 4
switch timer interval: 0
read timer interval: 0
trace file count: 2
trace file size (bytes): 2000000
output: mmap()
Events:
  * (type: tracepoint) [enabled] [has exclusions]

node-a # lttng list mysession
Tracing session mysession: [active]
Trace path: tcp4://128.0.0.4:5342/ [data: 5343]

=== Domain: UST global ===

Buffer type: per PID

Channels:
-------------
- mychannel: [enabled]

Attributes:
overwrite mode: 0
subbufers size: 4096
number of subbufers: 4
switch timer interval: 0
read timer interval: 0
trace file count: 2
trace file size (bytes): 2000000
output: mmap()
Events:
  * (type: tracepoint) [enabled] [has exclusions]

node-a # lttng destroy mys5 <<<<<<<<<<<<<<<<<<<< destroying session here
Error: Pushing metadata
Error: Handling metadata request
Error: Health error occurred in thread_manage_consumer
Error: Pushing metadata
Error: Handling metadata request
Error: Health error occurred in thread_manage_consumer
PERROR - 04:36:15.258880 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
PERROR - 04:36:15.259520 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.260076 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
Session mys5 destroyed
node-a # PERROR - 04:36:15.106369 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106410 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106437 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106474 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106490 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106500 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106509 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106519 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106528 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106537 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106546 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106555 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106564 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106573 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106581 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106590 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106599 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106608 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106616 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106625 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106636 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106645 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106654 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106663 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106672 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106681 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106690 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106699 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106708 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond
PERROR - 04:36:15.106717 [1651/1656]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:218)
Error: Asking metadata to sessiond

node-a # ps aux | grep lttng
root 1622 0.1 0.0 495020 1160 ? Ssl 04:34 0:00 lttng-relayd -o /var/tmp/lttng-traces -d
root 1631 0.0 0.1 854020 6360 ? Ssl 04:34 0:00 lttng-sessiond --consumerd32-path /usr/lib/lttng/libexec/lttng-consumerd --consumerd32-libdir /usr/lib/ --consumerd64-path /usr/lib64/lttng/libexec/lttng-consumerd --consumerd64-libdir /usr/lib64/ -b --no-kernel
root 1643 0.0 0.0 546356 3424 ? Sl 04:34 0:00 lttng-consumerd -u --consumerd-cmd-sock /var/run/lttng/ustconsumerd64/command --consumerd-err-sock /var/run/lttng/ustconsumerd64/error --group tracing
root 1651 0.0 0.0 64068 1852 ? Sl 04:34 0:00 lttng-consumerd -u --consumerd-cmd-sock /var/run/lttng/ustconsumerd32/command --consumerd-err-sock /var/run/lttng/ustconsumerd32/error --group tracing

#1

Updated by Anonymous over 5 years ago

  • Project changed from Userspace RCU to LTTng-tools
#2

Updated by Jonathan Rajotte Julien almost 5 years ago

  • Assignee set to Jonathan Rajotte Julien
#3

Updated by Jonathan Rajotte Julien almost 5 years ago

  • Status changed from New to Feedback

Hi Anand,

I might be able to reproduce part of the bug. Could you provide me with the full order of command you execute (start of the relayd, session creation,sessiond etc.).

I get the error on push metadata but no other session stop.

Thanks

#4

Updated by Jérémie Galarneau about 4 years ago

  • Status changed from Feedback to Resolved

Closing since there were no updates for 9 months and the relay daemon has significantly changed since then. Please reopen if reproducible.

Also available in: Atom PDF