Project

General

Profile

Actions

Bug #1413

open

lttng-sessiond is reporting error in syslog file while terminating the same process using systemd

Added by Sougata Barik about 1 month ago. Updated 13 days ago.

Status:
Feedback
Priority:
Normal
Target version:
-
Start date:
04/18/2024
Due date:
% Done:

0%

Estimated time:

Description

Hi,

I am getting below errors in /var/log/syslog file after systemctl stop lttng-sessiond. Please re-assign to actual owner if needed.

Steps to reproduce:
1. Start monitoring syslog file on a terminal( sudo tail -f /var/log/syslog)
1. Open another terminal and stop any lttng-sessiond client using systemd command ( sudo systemctl stop frr , frr daemon is a lttng-sessiond client in our product)
2. Stop lttng-sessiond using systemd command ( sudo systemctl stop lttng-sessiond )

Problem: Error logs are seen in syslog file.
2024-04-10T15:05:44.214240+00:00 cumulus systemd1: Stopping lttng-sessiond.service - LTTng session daemon...
2024-04-10T15:05:44.215127+00:00 cumulus lttng-sessiond679: PERROR - 15:05:44.214393532 [Main]: sendmsg: Broken pipe (in lttcomm_send_unix_sock() at unix.c:306)
2024-04-10T15:05:44.215306+00:00 cumulus lttng-sessiond679: Error: Failed to close trace chunk on user space consumer
2024-04-10T15:05:44.215402+00:00 cumulus lttng-sessiond679: Error: Failed to perform a quiet rotation as part of the destruction of session "frr-local-onboot": Trace chunk close failed on consumer
2024-04-10T15:05:44.215469+00:00 cumulus lttng-sessiond679: Error: Failed to close trace chunk on user space consumer
2024-04-10T15:05:44.215528+00:00 cumulus lttng-sessiond679: Error: Failed to perform a quiet rotation as part of the destruction of session "switchd-local-onboot": Trace chunk close failed on consumer
2024-04-10T15:05:44.215586+00:00 cumulus lttng-sessiond679: Error: Failed to close trace chunk on user space consumer
2024-04-10T15:05:44.215648+00:00 cumulus lttng-sessiond679: Error: Failed to perform a quiet rotation as part of the destruction of session "frr-local-zebra-onboot": Trace chunk close failed on consumer
2024-04-10T15:05:44.230005+00:00 cumulus systemd1: update-ports.service: Deactivated successfully. 2024-04-10T15:05:44.230219+00:00 cumulus systemd1: Stopped update-ports.service - Cumulus Linux switch port setup.
2024-04-10T15:05:44.245677+00:00 cumulus systemd1: update-ports.service: Consumed 1.057s CPU time.
2024-04-10T15:05:44.277922+00:00 cumulus lttng-sessiond717: Error: 3 trace chunks are leaked by lttng-consumerd. This can be caused by an internal error of the session daemon. 2024-04-10T15:05:44.278156+00:00 cumulus systemd1: Stopping hw-management-tc.service - Thermal control service (ver 2.0) of Mellanox systems...

Additional info:
cumulus@cumulus:mgmt:~$ uname -a
Linux cumulus 6.1.0-cl-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4+cl5.9.0u39 (2024-04-07) x86_64 GNU/Linux
cumulus@cumulus:mgmt:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 36 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU: 2
On-line CPU list: 0,1
Vendor ID: GenuineIntel
Model name: Intel(R) Celeron(R) CPU 1047UE @ 1.40GHz
CPU family: 6 Model: 58
Thread(s) per core: 1
Core(s) per socket: 2 Socket(s): 1
Stepping: 9
CPU scaling MHz: 79%
CPU max MHz: 1400.0000 CPU min MHz: 800.0000
BogoMIPS: 2796.35
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_ 2 x2apic popcnt tsc_deadline_timer xsave lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 64 KiB (2 instances)
L1i: 64 KiB (2 instances)
L2: 512 KiB (2 instances)
L3: 2 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU: 0,1
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
Meltdown: Mitigation; PTI Mmio stale data: Unknown: No mitigations
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
cumulus@cumulus:mgmt:~$ cat /etc/lttng/lttng.conf
cat: /etc/lttng/lttng.conf: No such file or directory
cumulus@cumulus:mgmt:~$ ps -ef | grep lttng-se
root 69597 1 0 14:28 ? 00:00:04 /usr/bin/lttng-sessiond --no-kernel -vvv --verbose-consumer > /lttng-sessiond-verbose.txt
root 69598 69597 0 14:28 ? 00:00:00 /usr/bin/lttng-sessiond --no-kernel -vvv --verbose-consumer > ~/lttng-sessiond-verbose.txt
cumulus 142157 16912 0 16:08 pts/1 00:00:00 grep lttng-se
cumulus@cumulus:mgmt:
$ lttng-sessiond --version
2.13.9
cumulus@cumulus:mgmt:~$ ps -wwaux | grep '[l]ttng'
root 69597 0.0 0.1 1024392 18168 ? Ssl 14:28 0:04 /usr/bin/lttng-sessiond --no-kernel -vvv --verbose-consumer > /lttng-sessiond-verbose.txt
root 69598 0.0 0.0 48868 2692 ? S 14:28 0:00 /usr/bin/lttng-sessiond --no-kernel -vvv --verbose-consumer > ~/lttng-sessiond-verbose.txt root 69612 0.0 0.2 503716 46324 ? Sl 14:28 0:05 lttng-consumerd --verbose -u --consumerd-cmd-sock /var/run/lttng/ustconsumerd64/command --consumerd-err-sock /var/run/lttng/ustconsumerd64/error --group tracing
root 69614 0.0 0.0 3684 1192 ? S 14:28 0:00 lttng-consumerd --verbose -u --consumerd-cmd-sock /var/run/lttng/ustconsumerd64/command --consumerd-err-sock /var/run/lttng/ustconsumerd64/error --group tracing
cumulus@cumulus:mgmt:
$ lsmod | grep lttng
cumulus@cumulus:mgmt:~$ lsmod | grep lttng


Files

Actions #1

Updated by Kienan Stewart about 1 month ago

  • Status changed from New to Feedback
  • Assignee changed from Erica Bugden to Kienan Stewart

Hi Sougata,

what commands are used by your systemd service (frr.service) to stop lttng-sessiond?

thanks,
kienan

Actions #2

Updated by Sougata Barik 29 days ago

sudo systemctl stop frr <<< This is the command to stop frr service
sudo systemctl stop lttng-sessiond << This is the command to stop lttng-service
If this is not the one what you want please elaborate your query.

Actions #3

Updated by Kienan Stewart 28 days ago

Hi Sougata,

I should have been clearer, my apologies.

With the systemctl stop frr, is an equivalent command to lttng stop [session] or lttng destroy [session] issued?

With the systemctl stop lttng-sessiond, does the command simply send sigterm to the lttng-sessiond process?

Could you provide the output of lttng-sessiond that is ran with the options: -vvv --verbose-consumer?

thanks,
kienan

Actions #4

Updated by Sougata Barik 13 days ago

Hi Kienan,

Please find the syslog with verbose.

Regards,
Sougata

Actions #5

Updated by Sougata Barik 13 days ago

With the systemctl stop frr, is an equivalent command to lttng stop [session] or lttng destroy [session] issued?
Sougata>> systmenctl is a linux tool name systemd to manage app running in the system. frr and lttng-sessiond both are different services and they can be start/restart/stop using systemctl(systemd).

With the systemctl stop lttng-sessiond, does the command simply send sigterm to the lttng-sessiond process?
Sougata>> I think so , we have not done any changes in this so, it should be a SIGTERM signal to lttng-sessiond.

Actions

Also available in: Atom PDF