Actions
Bug #1275
closedkernel trace metadata file is placed outside the kernel/ directory
Start date:
06/29/2020
Due date:
% Done:
0%
Estimated time:
Description
Host:
- Ubuntu 18.04 5.3.0-61-generic
- Docker version 19.03.12, build 48a66213fe
- lttng-modules-dkms 2.11.2-1~ubuntu18.04.1 from the stable-2.11 PPA (I think my host also has lttng-ust & lttng-tools from the same PPA, but I assume it doesn't matter here)
lttng (LTTng Trace Control) 2.11.2 - Lafontaine
Docker container:
- Ubuntu 18.04.1 (same: 5.3.0-61-generic)
- The `/lib/modules/5.3.0-61-generic` dir from the host is mounted inside the container (same directory), and I do see the `lttng-*.ko` files under `/lib/modules/`uname -r`/updates/dkms/`
- Compiled from source from the `stable-2.11` branches: lttng-ust (89909e362150f437c619f627e20bb4434a1566aa), lttng-tools (14a5bf2843c2bdc13c3a4ec3335a13c867ba3619)
lttng (LTTng Trace Control) 2.11.4 - Lafontaine - v2.11.4-23-g14a5bf28
If I setup simple session with a few kernel syscall events, the resulting trace cannot be read with babeltrace. Upon inspection, I found that the metadata file is outside the kernel/ directory (so right under the trace directory itself). If I move it to the kernel/ directory, babeltrace can read the trace fine.
E.g.
$ lttng create ksesh $ lttng enable-event -k --syscall open,read,poll $ lttng start $ # wait $ lttng stop $ lttng destroy $ cd lttng-traces/ $ babeltrace ksesh-20200629-120439/ [warning] Skipping directory 'kernel' found in trace $ ll ksesh-20200629-120439/ kernel metadata $ mv ksesh-20200629-120439/metadata ksesh-20200629-120439/kernel/ $ babeltrace ksesh-20200629-120439/ (a lot of events)
I'm currently trying to setup a minimal docker image to reproduce.
Updated by Christophe Bedard almost 4 years ago
I was able to reproduce.
Install `lttng-modules-dkms` from the stable-2.11 PPA on the host:
$ sudo apt-add-repository ppa:lttng/stable-2.11 $ sudo apt-get update $ sudo apt-get install lttng-modules-dkms
Then start a Docker container with some options (make sure to modify if the kernel is different):
$ export DOCKER_RUN_ARGS=" --dns=8.8.8.8 --dns=8.8.4.4 --cap-add=SYS_PTRACE --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --ulimit rtprio=100:100 -v /var/log/coredumps:/var/log/coredumps -v /lib/modules/5.3.0-61-generic:/lib/modules/5.3.0-61-generic -v /sys:/sys --privileged " $ docker run $DOCKER_RUN_ARGS -it ubuntu:18.04
Then, inside the container, compile and install lttng-ust & lttng-tools and trace:
#!/usr/bin/env bash cd /home apt-get update apt-get install -q -y vim git software-properties-common autoconf libtool libnuma-dev make # lttng-ust git clone https://github.com/lttng/lttng-ust.git --branch stable-2.11 cd lttng-ust/ apt-get update && apt-get install -y liburcu-dev asciidoc xmlto ./bootstrap ./configure --prefix=/usr make make install ldconfig # lttng-tools cd ../ git clone https://github.com/lttng/lttng-tools.git --branch stable-2.11 cd lttng-tools/ apt-get update && apt-get install -y bison flex libpfm4-dev uuid-dev python3-dev man libpopt-dev libxml2-dev libkmod-dev swig ./bootstrap ./configure --prefix=/usr --enable-python-bindings --enable-embedded-help make make install ldconfig # test kernel tracing lttng-sessiond --daemonize lttng create ksesh --output=/root/lttng-traces/ksesh # --output is just to make the next com mands more explicit, but it doesn't change anything lttng enable-event -k --syscall open,read,poll lttng start #... lttng stop lttng destroy # read apt-get install -q -y babeltrace cd /root/lttng-traces/ babeltrace ksesh/ # should see the [warning] ll ksesh/ # should see the kernel/ dir and the metadata file right under the trace dir mv ksesh/metadata ksesh/kernel/ babeltrace ksesh # should work!
Updated by Christophe Bedard almost 4 years ago
This change seemed suspicious:
so I applied this fix:
diff --git a/src/bin/lttng-sessiond/kernel-consumer.c b/src/bin/lttng-sessiond/kernel-consumer.c
index a3295e92..f6eb497b 100644
--- a/src/bin/lttng-sessiond/kernel-consumer.c
+++ b/src/bin/lttng-sessiond/kernel-consumer.c
@@ -231,7 +231,7 @@ int kernel_consumer_add_metadata(struct consumer_socket *sock,
consumer_init_add_channel_comm_msg(&lkm,
ksession->metadata->key,
ksession->id,
- "",
+ DEFAULT_KERNEL_TRACE_DIR,
ksession->uid,
ksession->gid,
consumer->net_seq_index,
and the metadata file is now correctly under the kernel/ directory and the trace can be read fine.
Therefore it probably has nothing to do with my particular setup (i.e. docker).
Updated by Jérémie Galarneau almost 4 years ago
- Status changed from New to Resolved
- Assignee set to Jérémie Galarneau
Fixed by:
commit 8b3be07a094ad88c73990d5c945ac1b24582f479
Author: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Date: Mon Jun 29 17:29:03 2020 -0400
Fix: kernel metadata file outside of kernel/ directory
3ef395a9, a backport of a2814ea, introduces a regression that
causes the kernel trace metadata file to be created outside of
the "kernel" domain sub-directory.
master and stable-2.12 no longer need the domain sub-directory
to be passed to the consumer daemon. However, it was still used
in the 2.11 release.
Fixes #1275
Reported-by: Christophe Bedard <bedard.christophe@gmail.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I1fe10ddd414cc87264b5e6d6a6d81e45ebc800d4
Actions