Bug #1275
closed
kernel trace metadata file is placed outside the kernel/ directory
Added by Christophe Bedard almost 4 years ago.
Updated almost 4 years ago.
Description
Host:
- Ubuntu 18.04 5.3.0-61-generic
- Docker version 19.03.12, build 48a66213fe
- lttng-modules-dkms 2.11.2-1~ubuntu18.04.1 from the stable-2.11 PPA (I think my host also has lttng-ust & lttng-tools from the same PPA, but I assume it doesn't matter here)
lttng (LTTng Trace Control) 2.11.2 - Lafontaine
Docker container:
- Ubuntu 18.04.1 (same: 5.3.0-61-generic)
- The `/lib/modules/5.3.0-61-generic` dir from the host is mounted inside the container (same directory), and I do see the `lttng-*.ko` files under `/lib/modules/`uname -r`/updates/dkms/`
- Compiled from source from the `stable-2.11` branches: lttng-ust (89909e362150f437c619f627e20bb4434a1566aa), lttng-tools (14a5bf2843c2bdc13c3a4ec3335a13c867ba3619)
lttng (LTTng Trace Control) 2.11.4 - Lafontaine - v2.11.4-23-g14a5bf28
If I setup simple session with a few kernel syscall events, the resulting trace cannot be read with babeltrace. Upon inspection, I found that the metadata file is outside the kernel/ directory (so right under the trace directory itself). If I move it to the kernel/ directory, babeltrace can read the trace fine.
E.g.
$ lttng create ksesh
$ lttng enable-event -k --syscall open,read,poll
$ lttng start
$ # wait
$ lttng stop
$ lttng destroy
$ cd lttng-traces/
$ babeltrace ksesh-20200629-120439/
[warning] Skipping directory 'kernel' found in trace
$ ll ksesh-20200629-120439/
kernel metadata
$ mv ksesh-20200629-120439/metadata ksesh-20200629-120439/kernel/
$ babeltrace ksesh-20200629-120439/
(a lot of events)
I'm currently trying to setup a minimal docker image to reproduce.
I was able to reproduce.
Install `lttng-modules-dkms` from the stable-2.11 PPA on the host:
$ sudo apt-add-repository ppa:lttng/stable-2.11
$ sudo apt-get update
$ sudo apt-get install lttng-modules-dkms
Then start a Docker container with some options (make sure to modify if the kernel is different):
$ export DOCKER_RUN_ARGS="
--dns=8.8.8.8
--dns=8.8.4.4
--cap-add=SYS_PTRACE
--cap-add=SYS_ADMIN
--cap-add=NET_ADMIN
--ulimit rtprio=100:100
-v /var/log/coredumps:/var/log/coredumps
-v /lib/modules/5.3.0-61-generic:/lib/modules/5.3.0-61-generic
-v /sys:/sys
--privileged
"
$ docker run $DOCKER_RUN_ARGS -it ubuntu:18.04
Then, inside the container, compile and install lttng-ust & lttng-tools and trace:
#!/usr/bin/env bash
cd /home
apt-get update
apt-get install -q -y vim git software-properties-common autoconf libtool libnuma-dev make
# lttng-ust
git clone https://github.com/lttng/lttng-ust.git --branch stable-2.11
cd lttng-ust/
apt-get update && apt-get install -y liburcu-dev asciidoc xmlto
./bootstrap
./configure --prefix=/usr
make
make install
ldconfig
# lttng-tools
cd ../
git clone https://github.com/lttng/lttng-tools.git --branch stable-2.11
cd lttng-tools/
apt-get update && apt-get install -y bison flex libpfm4-dev uuid-dev python3-dev man libpopt-dev libxml2-dev libkmod-dev swig
./bootstrap
./configure --prefix=/usr --enable-python-bindings --enable-embedded-help
make
make install
ldconfig
# test kernel tracing
lttng-sessiond --daemonize
lttng create ksesh --output=/root/lttng-traces/ksesh # --output is just to make the next com
mands more explicit, but it doesn't change anything
lttng enable-event -k --syscall open,read,poll
lttng start
#...
lttng stop
lttng destroy
# read
apt-get install -q -y babeltrace
cd /root/lttng-traces/
babeltrace ksesh/ # should see the [warning]
ll ksesh/ # should see the kernel/ dir and the metadata file right under the trace dir
mv ksesh/metadata ksesh/kernel/
babeltrace ksesh # should work!
This change seemed suspicious:
https://github.com/ApexAI/lttng-tools/commit/3ef395a9e7ef9adfc3cbb7d93ab8618bcc373f70#diff-cfa481df90353f26a4cbfdbfad37aa86L233
so I applied this fix:
diff --git a/src/bin/lttng-sessiond/kernel-consumer.c b/src/bin/lttng-sessiond/kernel-consumer.c
index a3295e92..f6eb497b 100644
--- a/src/bin/lttng-sessiond/kernel-consumer.c
+++ b/src/bin/lttng-sessiond/kernel-consumer.c
@@ -231,7 +231,7 @@ int kernel_consumer_add_metadata(struct consumer_socket *sock,
consumer_init_add_channel_comm_msg(&lkm,
ksession->metadata->key,
ksession->id,
- "",
+ DEFAULT_KERNEL_TRACE_DIR,
ksession->uid,
ksession->gid,
consumer->net_seq_index,
and the metadata file is now correctly under the kernel/ directory and the trace can be read fine.
Therefore it probably has nothing to do with my particular setup (i.e. docker).
- Status changed from New to Resolved
- Assignee set to Jérémie Galarneau
Fixed by:
commit 8b3be07a094ad88c73990d5c945ac1b24582f479
Author: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Date: Mon Jun 29 17:29:03 2020 -0400
Fix: kernel metadata file outside of kernel/ directory
3ef395a9, a backport of a2814ea, introduces a regression that
causes the kernel trace metadata file to be created outside of
the "kernel" domain sub-directory.
master and stable-2.12 no longer need the domain sub-directory
to be passed to the consumer daemon. However, it was still used
in the 2.11 release.
Fixes #1275
Reported-by: Christophe Bedard <bedard.christophe@gmail.com>
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Change-Id: I1fe10ddd414cc87264b5e6d6a6d81e45ebc800d4
Also available in: Atom
PDF