Bug #1185
closedFailing to enable kernel channel through Python bindings
0%
Description
I'm trying to use the Python bindings to setup & start a tracing session.
Creating a (kernel) channel fails with a "Not enough memory" error message (returned by the enable_channel() call). Removing the kernel stuff and only using a UST domain+tracepoints works fine. Also, using a bash script to setup a UST+kernel session through CLI works fine.
Note that I'm calling the Python script through a ROS2 command, but it shouldn't affect anything.
Python script using bindings: https://pastebin.com/RWjPcReg
Command output (with LTTNG_UST_DEBUG): https://pastebin.com/mnA5sT2L
lttng-sessiond logs (after a lttng-sessiond stop+start): https://pastebin.com/Lq9LCyG9
Version/other info
Both from PPA:
liblttng-ust-dev:amd64/bionic 2.10.1-1
lttng-tools:amd64/bionic 2.10.2-1
$ uname -a
Linux HP-EliteBook-840-G3 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 7,8G 178M 7,6G 3% /dev/shm
$ cat /proc/meminfo
https://pastebin.com/8ihp27YD
$ lsmod | grep lttng
https://pastebin.com/iT0srSjy
Files
Updated by Christophe Bedard over 5 years ago
lttng-modules-dkms:all/bionic-updates 2.10.5-1ubuntu1.2
Updated by Christophe Bedard over 5 years ago
$ lttng-sessiond -vvv --verbose-consumer
https://pastebin.com/VdDEVRpA
and the corresponding command that was run
$ ros2 run tracetools_analysis trace my-session-nameee /tmp
https://pastebin.com/qPCepvta
Updated by Christophe Bedard over 5 years ago
$ groups
<username> adm sudo lpadmin sambashare tracing
(with <username> being my username)
Updated by Christophe Bedard over 5 years ago
$ sudo lttng-sessiond -vvv --verbose-consumer
https://gist.github.com/christophebedard/693ac2100e84ecefe60fc2c7134d75e5
Updated by Jonathan Rajotte Julien over 5 years ago
- Status changed from New to Confirmed
Hi,
The problem is due to the lack of update of the python binding through time and the addition of extended attributes to the channel object.
commit cf0bcb51ea857687a353d2851e572dba6cc63cb0 Author: Jérémie Galarneau <jeremie.galarneau@efficios.com> Date: Thu May 4 23:53:50 2017 -0400 Expose monitor timer interval to lttngctl and client Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>
The relevant part in lttng_enable_channel:
if (!in_chan->attr.extended.ptr) { struct lttng_channel *channel; struct lttng_channel_extended *extended; channel = lttng_channel_create(&handle->domain); if (!channel) { return -LTTNG_ERR_NOMEM; } /* * Create a new channel in order to use default extended * attribute values. */ extended = (struct lttng_channel_extended *) channel->attr.extended.ptr; memcpy(&lsm.u.channel.extended, extended, sizeof(*extended)); lttng_channel_destroy(channel);
Notice the LTTNG_ERR_NOMEM. Using gdb we find that lttng_channel_create fail on the validation for the domain buffer type. Note that currently the buffer type is not exposed for the Domain object in the bindings and default to 0.
case LTTNG_DOMAIN_KERNEL: if (domain->buf_type != LTTNG_BUFFER_GLOBAL) { goto error; } break;
The default value (0) is LTTNG_BUFFER_PER_PID which correctly lead to an error in lttng_create_channel.
We are not sure yet how to fix this mess. Will keep you updated.
Cheers
Updated by Jonathan Rajotte Julien over 5 years ago
- File 0001-Fix-python-binding-expose-domain-buffer-type.patch 0001-Fix-python-binding-expose-domain-buffer-type.patch added
- File 0002-Update-base-test-for-binding.patch 0002-Update-base-test-for-binding.patch added
Hi Christophe,
Would you be able to build a 2.10 lttng-tools from source and test the attached patches?
When building from source you will need to enable the python bindings at configure time.
./configure --enable-python-bindings
You will also need to add the following for both your kernel and userspace Domain object.
domain_kernel.buf_type = BUFFER_GLOBAL domain_ust.buf_type = BUFFER_PER_UID
Unless you want per-pid buffering, which is rarely the case.
Cheers
Updated by Christophe Bedard over 5 years ago
Hi,
So I applied your two patches to stable-2.10
, built it (and obviously had to play around with sys.path
, but that's unrelated), and I confirm that it works!
Here's the updated Python script, just so everything's clear: https://gist.github.com/christophebedard/ecc6027dadb3ef4f8d9f39ab0808f809
Let me know if you need me to test anything else/need anything else from me!
Updated by Jonathan Rajotte Julien over 5 years ago
Unless we decide to fix it in another way, nothing should be required from your end.
Will let you know, when we have a fix upstream.
Thanks for testing it.
Cheers
Updated by Jonathan Rajotte Julien over 5 years ago
- Status changed from Confirmed to Resolved
Hi,
The fix is part of lttng 2.10.7 stable release.
https://github.com/lttng/lttng-tools/commit/93fa2c9ff6b52c30173bee80445501ce8677fecc
The ubuntu latest stable ppa should do the trick [1].
The fix should land in the stable 2.10 ppa soon.
[1] https://launchpad.net/~lttng/+archive/ubuntu/ppa
Cheers
Updated by Christophe Bedard about 5 years ago
Hi,
Do you know if and how we could request for >=2.10.7 to be released to Ubuntu packages for bionic [1]?
[1] https://packages.ubuntu.com/bionic/python3-lttng
Thanks!
Updated by Mathieu Desnoyers about 5 years ago
Adding our packager Michael Jeanson to the Watchers list.
Updated by Michael Jeanson about 5 years ago
Christophe Bedard wrote:
Hi,
Do you know if and how we could request for >=2.10.7 to be released to Ubuntu packages for bionic [1]?
[1] https://packages.ubuntu.com/bionic/python3-lttng
Thanks!
Ubuntu doesn't allow minor version updates of packages in their stable releases (there are some exceptions, but we are not one of them), you could open a bug about your specific problem and a package of the current version plus a patch fixing it could be uploaded.
You can also use the stable-2.10 PPA1 which has the latest releases, it lagged a bit in the lasts months because of the Debian freeze but it's up to date now.
[1] https://launchpad.net/~lttng/+archive/ubuntu/stable-2.10