Project

General

Profile

Actions

Bug #1185

closed

Failing to enable kernel channel through Python bindings

Added by Christophe Bedard almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
05/21/2019
Due date:
% Done:

0%

Estimated time:

Description

I'm trying to use the Python bindings to setup & start a tracing session.

Creating a (kernel) channel fails with a "Not enough memory" error message (returned by the enable_channel() call). Removing the kernel stuff and only using a UST domain+tracepoints works fine. Also, using a bash script to setup a UST+kernel session through CLI works fine.

Note that I'm calling the Python script through a ROS2 command, but it shouldn't affect anything.

Python script using bindings: https://pastebin.com/RWjPcReg
Command output (with LTTNG_UST_DEBUG): https://pastebin.com/mnA5sT2L
lttng-sessiond logs (after a lttng-sessiond stop+start): https://pastebin.com/Lq9LCyG9

Version/other info

Both from PPA:
liblttng-ust-dev:amd64/bionic 2.10.1-1
lttng-tools:amd64/bionic 2.10.2-1

$ uname -a
Linux HP-EliteBook-840-G3 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 7,8G 178M 7,6G 3% /dev/shm

$ cat /proc/meminfo
https://pastebin.com/8ihp27YD

$ lsmod | grep lttng
https://pastebin.com/iT0srSjy


Files

Actions #1

Updated by Christophe Bedard almost 5 years ago

lttng-modules-dkms:all/bionic-updates 2.10.5-1ubuntu1.2

Actions #2

Updated by Christophe Bedard almost 5 years ago

$ lttng-sessiond -vvv --verbose-consumer
https://pastebin.com/VdDEVRpA

and the corresponding command that was run

$ ros2 run tracetools_analysis trace my-session-nameee /tmp
https://pastebin.com/qPCepvta

Actions #3

Updated by Christophe Bedard almost 5 years ago

$ groups
<username> adm sudo lpadmin sambashare tracing

(with <username> being my username)

Actions #5

Updated by Jonathan Rajotte Julien almost 5 years ago

  • Status changed from New to Confirmed

Hi,

The problem is due to the lack of update of the python binding through time and the addition of extended attributes to the channel object.

commit cf0bcb51ea857687a353d2851e572dba6cc63cb0
Author: Jérémie Galarneau <jeremie.galarneau@efficios.com>
Date:   Thu May 4 23:53:50 2017 -0400

    Expose monitor timer interval to lttngctl and client

    Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>

The relevant part in lttng_enable_channel:

if (!in_chan->attr.extended.ptr) {
        struct lttng_channel *channel;
        struct lttng_channel_extended *extended;

        channel = lttng_channel_create(&handle->domain);
        if (!channel) {
            return -LTTNG_ERR_NOMEM;
        }

        /*
         * Create a new channel in order to use default extended
         * attribute values.
         */
        extended = (struct lttng_channel_extended *)
                channel->attr.extended.ptr;
        memcpy(&lsm.u.channel.extended, extended, sizeof(*extended));
        lttng_channel_destroy(channel);

Notice the LTTNG_ERR_NOMEM. Using gdb we find that lttng_channel_create fail on the validation for the domain buffer type. Note that currently the buffer type is not exposed for the Domain object in the bindings and default to 0.

    case LTTNG_DOMAIN_KERNEL:
        if (domain->buf_type != LTTNG_BUFFER_GLOBAL) {
            goto error;
        }
        break;

The default value (0) is LTTNG_BUFFER_PER_PID which correctly lead to an error in lttng_create_channel.

We are not sure yet how to fix this mess. Will keep you updated.

Cheers

Updated by Jonathan Rajotte Julien almost 5 years ago

Hi Christophe,

Would you be able to build a 2.10 lttng-tools from source and test the attached patches?

When building from source you will need to enable the python bindings at configure time.

./configure --enable-python-bindings

You will also need to add the following for both your kernel and userspace Domain object.

domain_kernel.buf_type = BUFFER_GLOBAL
domain_ust.buf_type = BUFFER_PER_UID

Unless you want per-pid buffering, which is rarely the case.

Cheers

Actions #7

Updated by Christophe Bedard almost 5 years ago

Hi,

So I applied your two patches to stable-2.10, built it (and obviously had to play around with sys.path, but that's unrelated), and I confirm that it works!

Here's the updated Python script, just so everything's clear: https://gist.github.com/christophebedard/ecc6027dadb3ef4f8d9f39ab0808f809

Let me know if you need me to test anything else/need anything else from me!

Actions #8

Updated by Jonathan Rajotte Julien almost 5 years ago

Unless we decide to fix it in another way, nothing should be required from your end.

Will let you know, when we have a fix upstream.

Thanks for testing it.

Cheers

Actions #9

Updated by Jonathan Rajotte Julien almost 5 years ago

  • Status changed from Confirmed to Resolved

Hi,

The fix is part of lttng 2.10.7 stable release.

https://github.com/lttng/lttng-tools/commit/93fa2c9ff6b52c30173bee80445501ce8677fecc

The ubuntu latest stable ppa should do the trick [1].

The fix should land in the stable 2.10 ppa soon.

[1] https://launchpad.net/~lttng/+archive/ubuntu/ppa

Cheers

Actions #10

Updated by Christophe Bedard almost 5 years ago

Awesome thank you!

Actions #11

Updated by Christophe Bedard almost 5 years ago

Hi,

Do you know if and how we could request for >=2.10.7 to be released to Ubuntu packages for bionic [1]?

[1] https://packages.ubuntu.com/bionic/python3-lttng

Thanks!

Actions #12

Updated by Mathieu Desnoyers almost 5 years ago

Adding our packager Michael Jeanson to the Watchers list.

Actions #13

Updated by Michael Jeanson almost 5 years ago

Christophe Bedard wrote:

Hi,

Do you know if and how we could request for >=2.10.7 to be released to Ubuntu packages for bionic [1]?

[1] https://packages.ubuntu.com/bionic/python3-lttng

Thanks!

Ubuntu doesn't allow minor version updates of packages in their stable releases (there are some exceptions, but we are not one of them), you could open a bug about your specific problem and a package of the current version plus a patch fixing it could be uploaded.

You can also use the stable-2.10 PPA1 which has the latest releases, it lagged a bit in the lasts months because of the Debian freeze but it's up to date now.

[1] https://launchpad.net/~lttng/+archive/ubuntu/stable-2.10

Actions

Also available in: Atom PDF