Project

General

Profile

Actions

Bug #1358

open

Failed to deploy lttng modules on NVIDIA jetson device

Added by liuhonggang liu 5 months ago. Updated 2 months ago.

Status:
New
Priority:
High
Target version:
Start date:
09/13/2022
Due date:
% Done:

0%

Estimated time:

Description

Hello, I installed lttng and lttng modules on NVIDIA Orin.
When using apt install, the results are as follows.

# lttng list --kernel
Error: Unable to list kernel events: Kernel tracer not available

#ps aux | grep lttng-sessiond
root       52100  0.0  0.0 1022064 12736 ?       Ssl  15:05   0:00 /usr/bin/lttng-sessiond
root       52101  0.0  0.0  41968   664 ?        S    15:05   0:00 /usr/bin/lttng-sessiond
orin-d     62549  0.0  0.0  11640   684 pts/0    S+   20:54   0:00 grep --color=auto lttng-sessiond

# dpkg -l | grep lttng
ii  liblttng-ctl0:arm64                                 2.12.4-1~ubuntu20.04.1                arm64        LTTng control and utility library
ii  liblttng-ust-ctl4:arm64                             2.12.2-1~ubuntu20.04.1                arm64        LTTng 2.0 Userspace Tracer (trace control library)
ii  liblttng-ust-dev:arm64                              2.12.2-1~ubuntu20.04.1                arm64        LTTng 2.0 Userspace Tracer (development files)
ii  liblttng-ust-python-agent0:arm64                    2.12.2-1~ubuntu20.04.1                arm64        LTTng 2.0 Userspace Tracer (Python agent native library)
ii  liblttng-ust0:arm64                                 2.12.2-1~ubuntu20.04.1                arm64        LTTng 2.0 Userspace Tracer (tracing libraries)
ii  lttng-modules-dkms                                  2.12.6-1~ubuntu20.04.1                all          Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii  lttng-tools                                         2.12.4-1~ubuntu20.04.1                arm64        LTTng control and utility programs
ii  python3-lttng                                       2.12.4-1~ubuntu20.04.1                arm64        LTTng control and utility Python bindings

The device information is as follows.

# uname -a
Linux orind-d 5.10.65-tegra #2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux

# cat /etc/os-release
NAME="Ubuntu" 
VERSION="20.04.4 LTS (Focal Fossa)" 
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS" 
VERSION_ID="20.04" 
HOME_URL="https://www.ubuntu.com/" 
SUPPORT_URL="https://help.ubuntu.com/" 
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

At the same time, I tried the method of source code, such as the source code installation method introduced in https://lttng.org/docs/v2.13/.

# dpkg -l | grep -e libuuid -e popt -e userspace -e libxml2
ii  can-utils                                           2018.02.0-1ubuntu1                    arm64        SocketCAN userspace utilities and tools
ii  dmsetup                                             2:1.02.167-1ubuntu1                   arm64        Linux Kernel Device Mapper userspace library
ii  gvfs:arm64                                          1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - GIO module
ii  gvfs-backends                                       1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - backends
ii  gvfs-bin                                            1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - deprecated command-line tools
ii  gvfs-common                                         1.44.1-1ubuntu1                       all          userspace virtual filesystem - common data files
ii  gvfs-daemons                                        1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - servers
ii  gvfs-fuse                                           1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - fuse server
ii  gvfs-libs:arm64                                     1.44.1-1ubuntu1                       arm64        userspace virtual filesystem - private libraries
ii  libdevmapper1.02.1:arm64                            2:1.02.167-1ubuntu1                   arm64        Linux Kernel Device Mapper userspace library
ii  libi2c0:arm64                                       4.1-2build2                           arm64        userspace I2C programming library
ii  libibverbs1:arm64                                   28.0-1ubuntu1                         arm64        Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii  libnftnl11:arm64                                    1.1.5-1                               arm64        Netfilter nftables userspace API library
ii  libpopt-dev:arm64                                   1.16-14                               arm64        lib for parsing cmdline parameters - development files
ii  libpopt0:arm64                                      1.16-14                               arm64        lib for parsing cmdline parameters
ii  liburcu-dev:arm64                                   0.12.2-1~ubuntu20.04.2                arm64        userspace RCU (read-copy-update) library - development files
ii  liburcu6:arm64                                      0.12.2-1~ubuntu20.04.2                arm64        userspace RCU (read-copy-update) library
ii  libusb-1.0-0:arm64                                  2:1.0.23-2build1                      arm64        userspace USB programming library
ii  libusb-1.0-0-dev:arm64                              2:1.0.23-2build1                      arm64        userspace USB programming library development files
ii  libuuid1:arm64                                      2.34-0.1ubuntu9.3                     arm64        Universally Unique ID library
ii  libxml2:arm64                                       2.9.10+dfsg-5ubuntu0.20.04.1          arm64        GNOME XML library
ii  libxml2-dev:arm64                                   2.9.10+dfsg-5ubuntu0.20.04.1          arm64        Development files for the GNOME XML library
ii  libxml2-utils                                       2.9.10+dfsg-5ubuntu0.20.04.3          arm64        XML utilities
ii  network-manager                                     1.22.10-1ubuntu2.3                    arm64        network management framework (daemon and userspace tools)
ii  nvidia-l4t-optee                                    34.1.0-20220406120854                 arm64        OP-TEE userspace daemons, test programs and libraries
ii  python3-lxml:arm64                                  4.5.0-1ubuntu0.5                      arm64        pythonic binding for the libxml2 and libxslt libraries

sudo ln -snf /usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10 /lib/modules/5.10.65-tegra/build

# orin-d@orind-d:~/tmp/lttng-modules-2.13.5$ make
/home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.c:20:3: error: #error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0" 
   20 | # error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0" 
      |   ^~~~~
make[2]: *** [scripts/Makefile.build:281: /home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.o] Error 1
make[1]: *** [Makefile:1852: /home/orin-d/tmp/lttng-modules-2.13.5/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10'
make: *** [Makefile:31: modules] Error 2

Files

1667362822097.png (92.7 KB) 1667362822097.png liuhonggang liu, 11/02/2022 12:20 AM
Actions #1

Updated by Mathieu Desnoyers 5 months ago

You will need to rebuild your kernel with CONFIG_KPROBES=y, as stated in the error message. This is required from kernel 5.7.0 onwards.

Actions #2

Updated by liuhonggang liu 3 months ago

Hello.
I rebuild my kernel with CONFIG_KPROBES=y.
  1. uname -a
    Linux orind-d 5.10.65-tegra #2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux
  2. cat /etc/os-release
    NAME="Ubuntu"
    VERSION="20.04.4 LTS (Focal Fossa)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 20.04.4 LTS"
    VERSION_ID="20.04"
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    VERSION_CODENAME=focal
    UBUNTU_CODENAME=focal
  1. zcat /proc/config.gz | grep -i kprobes
    CONFIG_KPROBES=y
    CONFIG_HAVE_KPROBES=y
I reinstalled lttng-modules-dkms and still got the error message.
  1. error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
    How can I solve it please?
    The platform I am using is shown below.
Actions #3

Updated by liuhonggang liu 3 months ago

Hello.
I rebuild my kernel with CONFIG_KPROBES=y.
  1. uname -a
    Linux orind-d 5.10.65-tegra #2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux
  2. cat /etc/os-release
    NAME="Ubuntu"
    VERSION="20.04.4 LTS (Focal Fossa)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 20.04.4 LTS"
    VERSION_ID="20.04"
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    VERSION_CODENAME=focal
    UBUNTU_CODENAME=focal
  1. zcat /proc/config.gz | grep -i kprobes
    CONFIG_KPROBES=y
    CONFIG_HAVE_KPROBES=y
I reinstalled lttng-modules-dkms and still got the error message.
  1. error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
    How can I solve it please?
    The platform I am using is shown below.
Actions #4

Updated by liuhonggang liu 3 months ago

Hello.
I rebuild my kernel with CONFIG_KPROBES=y.
  1. uname -a
    Linux orind-d 5.10.65-tegra #2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux
  2. cat /etc/os-release
    NAME="Ubuntu"
    VERSION="20.04.4 LTS (Focal Fossa)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 20.04.4 LTS"
    VERSION_ID="20.04"
    HOME_URL="https://www.ubuntu.com/"
    SUPPORT_URL="https://help.ubuntu.com/"
    BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
    PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
    VERSION_CODENAME=focal
    UBUNTU_CODENAME=focal
  1. zcat /proc/config.gz | grep -i kprobes
    CONFIG_KPROBES=y
    CONFIG_HAVE_KPROBES=y
I reinstalled lttng-modules-dkms and still got the error message.
  1. error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
    How can I solve it please?
    The platform I am using is shown below.
Actions #5

Updated by Michael Jeanson 3 months ago

The output from '/proc/config.gz' confirms that the running kernel does indeed have CONFIG_KPROBE, the lttng-modules-dkms package uses the kernel headers and config located in '/lib/modules/$(uname -r)/build/', those have to match the running kernel.

Actions #6

Updated by liuhonggang liu 3 months ago

I rebuild my kernel with CONFIG_KPROBES=y.

uname -a

Linux inspur 5.10.104 #1 SMP PREEMPT Mon Oct 24 19:42:41 CST 2022 aarch64 aarch64 aarch64 GNU/Linux
sudo apt-get install lttng-modules-dkm

the log is as follows.
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  babeltrace liblttng-ctl0 liblttng-ust-ctl4 liburcu6
Suggested packages:
  lttng-modules-dkms
The following NEW packages will be installed:
  babeltrace liblttng-ctl0 liblttng-ust-ctl4 liburcu6 lttng-tools
0 upgraded, 5 newly installed, 0 to remove and 219 not upgraded.
Need to get 0 B/1049 kB of archives.
After this operation, 4376 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package liburcu6:arm64.
(Reading database ... 148021 files and directories currently installed.)
Preparing to unpack .../liburcu6_0.12.2-1~ubuntu20.04.2_arm64.deb ...
Unpacking liburcu6:arm64 (0.12.2-1~ubuntu20.04.2) ...
Selecting previously unselected package liblttng-ctl0:arm64.
Preparing to unpack .../liblttng-ctl0_2.12.4-1~ubuntu20.04.1_arm64.deb ...
Unpacking liblttng-ctl0:arm64 (2.12.4-1~ubuntu20.04.1) ...
Selecting previously unselected package liblttng-ust-ctl4:arm64.
Preparing to unpack .../liblttng-ust-ctl4_2.12.2-1~ubuntu20.04.1_arm64.deb ...
Unpacking liblttng-ust-ctl4:arm64 (2.12.2-1~ubuntu20.04.1) ...
Selecting previously unselected package lttng-tools.
Preparing to unpack .../lttng-tools_2.12.4-1~ubuntu20.04.1_arm64.deb ...
Unpacking lttng-tools (2.12.4-1~ubuntu20.04.1) ...
Selecting previously unselected package babeltrace.
Preparing to unpack .../babeltrace_1.5.8-1build1_arm64.deb ...
Unpacking babeltrace (1.5.8-1build1) ...
Setting up liblttng-ust-ctl4:arm64 (2.12.2-1~ubuntu20.04.1) ...
Setting up babeltrace (1.5.8-1build1) ...
Setting up liburcu6:arm64 (0.12.2-1~ubuntu20.04.2) ...
Setting up liblttng-ctl0:arm64 (2.12.4-1~ubuntu20.04.1) ...
Processing triggers for sgml-base (1.29.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.17) ...
Processing triggers for man-db (2.9.1-1) ...
Setting up lttng-tools (2.12.4-1~ubuntu20.04.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/lttng-sessiond.service → /lib/systemd/system/lttng-sessiond.service.
Processing triggers for systemd (245.4-4ubuntu3.17) ...
orin-c@inspur:~$ sudo apt-get install lttng-modules-dkms
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  dkms linux-headers-5.4.0-131 linux-headers-5.4.0-131-generic linux-headers-generic
Suggested packages:
  menu
The following NEW packages will be installed:
  dkms linux-headers-5.4.0-131 linux-headers-5.4.0-131-generic linux-headers-generic lttng-modules-dkms
0 upgraded, 5 newly installed, 0 to remove and 219 not upgraded.
Need to get 0 B/12.3 MB of archives.
After this operation, 85.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package dkms.
(Reading database ... 148110 files and directories currently installed.)
Preparing to unpack .../dkms_2.8.1-5ubuntu2_all.deb ...
Unpacking dkms (2.8.1-5ubuntu2) ...
Selecting previously unselected package linux-headers-5.4.0-131.
Preparing to unpack .../linux-headers-5.4.0-131_5.4.0-131.147_all.deb ...
Unpacking linux-headers-5.4.0-131 (5.4.0-131.147) ...
Selecting previously unselected package linux-headers-5.4.0-131-generic.
Preparing to unpack .../linux-headers-5.4.0-131-generic_5.4.0-131.147_arm64.deb ...
Unpacking linux-headers-5.4.0-131-generic (5.4.0-131.147) ...
Selecting previously unselected package linux-headers-generic.
Preparing to unpack .../linux-headers-generic_5.4.0.131.131_arm64.deb ...
Unpacking linux-headers-generic (5.4.0.131.131) ...
Selecting previously unselected package lttng-modules-dkms.
Preparing to unpack .../lttng-modules-dkms_2.12.6-1~ubuntu20.04.1_all.deb ...
Unpacking lttng-modules-dkms (2.12.6-1~ubuntu20.04.1) ...
Setting up linux-headers-5.4.0-131 (5.4.0-131.147) ...
Setting up dkms (2.8.1-5ubuntu2) ...
Setting up lttng-modules-dkms (2.12.6-1~ubuntu20.04.1) ...
Loading new lttng-modules-2.12.6 DKMS files...
It is likely that 5.10.104 belongs to a chroot's host
Building for 5.10.104, 5.10.104-tegra and 5.4.0-131-generic
Building initial module for 5.10.104
ERROR (dkms apport): kernel package linux-headers-5.10.104 is not supported
Error! Bad return status for module build on kernel: 5.10.104 (aarch64)
Consult /var/lib/dkms/lttng-modules/2.12.6/build/make.log for more information.
dpkg: error processing package lttng-modules-dkms (--configure):
 installed lttng-modules-dkms package post-installation script subprocess returned error exit status 10
Setting up linux-headers-5.4.0-131-generic (5.4.0-131.147) ...
/etc/kernel/header_postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.4.0-131-generic

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make -j4 KERNELRELEASE=5.4.0-131-generic -C /lib/modules/5.4.0-131-generic/build M=/var/lib/dkms/lttng-modules/2.12.6/build modules CONFIG_LTTNG=m CONFIG_LTTNG_CLOCK_PLUGIN_TEST=m.......(bad exit status: 2)
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/lttng-modules-dkms.0.crash'
Error! Bad return status for module build on kernel: 5.4.0-131-generic (aarch64)
Consult /var/lib/dkms/lttng-modules/2.12.6/build/make.log for more information.
   ...done.
Setting up linux-headers-generic (5.4.0.131.131) ...
Processing triggers for man-db (2.9.1-1) ...
Errors were encountered while processing:
 lttng-modules-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

/var/lib/dkms/lttng-modules/2.12.6/build/make.log. The content is as follows.


DKMS make.log for lttng-modules-2.12.6 for kernel 5.4.0-131-generic (aarch64)
Wed Nov  2 04:04:29 UTC 2022
make: Entering directory '/usr/src/linux-headers-5.4.0-131-generic'
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-client-discard.o
/var/lib/dkms/lttng-modules/2.12.6/build/probes/Kbuild:166: Files ./fs/btrfs/*.h not found. Probe "btrfs" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.12.6/build/probes/Kbuild:177: Files ./fs/ext4/*.h not found. Probe "ext4" is disabled. Use full kernel source tree to enable it.
/var/lib/dkms/lttng-modules/2.12.6/build/probes/Kbuild:208: File ./drivers/base/regmap/trace.h not found. Probe "regmap" is disabled. Need Linux 4.1+ kernel source tree to enable it.
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_backend.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/tests/probes/lttng-test.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-sched.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/tests/clock-plugin/lttng-clock-plugin-test.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_frontend.o
  LD [M]  /var/lib/dkms/lttng-modules/2.12.6/build/tests/lttng-test.o
  LD [M]  /var/lib/dkms/lttng-modules/2.12.6/build/tests/lttng-clock-plugin-test.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_iterator.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_vfs.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-irq.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-timer.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-kmem.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-client-overwrite.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_splice.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/ringbuffer/ring_buffer_mmap.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/prio_heap/lttng_prio_heap.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-module.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-power.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-metadata-client.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/../wrapper/splice.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-statedump.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-9p.o
  LD [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lib/lttng-lib-ring-buffer.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-client-mmap-discard.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-i2c.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-kvm.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-signal.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-block.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-client-mmap-overwrite.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-ring-buffer-metadata-mmap-client.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-clock.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-napi.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-skb.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-net.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-events.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-abi.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-string-utils.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-sock.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-probes.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-udp.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-pid.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-asoc.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-procname.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-gpio.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-jbd2.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-regulator.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-scsi.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-vmscan.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-prio.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-nice.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-vpid.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-compaction.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-tid.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-vtid.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-printk.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-ppid.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-random.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-rcu.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-vppid.o
/var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-random.c:18:10: fatal error: trace/events/random.h: No such file or directory
   18 | #include <trace/events/random.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:270: /var/lib/dkms/lttng-modules/2.12.6/build/probes/lttng-probe-random.o] Error 1
make[2]: *** Waiting for unfinished jobs....
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-cpu-id.o
  CC [M]  /var/lib/dkms/lttng-modules/2.12.6/build/lttng-context-uid.o
make[1]: *** [scripts/Makefile.build:519: /var/lib/dkms/lttng-modules/2.12.6/build/probes] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:1762: /var/lib/dkms/lttng-modules/2.12.6/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.4.0-131-generic'

According to the above log, it looks like the cause of the problem may be that the kernel headers for this kernel does not seem to be installed. And the header files in the following directory are being used /usr/src/linux-headers-5.4.0-131-generic.

From the image below, there is no linux-headers-5.10.104 in /usr/src.
Is this the cause of the problem? Or do you know the root cause of the above problem

Actions #7

Updated by Michael Jeanson 3 months ago

There is 2 different issues here, the lttng-modules-dkms will by default only build modules for packaged kernels. In this case the build failure you see in the dkms.log is for the '5.4.0-131-generic' kernel shipped by ubuntu.

The second issue is that you don't seem to have the headers for your '5.10.104' kernel, there is a dangling symlink for the 'source' directory but the headers we are interrested in should be in '/lib/modules/5.10.104/build'. This directory exists but I don't see what its content is.

If you can confirm that the kernel headers are indeed installed, the next step would be to try building lttng-modules from the latest 2.12.x source tarball.

Actions #8

Updated by liuhonggang liu 3 months ago

Install lttng-modules with the correct version of the header files.

orin-c@inspur:/lib/modules/5.10.104-tegra$ ll
total 2012
drwxr-xr-x 4 root root   4096 Oct 24 11:32 ./
drwxr-xr-x 3 root root   4096 Nov  3 02:35 ../
lrwxrwxrwx 1 root root     69 Aug 11 03:37 build -> /usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-5.10/
drwxr-xr-x 3 root root   4096 Oct 24 11:31 extra/
drwxr-xr-x 9 root root   4096 Oct 24 11:31 kernel/
-rw-r--r-- 1 root root 506756 Oct 24 11:32 modules.alias
-rw-r--r-- 1 root root 497516 Oct 24 11:32 modules.alias.bin
-rw-r--r-- 1 root root  27603 Aug 11 03:37 modules.builtin
-rw-r--r-- 1 root root  50099 Oct 24 11:32 modules.builtin.alias.bin
-rw-r--r-- 1 root root  30111 Oct 24 11:32 modules.builtin.bin
-rw-r--r-- 1 root root 154311 Aug 11 03:37 modules.builtin.modinfo
-rw-r--r-- 1 root root  88190 Oct 24 11:32 modules.dep
-rw-r--r-- 1 root root 138538 Oct 24 11:32 modules.dep.bin
-rw-r--r-- 1 root root    187 Oct 24 11:32 modules.devname
-rw-r--r-- 1 root root  45829 Aug 11 03:37 modules.order
-rw-r--r-- 1 root root    786 Oct 24 11:32 modules.softdep
-rw-r--r-- 1 root root 210396 Oct 24 11:32 modules.symbols
-rw-r--r-- 1 root root 261524 Oct 24 11:32 modules.symbols.bin

Install lttng-modules from source code.

orin-c@inspur:~/lhg/lttng-modules-2.12.10$ make
make -C /lib/modules/5.10.104-tegra/build M=/home/orin-c/lhg/lttng-modules-2.12.10 \
        CONFIG_LTTNG=m CONFIG_LTTNG_CLOCK_PLUGIN_TEST=m \
        KCPPFLAGS='' \
        modules
make[1]: Entering directory '/usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel
/home/orin-c/lhg/lttng-modules-2.12.10/probes/Kbuild:166: Files ./fs/btrfs/*.h not found. Pro
/home/orin-c/lhg/lttng-modules-2.12.10/probes/Kbuild:179: Files ./fs/ext4/*.h not found. Prob
/home/orin-c/lhg/lttng-modules-2.12.10/probes/Kbuild:208: File ./drivers/base/regmap/trace.h enable it.
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-sched.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-irq.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-timer.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-kmem.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-module.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-power.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-statedump.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-9p.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-i2c.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-kvm.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-signal.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-block.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-napi.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-skb.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-net.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-sock.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-udp.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-asoc.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-gpio.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-jbd2.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-regulator.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-scsi.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-vmscan.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-compaction.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-printk.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-random.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-rcu.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-sunrpc.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-v4l2.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-workqueue.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-writeback.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_backend.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_frontend.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_iterator.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_vfs.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_splice.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/ringbuffer/ring_buffer_mmap.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/prio_heap/lttng_prio_heap.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/../wrapper/splice.o
  LD [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lib/lttng-lib-ring-buffer.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/tests/probes/lttng-test.o
  LD [M]  /home/orin-c/lhg/lttng-modules-2.12.10/tests/lttng-test.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/tests/clock-plugin/lttng-clock-plugin-test.o
  LD [M]  /home/orin-c/lhg/lttng-modules-2.12.10/tests/lttng-clock-plugin-test.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-client-discard.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-client-overwrite.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-metadata-client.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-client-mmap-discard.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-client-mmap-overwrite.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-ring-buffer-metadata-mmap-client.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-clock.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-events.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-abi.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-string-utils.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-probes.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-pid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-procname.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-prio.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-nice.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vpid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-tid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vtid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-ppid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vppid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-cpu-id.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-uid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-euid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-suid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-gid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-egid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-sgid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vuid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-veuid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vsuid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vgid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vegid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-vsgid.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-interruptible.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-need-reschedule.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-callstack.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-calibrate.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-hostname.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-tracker-id.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-filter.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-filter-interpreter.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-filter-specialize.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-filter-validator.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/probes/lttng-probe-user.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-tp-mempool.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-syscalls.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-perf-counters.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-preemptible.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-tracepoint.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-cgroup-ns.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-ipc-ns.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-net-ns.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-pid-ns.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-user-ns.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-context-uts-ns.o
  LD [M]  /home/orin-c/lhg/lttng-modules-2.12.10/lttng-tracer.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/wrapper/page_alloc.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/wrapper/random.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/wrapper/trace-clock.o
  CC [M]  /home/orin-c/lhg/lttng-modules-2.12.10/wrapper/kallsyms.o
/home/orin-c/lhg/lttng-modules-2.12.10/wrapper/kallsyms.c:20:3: error: #error "LTTng-modules
   20 | # error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0" 
      |   ^~~~~
make[2]: *** [scripts/Makefile.build:281: /home/orin-c/lhg/lttng-modules-2.12.10/wrapper/kall
make[1]: *** [Makefile:1854: /home/orin-c/lhg/lttng-modules-2.12.10] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.104-tegra-ubuntu20.04_aarch64/kernel-
make: *** [Makefile:162: modules] Error 2
zcat /proc/config.gz | grep -i kprobes
CONFIG_KPROBES=y
CONFIG_HAVE_KPROBES=y

The error still appears in the log, how can I go deeper to find the cause of the error?

Actions #9

Updated by Michael Jeanson 3 months ago

The kernel headers in '/lib/modules/5.10.104-tegra/build' need to be configured with the same '.config' file as the running kernel.

Actions #10

Updated by liuhonggang liu 3 months ago

Hello.
I’m running the last kernel available from JetPack5.0.1 Linux for Tegra r35.1, which is a LTS ubuntu 20.04 LTS. Install lttng-modules-2.12.10 from source on this device. I confirm that the kernel headers are indeed installed and CONFIG_KPROBES = y on my system.
1. Does lttng-modules support installation on this system?
2. If installation is supported, can you reproduce the following error

    # error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0" 

3. If you do not have the conditions to reproduce, I can support you in every step of analysis and give you feedback on the results of every step.

Actions #11

Updated by liuhonggang liu 3 months ago

Hello.
I’m running the last kernel available from JetPack5.0.1 Linux for Tegra r35.1, which is a LTS ubuntu 20.04 LTS. Install lttng-modules-2.12.10 from source on this device. I confirm that the kernel headers are indeed installed and CONFIG_KPROBES = y on my system.
1. Does lttng-modules support installation on this system?
2. If installation is supported, can you reproduce the following error

error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"

3. If you do not have the conditions to reproduce, I can support you in every step of analysis and give you feedback on the results of every step.

Actions #12

Updated by Michael Jeanson 3 months ago

I downloaded the 5.10.104 kernel sources from Nvidia on this page:

https://developer.nvidia.com/embedded/jetson-linux-r351

Then used the defconfig to which I added CONFIG_KPROBES and had no issues building lttng-modules '2.12' and '2.13'.

I'm still convinced your issue is that the headers / kernel sources which you use to build lttng-modules don't match you running kernel.

We do offer professional services at "https://efficios.com", including packaging and deployment, it might be an option for your organisation.

Actions #13

Updated by liuhonggang liu 3 months ago

We use JETSON AGX ORIN, the .config file shows no problem when flashing, and the kernel compilation and flashing are no problem.
1. Did you modify the kernel before compiling? If so, how did you compile lttng-modules-dkms into the kernel?
2. Have you tested installing lttng-modules-dkms on orin devkit? Is it convenient to show your environment here?
3. How much does it cost via "https://efficios.com"?
4. Would you like to introduce how you used https://developer.nvidia.com/embedded/jetson-linux-r351 to install lttng-module-dkms?
Your answer will give me a great help, looking forward to your answer.
Thank you.

Actions #14

Updated by liuhonggang liu 3 months ago

We use JETSON AGX ORIN, the .config file shows no problem when flashing, and the kernel compilation and flashing are no problem.
1. Did you modify the kernel before compiling? If so, how did you compile lttng-modules-dkms into the kernel?
2. Have you tested installing lttng-modules-dkms on orin devkit? Is it convenient to show your environment here?
3. How much does it cost via "https://efficios.com"?
4. Would you like to introduce how you used https://developer.nvidia.com/embedded/jetson-linux-r351 to install lttng-module-dkms?
Your answer will give me a great help, looking forward to your answer.
Thank you.

Actions #15

Updated by Michael Jeanson 3 months ago

liuhonggang liu wrote in #note-14:

We use JETSON AGX ORIN, the .config file shows no problem when flashing, and the kernel compilation and flashing are no problem.
1. Did you modify the kernel before compiling? If so, how did you compile lttng-modules-dkms into the kernel?

No, I built a kernel from the Nvidia source tarball and then built lttng-modules also from a source tarball.

2. Have you tested installing lttng-modules-dkms on orin devkit? Is it convenient to show your environment here?

No

3. How much does it cost via "https://efficios.com"?

It really depends on the scope of the support you want, you can contact "".

4. Would you like to introduce how you used https://developer.nvidia.com/embedded/jetson-linux-r351 to install lttng-module-dkms?
Your answer will give me a great help, looking forward to your answer.
Thank you.

Actions #16

Updated by liuhonggang liu 2 months ago

thank you.
Can lttng-modules-dkms be deployed and used on nvidia's tegra platform?

Actions

Also available in: Atom PDF