LTTng » LTTng-tools

This is with lttng-tools 2.13.14 on a 6.12.19 kernel, lttng-modules 2.13.17 built with Yocto Project master (498666694ff22d3d851d429f760fb5504a8995af).

Hi Kienan,

what would I need to set to get the verbose log? The kernel log says:

[ 56.079831] lttng_kretprobes: loading out-of-tree module taints kernel.
[ 56.127762] LTTng: Loaded modules v2.13.17 (Nordicit\xc3\xa9)
[ 56.497677] LTTng: block device enumeration is not supported by kernel
[ 57.516029] LTTng: block device enumeration is not supported by kernel
[ 58.459236] LTTng: block device enumeration is not supported by kernel
[ 59.291679] LTTng: block device enumeration is not supported by kernel
[ 60.131897] LTTng: block device enumeration is not supported by kernel
[ 60.146770] select_poll_epo⁶⁷⁹: segfault at 208178 ip 0049b614 sp bfff8164 error 4 in select_poll_epoll[2614,49b000+5000] likely on CPU 3 (core 3, socket 0)
[ 60.146954] Code: 08 80 78 04 00 89 93 34 01 00 00 0f 84 bf 02 00 00 6b f6 0c 83 ec 0c ff b5 50 fe ff ff ff 94 33 d0 fb ff ff 83 c4 10 83 ec 0c <ff> b5 50 fe f0
[ 60.969805] LTTng: block device enumeration is not supported by kernel
[ 61.796216] LTTng: block device enumeration is not supported by kernel
[ 62.635524] LTTng: block device enumeration is not supported by kernel
[ 63.455461] LTTng: block device enumeration is not supported by kernel
[ 64.313483] LTTng: block device enumeration is not supported by kernel
[ 65.586331] LTTng: block device enumeration is not supported by kernel
[ 70.923085] LTTng: Unloaded modules v2.13.17 (Nordicit\xc3\xa9)

I was able to extract a standalone way to reproduce the errors with the steps below. This downloads one of the failing images from our CI system and then reproduces it under qemu:

wget https://valkyrie.yocto.io/pub/shared-failure-data/qemux86-ptest/core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4.bz2
wget https://valkyrie.yocto.io/pub/shared-failure-data/qemux86-ptest/bzImage-qemux86.bin
bunzip2 core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4.bz2

Run the image under kvm qemu. You may need to tweak the options for your version of qemu, I tried the one from our build and one from my own Ubuntu system. The options don't seem to change the test result.

qemu-system-i386 --accel kvm -snapshot -nographic -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -drive file=core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4,if=virtio,format=raw -cpu IvyBridge -machine q35,i8042=off -smp 4 -m 512 -kernel bzImage-qemux86.bin -append 'root=/dev/vda rw ip=192.168.7.2::192.168.7.1:255.255.255.0::eth0:off:8.8.8.8'

Login as root, no password

cd /usr/lib/lttng-tools/ptest
vi run-ptest

Patch our runner script just to run the failing test by changing "make -k -s $makeargs check 2>error.log | sed -e 's#/tmp/tmp\...........#/tmp/tmp.XXXXXXXXXX#g'" 
  to "make -k -s $makeargs check -C tests/regression TESTS='kernel/test_select_poll_epoll'"

./run-ptest

and you should see the failure.

For the open files limit, I did set "ulimit -n 40960" before running the tests but it doesn't seem to change anything. Is that the correct limit to change?

Hi Richard,

I've been trying a few things in the image you sent (thank you again!).

I modified the tests/utils/utils.sh to output the babeltrace to stderr in 'validate_trace()', and modified the test script to enable all kernel events rather than just those are being verified. I suspect this issue is a failure in the tests to handle glibc versions which are using pselect6_time64, ppoll_time64. The select failure puzzles me a little bit still, but my hypothesis is glibc is using a different syscall under the hood.

From a single run of the working test case (all other tests were skipped), I logged the following unknown syscall numbers (c.f. https://gpages.juszkiewicz.com.pl/syscalls-table/syscalls.html):

# grep syscall_entry_unknown log | grep -Eo ' id = [0-9]{1,4}' | cut -d ' ' -f4 | sort | uniq -c | sort -n -k1
      1 359
      1 362
      1 364
      1 366
      1 369
      1 373
      1 414 # ppoll_time64
      2 119
      3 243
      3 355
      3 386
      3 413 # pselect6_time64
      4 370
      4 372
     22 383
     38 272
     98 403

Now you mention time64, I think I can explain this, or perhaps at least hint at where the problem may lie. We build our 32 bit platforms with -D_TIME_BITS=64 -D_FILE_OFFSET_BITS=64, i.e. we try and force the use of 64 bit time everywhere. I suspect that may be breaking the tests?

Hi Richard,

I'm wondering if the patches for modules are sufficient to restore the expected mappings. These are not present in modules 2.13.x, but are in master:

https://github.com/lttng/lttng-modules/commit/86040c9e2ff8a79633db34220e32bdda536b00e6
https://github.com/lttng/lttng-modules/commit/92dd6990b635d31390e073507ec51c19128a562f

I will test with a backport of those this afternoon.

Great to know re: gblic defines

thanks,
kienan

Hi Richard, with regards to -D_TIME_BITS=64, is that a glibc define?

I was looking at the configuration for the linux kernel for yocto master on i386 and I see CONFIG_COMPAT_32BIT=y, so the pselect6 syscall goes to the 32bit compat version.

Kienan Stewart wrote in #note-8:

Hi Richard, with regards to -D_TIME_BITS=64, is that a glibc define?

I was looking at the configuration for the linux kernel for yocto master on i386 and I see CONFIG_COMPAT_32BIT=y, so the pselect6 syscall goes to the 32bit compat version.

We pass it in to the compiler globally and it changes te glibc headers:

https://web.git.yoctoproject.org/poky/tree/meta/conf/distro/include/time64.inc

Should we be using different kernel config options? I'd need to check into exactly what that one is doing...

Thanks! I was meaning to test the changes but couldn't quite work out how to get them from the system and apply them easily.

I did update our build to LTTng-tools 2.13.15 and LTTng-modules 2.13.18 and this does improve the test results but the build did still show some failures:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/44/builds/4/steps/12/logs/stdio

{'lttng-tools': ['kernel/test_select_poll_epoll_35_-_Validate_trace_for_event_pselect6',
'kernel/test_select_poll_epoll_59_-_Validate_trace_for_event_pselect6',
'kernel/test_select_poll_epoll_67_-_Validate_trace_for_event_ppoll',
'kernel/test_select_poll_epoll']}

Looking at the more detailed log:

https://valkyrie.yocto.io/pub/non-release/20250421-113/testresults/qemux86-ptest/lttng-tools.log

which shows:

PASS: kernel/test_select_poll_epoll 1 - Start session daemon
PASS: kernel/test_select_poll_epoll 2 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_working_cases_trace_path.AwVk2b
PASS: kernel/test_select_poll_epoll 3 - Enable kernel syscall select,pselect6,poll,ppoll,epoll_ctl,epoll_wait,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 4 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 5 - Start tracing for session
PASS: kernel/test_select_poll_epoll 6 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 7 - Validate trace for event select, 6 events
PASS: kernel/test_select_poll_epoll 8 - Validate trace for event pselect6, 6 events
PASS: kernel/test_select_poll_epoll 9 - Validate trace for event poll, 18 events
PASS: kernel/test_select_poll_epoll 10 - Validate trace for event ppoll, 2 events
PASS: kernel/test_select_poll_epoll 11 - Validate trace for event epoll_ctl, 4 events
PASS: kernel/test_select_poll_epoll 12 - Validate trace for event epoll_wait, 6 events
PASS: kernel/test_select_poll_epoll 13 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 14 - Validation skipped
PASS: kernel/test_select_poll_epoll 15 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 16 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_timeout_cases_trace_path.az3MF1
PASS: kernel/test_select_poll_epoll 17 - Enable kernel syscall select,pselect6,poll,ppoll,epoll_ctl,epoll_wait,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 18 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 19 - Start tracing for session
PASS: kernel/test_select_poll_epoll 20 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 21 - Validate trace for event select, 6 events
PASS: kernel/test_select_poll_epoll 22 - Validate trace for event pselect6, 6 events
PASS: kernel/test_select_poll_epoll 23 - Validate trace for event poll, 20 events
PASS: kernel/test_select_poll_epoll 24 - Validate trace for event ppoll, 2 events
PASS: kernel/test_select_poll_epoll 25 - Validate trace for event epoll_ctl, 4 events
PASS: kernel/test_select_poll_epoll 26 - Validate trace for event epoll_wait, 6 events
PASS: kernel/test_select_poll_epoll 27 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 28 - Validation skipped
PASS: kernel/test_select_poll_epoll 29 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 30 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_pselect_invalid_fd_trace_path.IQz8F0
PASS: kernel/test_select_poll_epoll 31 - Enable kernel syscall pselect6 for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 32 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 33 - Start tracing for session
PASS: kernel/test_select_poll_epoll 34 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 35 - Validate trace for event pselect6
PASS: kernel/test_select_poll_epoll 36 - Validation skipped
PASS: kernel/test_select_poll_epoll 37 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 38 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_big_ppoll_trace_path.KoPi86
PASS: kernel/test_select_poll_epoll 39 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 40 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 41 - Start tracing for session
PASS: kernel/test_select_poll_epoll 42 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 43 - Validate trace for event ppoll, 2 events
PASS: kernel/test_select_poll_epoll 44 - Validation skipped
PASS: kernel/test_select_poll_epoll 45 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 46 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_overflow_trace_path.eFcOH6
PASS: kernel/test_select_poll_epoll 47 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 48 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 49 - Start tracing for session
PASS: kernel/test_select_poll_epoll 50 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 51 - Validate trace for event ppoll, 4 events
PASS: kernel/test_select_poll_epoll 52 - Validation skipped
PASS: kernel/test_select_poll_epoll 53 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 54 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_pselect_invalid_ptr_trace_path.rI6h29
PASS: kernel/test_select_poll_epoll 55 - Enable kernel syscall pselect6 for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 56 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 57 - Start tracing for session
PASS: kernel/test_select_poll_epoll 58 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 59 - Validate trace for event pselect6
PASS: kernel/test_select_poll_epoll 60 - Validation skipped
PASS: kernel/test_select_poll_epoll 61 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 62 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_ulong_max_trace_path.R48aop
PASS: kernel/test_select_poll_epoll 63 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 64 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 65 - Start tracing for session
PASS: kernel/test_select_poll_epoll 66 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 67 - Validate trace for event ppoll
PASS: kernel/test_select_poll_epoll 68 - Validation skipped
PASS: kernel/test_select_poll_epoll 69 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 70 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_invalid_ptr_trace_path.xixTkw
PASS: kernel/test_select_poll_epoll 71 - Enable kernel syscall epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 72 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 73 - Start tracing for session
PASS: kernel/test_select_poll_epoll 74 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 75 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 76 - Validation skipped
PASS: kernel/test_select_poll_epoll 77 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 78 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_int_max_trace_path.JjarFZ
PASS: kernel/test_select_poll_epoll 79 - Enable kernel syscall epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 80 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 81 - Start tracing for session
PASS: kernel/test_select_poll_epoll 82 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 83 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 84 - Validation skipped
PASS: kernel/test_select_poll_epoll 85 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 86 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_concurrent_trace_path.xpJ9AQ
PASS: kernel/test_select_poll_epoll 87 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 88 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 89 - Start tracing for session
PASS: kernel/test_select_poll_epoll 90 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 91 - Validate trace for event ppoll, 2189 events
PASS: kernel/test_select_poll_epoll 92 - Validation skipped
PASS: kernel/test_select_poll_epoll 93 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 94 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_concurrent_trace_path.FcLLAp
PASS: kernel/test_select_poll_epoll 95 - Enable kernel syscall epoll_ctl,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 96 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 97 - Start tracing for session
PASS: kernel/test_select_poll_epoll 98 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 99 - Validate trace for event epoll_ctl, 409400 events
PASS: kernel/test_select_poll_epoll 100 - Validation skipped
PASS: kernel/test_select_poll_epoll 101 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 102 - Wait after kill session daemon

was there anything else I needed to configure or set to make this work?