Bug #1425
openlttng tools test failure in regression/kernel/test_select_poll_epoll on 32 bit x86
0%
Description
On 32 bit x86 images, running the lttng-tools tests we see some failures:
PASS: kernel/test_select_poll_epoll 1 - Start session daemon
PASS: kernel/test_select_poll_epoll 2 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_working_cases_trace_path.ms9uft
PASS: kernel/test_select_poll_epoll 3 - Enable kernel syscall select,pselect6,poll,ppoll,epoll_ctl,epoll_wait,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 4 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 5 - Start tracing for session
PASS: kernel/test_select_poll_epoll 6 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 7 - Validate trace for event select
FAIL: kernel/test_select_poll_epoll 8 - Validate trace for event pselect6
PASS: kernel/test_select_poll_epoll 9 - Validate trace for event poll, 16 events
FAIL: kernel/test_select_poll_epoll 10 - Validate trace for event ppoll
PASS: kernel/test_select_poll_epoll 11 - Validate trace for event epoll_ctl, 4 events
PASS: kernel/test_select_poll_epoll 12 - Validate trace for event epoll_wait, 6 events
PASS: kernel/test_select_poll_epoll 13 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 14 - Validation skipped
PASS: kernel/test_select_poll_epoll 15 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 16 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_timeout_cases_trace_path.zFWbtZ
PASS: kernel/test_select_poll_epoll 17 - Enable kernel syscall select,pselect6,poll,ppoll,epoll_ctl,epoll_wait,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 18 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 19 - Start tracing for session
PASS: kernel/test_select_poll_epoll 20 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 21 - Validate trace for event select
FAIL: kernel/test_select_poll_epoll 22 - Validate trace for event pselect6
PASS: kernel/test_select_poll_epoll 23 - Validate trace for event poll, 16 events
FAIL: kernel/test_select_poll_epoll 24 - Validate trace for event ppoll
PASS: kernel/test_select_poll_epoll 25 - Validate trace for event epoll_ctl, 4 events
PASS: kernel/test_select_poll_epoll 26 - Validate trace for event epoll_wait, 6 events
PASS: kernel/test_select_poll_epoll 27 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 28 - Validation skipped
PASS: kernel/test_select_poll_epoll 29 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 30 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_pselect_invalid_fd_trace_path.FoUxnX
PASS: kernel/test_select_poll_epoll 31 - Enable kernel syscall pselect6 for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 32 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 33 - Start tracing for session
PASS: kernel/test_select_poll_epoll 34 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 35 - Validate trace for event pselect6, 2 events
PASS: kernel/test_select_poll_epoll 36 - Validation skipped
PASS: kernel/test_select_poll_epoll 37 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 38 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_big_ppoll_trace_path.BjArOd
PASS: kernel/test_select_poll_epoll 39 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 40 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 41 - Start tracing for session
PASS: kernel/test_select_poll_epoll 42 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 43 - Validate trace for event ppoll
PASS: kernel/test_select_poll_epoll 44 - Validation skipped
PASS: kernel/test_select_poll_epoll 45 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 46 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_overflow_trace_path.r1OnlC
PASS: kernel/test_select_poll_epoll 47 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 48 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 49 - Start tracing for session
PASS: kernel/test_select_poll_epoll 50 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 51 - Validate trace for event ppoll, 2 events
PASS: kernel/test_select_poll_epoll 52 - Validation skipped
PASS: kernel/test_select_poll_epoll 53 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 54 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_pselect_invalid_ptr_trace_path.na4y64
PASS: kernel/test_select_poll_epoll 55 - Enable kernel syscall pselect6 for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 56 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 57 - Start tracing for session
PASS: kernel/test_select_poll_epoll 58 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 59 - Validate trace for event pselect6, 2 events
PASS: kernel/test_select_poll_epoll 60 - Validation skipped
PASS: kernel/test_select_poll_epoll 61 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 62 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_ulong_max_trace_path.qqIdrR
PASS: kernel/test_select_poll_epoll 63 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 64 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 65 - Start tracing for session
PASS: kernel/test_select_poll_epoll 66 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 67 - Validate trace for event ppoll, 2 events
PASS: kernel/test_select_poll_epoll 68 - Validation skipped
PASS: kernel/test_select_poll_epoll 69 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 70 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_invalid_ptr_trace_path.xVodau
PASS: kernel/test_select_poll_epoll 71 - Enable kernel syscall epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 72 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 73 - Start tracing for session
PASS: kernel/test_select_poll_epoll 74 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 75 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 76 - Validation skipped
PASS: kernel/test_select_poll_epoll 77 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 78 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_int_max_trace_path.xKJfa6
PASS: kernel/test_select_poll_epoll 79 - Enable kernel syscall epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 80 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 81 - Start tracing for session
PASS: kernel/test_select_poll_epoll 82 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 83 - Validate trace for event epoll_pwait, 2 events
PASS: kernel/test_select_poll_epoll 84 - Validation skipped
PASS: kernel/test_select_poll_epoll 85 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 86 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_ppoll_concurrent_trace_path.wOIxYY
PASS: kernel/test_select_poll_epoll 87 - Enable kernel syscall ppoll for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 88 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 89 - Start tracing for session
PASS: kernel/test_select_poll_epoll 90 - Stop lttng tracing for session
FAIL: kernel/test_select_poll_epoll 91 - Validate trace for event ppoll
PASS: kernel/test_select_poll_epoll 92 - Validation skipped
PASS: kernel/test_select_poll_epoll 93 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 94 - Create session syscall_payload in -o /tmp/tmp.XXXXXXXXXXl_select_poll_epoll_test_epoll_pwait_concurrent_trace_path.bZvklK
PASS: kernel/test_select_poll_epoll 95 - Enable kernel syscall epoll_ctl,epoll_pwait for session syscall_payload on channel
PASS: kernel/test_select_poll_epoll 96 - Add context command for type: pid
PASS: kernel/test_select_poll_epoll 97 - Start tracing for session
PASS: kernel/test_select_poll_epoll 98 - Stop lttng tracing for session
PASS: kernel/test_select_poll_epoll 99 - Validate trace for event epoll_ctl, 409400 events
PASS: kernel/test_select_poll_epoll 100 - Validation skipped
PASS: kernel/test_select_poll_epoll 101 - Destroy session syscall_payload
PASS: kernel/test_select_poll_epoll 102 - Wait after kill session daemon
as seen in Yocto Project CI here:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/44/builds/1
{'lttng-tools': ['kernel/test_select_poll_epoll_7_-_Validate_trace_for_event_select',
'kernel/test_select_poll_epoll_8_-_Validate_trace_for_event_pselect6',
'kernel/test_select_poll_epoll_10_-_Validate_trace_for_event_ppoll',
'kernel/test_select_poll_epoll_21_-_Validate_trace_for_event_select',
'kernel/test_select_poll_epoll_22_-_Validate_trace_for_event_pselect6',
'kernel/test_select_poll_epoll_24_-_Validate_trace_for_event_ppoll',
'kernel/test_select_poll_epoll_43_-_Validate_trace_for_event_ppoll',
'kernel/test_select_poll_epoll_91_-_Validate_trace_for_event_ppoll',
'kernel/test_select_poll_epoll']}
I've attached the log from the test directory from the target image. These are the only failures we're seeing for 32 bit x86.
Files
Updated by Richard Purdie 5 days ago
This is with lttng-tools 2.13.14 on a 6.12.19 kernel, lttng-modules 2.13.17 built with Yocto Project master (498666694ff22d3d851d429f760fb5504a8995af).
Updated by Kienan Stewart 1 day ago
- Status changed from New to Feedback
- Assignee set to Richard Purdie
Hi Richard,
I haven't been able to reproduce the test failure in an qemu VM
lttng-tools: 2.13.14
lttng-modules: 2.13.17
urcu: master
babeltrace: tested both babeltrace 1.5 and babeltrace master
linux: 6.12.21 (voidlinux)
In your build log I notice that towards the end there's a lot of too many open files errors that are emitted. While positionally after the failing tests, it might be good to double check the limits on the builder.
Do you have the session-daemon verbose log and/or the kernel log available?
thanks,
kienan
Updated by Richard Purdie about 18 hours ago
Hi Kienan,
what would I need to set to get the verbose log? The kernel log says:
[ 56.079831] lttng_kretprobes: loading out-of-tree module taints kernel.
[ 56.127762] LTTng: Loaded modules v2.13.17 (Nordicit\xc3\xa9)
[ 56.497677] LTTng: block device enumeration is not supported by kernel
[ 57.516029] LTTng: block device enumeration is not supported by kernel
[ 58.459236] LTTng: block device enumeration is not supported by kernel
[ 59.291679] LTTng: block device enumeration is not supported by kernel
[ 60.131897] LTTng: block device enumeration is not supported by kernel
[ 60.146770] select_poll_epo679: segfault at 208178 ip 0049b614 sp bfff8164 error 4 in select_poll_epoll[2614,49b000+5000] likely on CPU 3 (core 3, socket 0)
[ 60.146954] Code: 08 80 78 04 00 89 93 34 01 00 00 0f 84 bf 02 00 00 6b f6 0c 83 ec 0c ff b5 50 fe ff ff ff 94 33 d0 fb ff ff 83 c4 10 83 ec 0c <ff> b5 50 fe f0
[ 60.969805] LTTng: block device enumeration is not supported by kernel
[ 61.796216] LTTng: block device enumeration is not supported by kernel
[ 62.635524] LTTng: block device enumeration is not supported by kernel
[ 63.455461] LTTng: block device enumeration is not supported by kernel
[ 64.313483] LTTng: block device enumeration is not supported by kernel
[ 65.586331] LTTng: block device enumeration is not supported by kernel
[ 70.923085] LTTng: Unloaded modules v2.13.17 (Nordicit\xc3\xa9)
I was able to extract a standalone way to reproduce the errors with the steps below. This downloads one of the failing images from our CI system and then reproduces it under qemu:
wget https://valkyrie.yocto.io/pub/shared-failure-data/qemux86-ptest/core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4.bz2
wget https://valkyrie.yocto.io/pub/shared-failure-data/qemux86-ptest/bzImage-qemux86.bin
bunzip2 core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4.bz2
Run the image under kvm qemu. You may need to tweak the options for your version of qemu, I tried the one from our build and one from my own Ubuntu system. The options don't seem to change the test result.
qemu-system-i386 --accel kvm -snapshot -nographic -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -drive file=core-image-ptest-lttng-tools-qemux86.rootfs-20250331222320.ext4,if=virtio,format=raw -cpu IvyBridge -machine q35,i8042=off -smp 4 -m 512 -kernel bzImage-qemux86.bin -append 'root=/dev/vda rw ip=192.168.7.2::192.168.7.1:255.255.255.0::eth0:off:8.8.8.8'
Login as root, no password
cd /usr/lib/lttng-tools/ptest
vi run-ptest
Patch our runner script just to run the failing test by changing "make -k -s $makeargs check 2>error.log | sed -e 's#/tmp/tmp\...........#/tmp/tmp.XXXXXXXXXX#g'"
to "make -k -s $makeargs check -C tests/regression TESTS='kernel/test_select_poll_epoll'"./run-ptest
and you should see the failure.
Updated by Richard Purdie about 18 hours ago
For the open files limit, I did set "ulimit -n 40960" before running the tests but it doesn't seem to change anything. Is that the correct limit to change?