LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912023-09-26T13:39:06ZLTTng bugs repository
Redmine LTTng-tools - Feature #1391 (New): Add 'enable-events' as a special argument to lttng-toolshttps://bugs.lttng.org/issues/13912023-09-26T13:39:06ZMatthew Khouzammatthew.khouzam@ericsson.com
<p>I have observed users trying to enable events by running</p>
<pre>
lttng enable-events -a -u
</pre><br /> or<br /><pre>
lttng enable-events -a -k
</pre><br /> with the logic that they are enabling several events. Then they panic when there is an error message. As this is an understandable issue, maybe it would be worth giving a clearer message: "It looks like you are trying to run 'enable-event'" or something similar. LTTng-modules - Bug #1358 (New): Failed to deploy lttng modules on NVIDIA jetson devicehttps://bugs.lttng.org/issues/13582022-09-13T13:22:48Zliuhonggang liu
<p>Hello, I installed lttng and lttng modules on NVIDIA Orin. <br />When using apt install, the results are as follows.<br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># lttng list --kernel</span>
Error: Unable to list kernel events: Kernel tracer not available
</code></pre></p>
<pre><code class="shell syntaxhl" data-language="shell"><span class="c">#ps aux | grep lttng-sessiond</span>
root 52100 0.0 0.0 1022064 12736 ? Ssl 15:05 0:00 /usr/bin/lttng-sessiond
root 52101 0.0 0.0 41968 664 ? S 15:05 0:00 /usr/bin/lttng-sessiond
orin-d 62549 0.0 0.0 11640 684 pts/0 S+ 20:54 0:00 <span class="nb">grep</span> <span class="nt">--color</span><span class="o">=</span>auto lttng-sessiond
</code></pre>
<pre>
# dpkg -l | grep lttng
ii liblttng-ctl0:arm64 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility library
ii liblttng-ust-ctl4:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (trace control library)
ii liblttng-ust-dev:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (development files)
ii liblttng-ust-python-agent0:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (Python agent native library)
ii liblttng-ust0:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (tracing libraries)
ii lttng-modules-dkms 2.12.6-1~ubuntu20.04.1 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii lttng-tools 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility programs
ii python3-lttng 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility Python bindings
</pre>
<p>The device information is as follows.<br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># uname -a</span>
Linux orind-d 5.10.65-tegra <span class="c">#2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux</span>
</code></pre><br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># cat /etc/os-release</span>
<span class="nv">NAME</span><span class="o">=</span><span class="s2">"Ubuntu"</span>
<span class="nv">VERSION</span><span class="o">=</span><span class="s2">"20.04.4 LTS (Focal Fossa)"</span>
<span class="nv">ID</span><span class="o">=</span>ubuntu
<span class="nv">ID_LIKE</span><span class="o">=</span>debian
<span class="nv">PRETTY_NAME</span><span class="o">=</span><span class="s2">"Ubuntu 20.04.4 LTS"</span>
<span class="nv">VERSION_ID</span><span class="o">=</span><span class="s2">"20.04"</span>
<span class="nv">HOME_URL</span><span class="o">=</span><span class="s2">"https://www.ubuntu.com/"</span>
<span class="nv">SUPPORT_URL</span><span class="o">=</span><span class="s2">"https://help.ubuntu.com/"</span>
<span class="nv">BUG_REPORT_URL</span><span class="o">=</span><span class="s2">"https://bugs.launchpad.net/ubuntu/"</span>
<span class="nv">PRIVACY_POLICY_URL</span><span class="o">=</span><span class="s2">"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"</span>
<span class="nv">VERSION_CODENAME</span><span class="o">=</span>focal
<span class="nv">UBUNTU_CODENAME</span><span class="o">=</span>focal
</code></pre></p>
<p>At the same time, I tried the method of source code, such as the source code installation method introduced in <a class="external" href="https://lttng.org/docs/v2.13/">https://lttng.org/docs/v2.13/</a>.<br /><pre>
# dpkg -l | grep -e libuuid -e popt -e userspace -e libxml2
ii can-utils 2018.02.0-1ubuntu1 arm64 SocketCAN userspace utilities and tools
ii dmsetup 2:1.02.167-1ubuntu1 arm64 Linux Kernel Device Mapper userspace library
ii gvfs:arm64 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - GIO module
ii gvfs-backends 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - backends
ii gvfs-bin 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - deprecated command-line tools
ii gvfs-common 1.44.1-1ubuntu1 all userspace virtual filesystem - common data files
ii gvfs-daemons 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - servers
ii gvfs-fuse 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - fuse server
ii gvfs-libs:arm64 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - private libraries
ii libdevmapper1.02.1:arm64 2:1.02.167-1ubuntu1 arm64 Linux Kernel Device Mapper userspace library
ii libi2c0:arm64 4.1-2build2 arm64 userspace I2C programming library
ii libibverbs1:arm64 28.0-1ubuntu1 arm64 Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii libnftnl11:arm64 1.1.5-1 arm64 Netfilter nftables userspace API library
ii libpopt-dev:arm64 1.16-14 arm64 lib for parsing cmdline parameters - development files
ii libpopt0:arm64 1.16-14 arm64 lib for parsing cmdline parameters
ii liburcu-dev:arm64 0.12.2-1~ubuntu20.04.2 arm64 userspace RCU (read-copy-update) library - development files
ii liburcu6:arm64 0.12.2-1~ubuntu20.04.2 arm64 userspace RCU (read-copy-update) library
ii libusb-1.0-0:arm64 2:1.0.23-2build1 arm64 userspace USB programming library
ii libusb-1.0-0-dev:arm64 2:1.0.23-2build1 arm64 userspace USB programming library development files
ii libuuid1:arm64 2.34-0.1ubuntu9.3 arm64 Universally Unique ID library
ii libxml2:arm64 2.9.10+dfsg-5ubuntu0.20.04.1 arm64 GNOME XML library
ii libxml2-dev:arm64 2.9.10+dfsg-5ubuntu0.20.04.1 arm64 Development files for the GNOME XML library
ii libxml2-utils 2.9.10+dfsg-5ubuntu0.20.04.3 arm64 XML utilities
ii network-manager 1.22.10-1ubuntu2.3 arm64 network management framework (daemon and userspace tools)
ii nvidia-l4t-optee 34.1.0-20220406120854 arm64 OP-TEE userspace daemons, test programs and libraries
ii python3-lxml:arm64 4.5.0-1ubuntu0.5 arm64 pythonic binding for the libxml2 and libxslt libraries
</pre></p>
<pre>
sudo ln -snf /usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10 /lib/modules/5.10.65-tegra/build
</pre>
<pre>
# orin-d@orind-d:~/tmp/lttng-modules-2.13.5$ make
/home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.c:20:3: error: #error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
20 | # error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
| ^~~~~
make[2]: *** [scripts/Makefile.build:281: /home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.o] Error 1
make[1]: *** [Makefile:1852: /home/orin-d/tmp/lttng-modules-2.13.5/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10'
make: *** [Makefile:31: modules] Error 2
</pre> LTTng-UST - Bug #1294 (New): add-context fails silently with perf_event_paranoid=4, no trace prod...https://bugs.lttng.org/issues/12942020-12-02T23:45:38ZJeff Trull
<p>Desired result:<br />A disallowed add-context (e.g. because of perf_event_paranoid) produces an error message and a non-zero exit code.</p>
<p>Actual result:<br />A disallowed add-context produces only a message in the system log, invisible to the normal user.<br />For example, when perf_event_paranoid forbids it you get a message of the form:</p>
<p><code>Error: UST app create channel context failed for app (pid: 76220) with ret -1024</code></p>
<p>But the only effects observable to the user are:</p>
<ol>
<li>no trace is produced, and</li>
<li>when you destroy the session there is a message like <code>Failed to perform a quiet rotation as part of the destruction of session "..."</code></li>
</ol>
<p>If other users are like me they will spend quite some time trying to figure out what is wrong in their code, setup etc. before figuring out what's wrong (or having someone on IRC identify it for them, as in my case).</p>
<p>This is Ubuntu 20.04.1, with lttng-ust 2.11.0</p> LTTng-UST - Bug #1285 (Feedback): about close file failed for lttng-ust modulehttps://bugs.lttng.org/issues/12852020-10-10T09:52:42Zkun song
<p>Hi,all</p>
<p>I found some relative modification for lttng-ust.<br />commit<br />20d1999d42392063fd530d1f6c00b93c25d178d6<br />793d29c9367a58c2a368bc1ccb11b01c2d2bdac2<br />48a143c09cc97bf7a2ace811277e7d60b294b5f6<br />96a6162e1925d4bd1dfc6f4167d75f396c0dbe4c<br />283f4bec47781f7a39dde82d02e53add02155f9c<br />c1be081a2f016fb6dcaef1d471389ede3aa00103<br />5a4d96d170a38f29bf147b84e248a47d45ee7d8e<br />7d34f27dce903a88241e96787c1f900b6ea08245<br />ba8dcc8ca5b8b7bebed4e10b3dd4fcbc783d529d</p>
<p>Test step<br />LD_PRELOAD=lttng-ust-fork.so dpdk-devbind.py -s<br />When exec command,close file failed.</p>
<p>#1 0x00007f31b9705524 in __GI_abort () at abort.c:79</p>
<p>[**ALERT: The abort() might not be exactly invoked from the following function line.<br />If the trail function contains multiple abort() calls, then you should cross check by other means to get correct abort() call location.<br />This is due to the optimized compilation which hides the debug info for multiple abort() calls in a given function.<br />Refer TR HU16995 for more information]<br />#2 0x00007f31b970540f in _<em>assert_fail_base (fmt=0x7f31b9862ec8<br />"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7f31b96bbcdd "0", file=0x7f31b96c4cd0 <del>Type <RET> for more, q to quit, c to continue without paging</del> "../../git/libringbuffer/shm.c", line=427, function=<optimized out>) at<br />assert.c:92<br />#3 0x00007f31b97127d2 in _GI</em>_assert_fail (assertion=assertion@entry =0x7f31b96bbcdd "0", file=file@entry=0x7f31b96c4cd0 "../../git/libringbuffer/shm.c", line=line@entry=427,<br />function=function@entry=0x7f31b96c4dc0 <_PRETTY_FUNCTION_.4744><br />"shmp_object_destroy") at assert.c:101<br />#4 0x00007f31b96b3f1d in shmp_object_destroy (obj=obj@entry=0x7f31b401 1740, consumer=consumer@entry=0) at ../../git/libringbuffer/shm.c:427<br />#5 0x00007f31b96b4d4f in shm_object_table_destroy (table=0x7f31b4011730, consumer=consumer@entry=0) at<br />../../git/libringbuffer/shm.c:451<br />#6 0x00007f31b96b034d in channel_free (consumer=0, handle=0x7f31b4012f10, chan=0x7f31b4011170) at<br />../../git/libringbuffer/ring_buffer_frontend.c:926<br /><a class="issue tracker-1 status-5 priority-5 priority-high2 closed" title="Bug: Double PID registering and unregistering race (Resolved)" href="https://bugs.lttng.org/issues/7">#7</a> channel_release (consumer=0, handle=0x7f31b4012f10,<br />chan=0x7f31b4011170) at<br />../../git/libringbuffer/ring_buffer_frontend.c:1129<br />#8 channel_destroy (chan=0x7f31b4011170, handle=0x7f31b4012f10,<br />consumer=consumer@entry=0) at</p>
<p>../../git/libringbuffer/ring_buffer_frontend.c:1161</p> LTTng - Bug #1208 (Feedback): 2.11 relayd in live mode attach window changedhttps://bugs.lttng.org/issues/12082019-11-06T19:59:35ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Previously on a lttng 2.10 relayd/tools the following sequence is acceptable and result in a hooked babeltrace:</p>
<pre>
lttng create
lttng enable-event -u -a
babeltrace -i lttng-live net://localhost
babeltrace -i lttng-live net://localhost/the_session_path
lttng start
</pre>
<p>On lttng 2.11, the same sequence leave babeltrace unable to hook itself before a "lttng start" is issued.</p>
<p>Nothing describe the guarantee that we expose on that side but we do have to ask ourselves if this sequence of action is valid or not.</p>
<p>A scenario where the user want to make sure to get the event ASAP on babeltrace side might want to hook itself to the session before any event can be recorded.</p>
<p>This was found using the lttng-ivc project. The test code was adapted to a functioning sequence but this should require more discussion overall. This is in no way a major concern.</p>
<p>Cheers</p> LTTng-UST - Bug #1132 (New): Fails to build with Crosscompilerhttps://bugs.lttng.org/issues/11322017-10-27T11:46:39ZNorbert Lange
<p>The Userspace libary will fail to build when crosscompiling, this is easily repruducable with Buildroot for example.</p>
<p>As far as I can tell, the transitive dependency to liblttng-ust-tracepoint.so.0 when referencing liblttng-ust.so is lost, <br />I suppose it would not appear if the host or sysroot contains an installed liblttng-ust-tracepoint.so.0.</p>
<p>Error Message is following:</p>
<p>[ 70%] Linking CXX executable tester<br />/usr/bin/cmake -E cmake_link_script CMakeFiles/tester.dir/link.txt --verbose=1<br />/tmp/build/host/bin/x86_64-buildroot-linux-gnu-g++ -g -O2 -rdynamic CMakeFiles/tester.dir/tester.cpp.o -o tester -Wl,-rpath,/tmp/a/doc/examples/cmake-multiple-shared-libraries/build:/tmp/a/liblttng-ust/.libs libaligner-lib.so libtester-lib.so libtracepoint-provider.so /tmp/a/liblttng-ust/.libs/liblttng-ust.so -ldl <br />/tmp/build/host/lib/gcc/x86_64-buildroot-linux-gnu/7.2.0/../../../../x86_64-buildroot-linux-gnu/bin/ld: warning: liblttng-ust-tracepoint.so.0, needed by /tmp/a/liblttng-ust/.libs/liblttng-ust.so, not found (try using -rpath or -rpath-link)<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `exit_tracepoint'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_unregister_queue_release'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_register_queue_release'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_prune_release_queue'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `init_tracepoint'<br />collect2: error: ld returned 1 exit status</p>
<p>This issue is atleast occuring in 2.9.0, 2.9.1 and 2.10.0</p> LTTng - Feature #968 (Feedback): lttng-modules kernel and user callstack contexthttps://bugs.lttng.org/issues/9682015-10-25T20:14:40ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Implementation from Francis Giraldeau reviewed and cleaned up available here:</p>
<p><a class="external" href="https://github.com/compudj/lttng-tools-dev/tree/callstack">https://github.com/compudj/lttng-tools-dev/tree/callstack</a><br /><a class="external" href="https://github.com/compudj/lttng-modules-dev/tree/callstack">https://github.com/compudj/lttng-modules-dev/tree/callstack</a></p>
<p>Documentation and tests are missing.</p> LTTng-UST - Feature #965 (New): Implement UST statedumphttps://bugs.lttng.org/issues/9652015-10-22T20:43:37ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Initial implementation: <a class="external" href="https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier">https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier</a></p>
<p>Missing tests in lttng-tools for this feature before we can merge it into lttng-ust.</p> LTTng-modules - Feature #964 (New): Implement support for persistent memory buffershttps://bugs.lttng.org/issues/9642015-10-22T20:41:43ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #963 (New): Implement user-space stack dump contexthttps://bugs.lttng.org/issues/9632015-10-22T20:41:10ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-tools - Feature #961 (New): Add additional criteria to disable-eventhttps://bugs.lttng.org/issues/9612015-10-22T19:42:43ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Add the necessary criteria to disable-event to procure a way to disable events based on filter string, loglevel etc.</p> LTTng-tools - Feature #960 (New): Add a -m /--matching argument to enable-eventhttps://bugs.lttng.org/issues/9602015-10-22T19:29:17ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>--matching would provide a way to reflect the behavior of disable-event which is a per field (name, filter, loglevel) event matching</p>
<p>By default enable-event would still try to enable a single event with static field check.</p>
<p>When in matching mode enable-event would iterate the current list of enabled events and enable events based on field matching (name, filter, loglevel) provided via arguments.</p> LTTng-tools - Feature #555 (Confirmed): add-context --help lists perf fields even when the machin...https://bugs.lttng.org/issues/5552013-05-31T18:13:39ZDaniel U. Thibaultdaniel.thibault@drdc-rddc.gc.ca
<p>This was taken from a machine running 3.2.0-40-virtual:</p>
<pre>
$ lttng add-context --help
usage: lttng add-context -t TYPE [-k|-u] [OPTIONS]
[...]
Context:
-t, --type TYPE Context type. You can repeat that option on
the command line to specify multiple contexts at once.
(--kernel preempts --userspace)
TYPE can be one of the strings below:
pid, procname, prio, nice, vpid, tid, pthread_id,
vtid, ppid, vppid, hostname, perf:cpu-cycles,
perf:cycles, perf:stalled-cycles-frontend,
perf:idle-cycles-frontend,
perf:stalled-cycles-backend,
perf:idle-cycles-backend, perf:instructions,
perf:cache-references, perf:cache-misses,
perf:branch-instructions, perf:branches,
perf:branch-misses, perf:bus-cycles,
perf:L1-dcache-loads, perf:L1-dcache-load-misses,
perf:L1-dcache-stores,
perf:L1-dcache-store-misses,
perf:L1-dcache-prefetches,
perf:L1-dcache-prefetch-misses,
perf:L1-icache-loads, perf:L1-icache-load-misses,
perf:L1-icache-stores,
perf:L1-icache-store-misses,
perf:L1-icache-prefetches,
perf:L1-icache-prefetch-misses, perf:LLC-loads,
perf:LLC-load-misses, perf:LLC-stores,
perf:LLC-store-misses, perf:LLC-prefetches,
perf:LLC-prefetch-misses, perf:dTLB-loads,
perf:dTLB-load-misses, perf:dTLB-stores,
perf:dTLB-store-misses, perf:dTLB-prefetches,
perf:dTLB-prefetch-misses, perf:iTLB-loads,
perf:iTLB-load-misses, perf:branch-loads,
perf:branch-load-misses, perf:cpu-clock,
perf:task-clock, perf:page-fault, perf:faults,
perf:major-faults, perf:minor-faults,
perf:context-switches, perf:cs,
perf:cpu-migrations, perf:migrations,
perf:alignment-faults, perf:emulation-faults
Example:
[...]
$ dmesg | grep "generic register"
$
</pre>
<p>Trying to <code>add-context -t perf:*</code> on such a machine fails, of course.</p>
<p>It would be a simple yet useful enhancement to forgo listing perf context types when none are actually possible.</p> LTTng-UST - Feature #527 (New): Add git version information to project versionhttps://bugs.lttng.org/issues/5272013-05-10T12:54:46ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-tools - Feature #286 (Confirmed): Add a --logfile option (or log through Syslog)https://bugs.lttng.org/issues/2862012-06-21T22:21:15ZYannick Brosseauyannick.brosseau@polymtl.ca
<p>When we start lttng-sessiond in deamon mode, there is no way to get the verbose log.</p>
<p>having an option to log to a file and to syslog would be nice to debug deamon cases.</p>