LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912022-09-13T13:22:48ZLTTng bugs repository
Redmine LTTng-modules - Bug #1358 (New): Failed to deploy lttng modules on NVIDIA jetson devicehttps://bugs.lttng.org/issues/13582022-09-13T13:22:48Zliuhonggang liu
<p>Hello, I installed lttng and lttng modules on NVIDIA Orin. <br />When using apt install, the results are as follows.<br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># lttng list --kernel</span>
Error: Unable to list kernel events: Kernel tracer not available
</code></pre></p>
<pre><code class="shell syntaxhl" data-language="shell"><span class="c">#ps aux | grep lttng-sessiond</span>
root 52100 0.0 0.0 1022064 12736 ? Ssl 15:05 0:00 /usr/bin/lttng-sessiond
root 52101 0.0 0.0 41968 664 ? S 15:05 0:00 /usr/bin/lttng-sessiond
orin-d 62549 0.0 0.0 11640 684 pts/0 S+ 20:54 0:00 <span class="nb">grep</span> <span class="nt">--color</span><span class="o">=</span>auto lttng-sessiond
</code></pre>
<pre>
# dpkg -l | grep lttng
ii liblttng-ctl0:arm64 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility library
ii liblttng-ust-ctl4:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (trace control library)
ii liblttng-ust-dev:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (development files)
ii liblttng-ust-python-agent0:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (Python agent native library)
ii liblttng-ust0:arm64 2.12.2-1~ubuntu20.04.1 arm64 LTTng 2.0 Userspace Tracer (tracing libraries)
ii lttng-modules-dkms 2.12.6-1~ubuntu20.04.1 all Linux Trace Toolkit (LTTng) kernel modules (DKMS)
ii lttng-tools 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility programs
ii python3-lttng 2.12.4-1~ubuntu20.04.1 arm64 LTTng control and utility Python bindings
</pre>
<p>The device information is as follows.<br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># uname -a</span>
Linux orind-d 5.10.65-tegra <span class="c">#2 SMP PREEMPT Thu Jun 16 18:24:26 CST 2022 aarch64 aarch64 aarch64 GNU/Linux</span>
</code></pre><br /><pre><code class="shell syntaxhl" data-language="shell"><span class="c"># cat /etc/os-release</span>
<span class="nv">NAME</span><span class="o">=</span><span class="s2">"Ubuntu"</span>
<span class="nv">VERSION</span><span class="o">=</span><span class="s2">"20.04.4 LTS (Focal Fossa)"</span>
<span class="nv">ID</span><span class="o">=</span>ubuntu
<span class="nv">ID_LIKE</span><span class="o">=</span>debian
<span class="nv">PRETTY_NAME</span><span class="o">=</span><span class="s2">"Ubuntu 20.04.4 LTS"</span>
<span class="nv">VERSION_ID</span><span class="o">=</span><span class="s2">"20.04"</span>
<span class="nv">HOME_URL</span><span class="o">=</span><span class="s2">"https://www.ubuntu.com/"</span>
<span class="nv">SUPPORT_URL</span><span class="o">=</span><span class="s2">"https://help.ubuntu.com/"</span>
<span class="nv">BUG_REPORT_URL</span><span class="o">=</span><span class="s2">"https://bugs.launchpad.net/ubuntu/"</span>
<span class="nv">PRIVACY_POLICY_URL</span><span class="o">=</span><span class="s2">"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"</span>
<span class="nv">VERSION_CODENAME</span><span class="o">=</span>focal
<span class="nv">UBUNTU_CODENAME</span><span class="o">=</span>focal
</code></pre></p>
<p>At the same time, I tried the method of source code, such as the source code installation method introduced in <a class="external" href="https://lttng.org/docs/v2.13/">https://lttng.org/docs/v2.13/</a>.<br /><pre>
# dpkg -l | grep -e libuuid -e popt -e userspace -e libxml2
ii can-utils 2018.02.0-1ubuntu1 arm64 SocketCAN userspace utilities and tools
ii dmsetup 2:1.02.167-1ubuntu1 arm64 Linux Kernel Device Mapper userspace library
ii gvfs:arm64 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - GIO module
ii gvfs-backends 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - backends
ii gvfs-bin 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - deprecated command-line tools
ii gvfs-common 1.44.1-1ubuntu1 all userspace virtual filesystem - common data files
ii gvfs-daemons 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - servers
ii gvfs-fuse 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - fuse server
ii gvfs-libs:arm64 1.44.1-1ubuntu1 arm64 userspace virtual filesystem - private libraries
ii libdevmapper1.02.1:arm64 2:1.02.167-1ubuntu1 arm64 Linux Kernel Device Mapper userspace library
ii libi2c0:arm64 4.1-2build2 arm64 userspace I2C programming library
ii libibverbs1:arm64 28.0-1ubuntu1 arm64 Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii libnftnl11:arm64 1.1.5-1 arm64 Netfilter nftables userspace API library
ii libpopt-dev:arm64 1.16-14 arm64 lib for parsing cmdline parameters - development files
ii libpopt0:arm64 1.16-14 arm64 lib for parsing cmdline parameters
ii liburcu-dev:arm64 0.12.2-1~ubuntu20.04.2 arm64 userspace RCU (read-copy-update) library - development files
ii liburcu6:arm64 0.12.2-1~ubuntu20.04.2 arm64 userspace RCU (read-copy-update) library
ii libusb-1.0-0:arm64 2:1.0.23-2build1 arm64 userspace USB programming library
ii libusb-1.0-0-dev:arm64 2:1.0.23-2build1 arm64 userspace USB programming library development files
ii libuuid1:arm64 2.34-0.1ubuntu9.3 arm64 Universally Unique ID library
ii libxml2:arm64 2.9.10+dfsg-5ubuntu0.20.04.1 arm64 GNOME XML library
ii libxml2-dev:arm64 2.9.10+dfsg-5ubuntu0.20.04.1 arm64 Development files for the GNOME XML library
ii libxml2-utils 2.9.10+dfsg-5ubuntu0.20.04.3 arm64 XML utilities
ii network-manager 1.22.10-1ubuntu2.3 arm64 network management framework (daemon and userspace tools)
ii nvidia-l4t-optee 34.1.0-20220406120854 arm64 OP-TEE userspace daemons, test programs and libraries
ii python3-lxml:arm64 4.5.0-1ubuntu0.5 arm64 pythonic binding for the libxml2 and libxslt libraries
</pre></p>
<pre>
sudo ln -snf /usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10 /lib/modules/5.10.65-tegra/build
</pre>
<pre>
# orin-d@orind-d:~/tmp/lttng-modules-2.13.5$ make
/home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.c:20:3: error: #error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
20 | # error "LTTng-modules requires CONFIG_KPROBES on kernels >= 5.7.0"
| ^~~~~
make[2]: *** [scripts/Makefile.build:281: /home/orin-d/tmp/lttng-modules-2.13.5/src/wrapper/kallsyms.o] Error 1
make[1]: *** [Makefile:1852: /home/orin-d/tmp/lttng-modules-2.13.5/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.10.65-tegra-ubuntu20.04_aarch64/kernel-5.10'
make: *** [Makefile:31: modules] Error 2
</pre> LTTng-UST - Bug #1132 (New): Fails to build with Crosscompilerhttps://bugs.lttng.org/issues/11322017-10-27T11:46:39ZNorbert Lange
<p>The Userspace libary will fail to build when crosscompiling, this is easily repruducable with Buildroot for example.</p>
<p>As far as I can tell, the transitive dependency to liblttng-ust-tracepoint.so.0 when referencing liblttng-ust.so is lost, <br />I suppose it would not appear if the host or sysroot contains an installed liblttng-ust-tracepoint.so.0.</p>
<p>Error Message is following:</p>
<p>[ 70%] Linking CXX executable tester<br />/usr/bin/cmake -E cmake_link_script CMakeFiles/tester.dir/link.txt --verbose=1<br />/tmp/build/host/bin/x86_64-buildroot-linux-gnu-g++ -g -O2 -rdynamic CMakeFiles/tester.dir/tester.cpp.o -o tester -Wl,-rpath,/tmp/a/doc/examples/cmake-multiple-shared-libraries/build:/tmp/a/liblttng-ust/.libs libaligner-lib.so libtester-lib.so libtracepoint-provider.so /tmp/a/liblttng-ust/.libs/liblttng-ust.so -ldl <br />/tmp/build/host/lib/gcc/x86_64-buildroot-linux-gnu/7.2.0/../../../../x86_64-buildroot-linux-gnu/bin/ld: warning: liblttng-ust-tracepoint.so.0, needed by /tmp/a/liblttng-ust/.libs/liblttng-ust.so, not found (try using -rpath or -rpath-link)<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `exit_tracepoint'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_unregister_queue_release'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_register_queue_release'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `__tracepoint_probe_prune_release_queue'<br />/tmp/a/liblttng-ust/.libs/liblttng-ust.so: undefined reference to `init_tracepoint'<br />collect2: error: ld returned 1 exit status</p>
<p>This issue is atleast occuring in 2.9.0, 2.9.1 and 2.10.0</p> LTTng - Feature #968 (Feedback): lttng-modules kernel and user callstack contexthttps://bugs.lttng.org/issues/9682015-10-25T20:14:40ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Implementation from Francis Giraldeau reviewed and cleaned up available here:</p>
<p><a class="external" href="https://github.com/compudj/lttng-tools-dev/tree/callstack">https://github.com/compudj/lttng-tools-dev/tree/callstack</a><br /><a class="external" href="https://github.com/compudj/lttng-modules-dev/tree/callstack">https://github.com/compudj/lttng-modules-dev/tree/callstack</a></p>
<p>Documentation and tests are missing.</p> LTTng-UST - Feature #965 (New): Implement UST statedumphttps://bugs.lttng.org/issues/9652015-10-22T20:43:37ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Initial implementation: <a class="external" href="https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier">https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier</a></p>
<p>Missing tests in lttng-tools for this feature before we can merge it into lttng-ust.</p> LTTng-modules - Feature #964 (New): Implement support for persistent memory buffershttps://bugs.lttng.org/issues/9642015-10-22T20:41:43ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #963 (New): Implement user-space stack dump contexthttps://bugs.lttng.org/issues/9632015-10-22T20:41:10ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-tools - Feature #286 (Confirmed): Add a --logfile option (or log through Syslog)https://bugs.lttng.org/issues/2862012-06-21T22:21:15ZYannick Brosseauyannick.brosseau@polymtl.ca
<p>When we start lttng-sessiond in deamon mode, there is no way to get the verbose log.</p>
<p>having an option to log to a file and to syslog would be nice to debug deamon cases.</p> Common Trace Format - Bug #265 (New): Specify where exactly the event ID must be in the headerhttps://bugs.lttng.org/issues/2652012-06-15T15:43:02ZPhilippe Proulxeeppeliteloop@gmail.com
<p>Here's how to read an event, having already parsed the metadata:</p>
<ol>
<li>read the event header (this is defined per-stream)</li>
<li><strong>find the ID</strong></li>
<li>find the event declaration for that ID</li>
<li>read the binary event according to its declaration</li>
</ol>
<p>Problem lies in step 2. There's no definition in the specs. regarding how to find the ID field within the event header. It cannot be as simple as finding the <code>id</code> field in the event header declaration since it can be elsewhere.</p>
<p>A good example is the LTTng event header declaration, which are often:</p>
<pre>
struct event_header_large {
enum : uint16_t { compact = 0 ... 65534, extended = 65535 } id;
variant <id> {
struct {
uint32_clock_monotonic_t timestamp;
} compact;
struct {
uint32_t id;
uint64_clock_monotonic_t timestamp;
} extended;
} v;
} align(8);
</pre>
<p>Here, <code>id</code> is most of the time the actual ID, but sometimes it's 65535 in order to extend the header using the variant and <code>v.extended.id</code> is the real ID. This is not specified in the specs.</p>
<p>We need a way to know (in the metadata) where is the real ID and how to know it once we read the header (between steps 1 and 2).</p> Common Trace Format - Bug #262 (New): Be clearer about fields of headers and contextshttps://bugs.lttng.org/issues/2622012-06-08T19:21:17ZPhilippe Proulxeeppeliteloop@gmail.com
<p>I've read the CTF specs. for hours by now and I still think there's something wrong about it. It seems like there's a division between some structure fields and their descriptions. Instead of clearly describing fields of some structures, only examples are given. But how are the examples related to the descriptions?</p>
<p>I would really appreciate that all fields of the following structures be very well defined in the specs.:</p>
<ul>
<li><code>trace.packet.header</code></li>
<li><code>stream.packet.context</code></li>
<li><code>stream.event.header</code></li>
</ul>
<p>This is actually done, but we don't see the relation between the field names and their descriptions.</p>
<p>Here's an example. The list following <em>Event packet context (all fields are optional, specified by TSDL meta-data):</em> is okay, but look at <em>Event packet content size (in bits).</em>: we don't know anything about this field yet. It's only later in the text that we learn:</p>
<pre>
If the content
size field is missing, the packet is filled (no padding). The content
and packet sizes include all headers.
</pre>
<p>Still, it's only when looking at the <em>example</em> that we see its name:</p>
<pre>
struct event_packet_context {
uint64_t timestamp_begin;
uint64_t timestamp_end;
uint32_t checksum;
uint32_t stream_packet_count;
uint32_t events_discarded;
uint32_t cpu_id;
uint32_t/uint16_t content_size;
uint32_t/uint16_t packet_size;
uint8_t compression_scheme;
uint8_t encryption_scheme;
uint8_t checksum_scheme;
};
</pre>
<p>But an example isn't very formal, is it? So if I want to know something about this field, I have to look at 3 different places in the specs.</p>
<p>And what about the <code>cpu_id</code> field in there? Is this LTTng-related or standard within CTF? I guess <em>this one</em> is really an example, but it's confusing because we learn the content size field name at the same place.</p>
<p>A good and easy format to understand/read would be a table, for each aforementioned structure, with the following columns:</p>
<ul>
<li>optional/mandatory/conditional?</li>
<li>absence of field meaning: what exactly to expect if the field is absent?</li>
<li>conditions if field is conditional (depends on other parameters)</li>
<li>field name (e.g. <code>content_size</code>)</li>
<li>complete description</li>
</ul>
<p>Maybe a <em>since version</em> column would also be great, so we may have some backward compatibility.</p>
<p>Also: for each <strong>structure</strong>, is it allowed to add custom fields?</p> Common Trace Format - Bug #254 (New): No specified charset for metadata packets payloadhttps://bugs.lttng.org/issues/2542012-06-05T21:19:18ZPhilippe Proulxeeppeliteloop@gmail.com
<p>There's no current specified charset for the metadata packets payload. This can be problematic because if a tracer makes this Unicode and we read it thinking it's ASCII, the displayed text will be weird and the reading could even break.</p>
Possible solutions are:
<ul>
<li>lock it in the specifications</li>
</ul>
<blockquote>
<ul>
<li>UTF-8: ASCII-compatible, so this shouldn't make a big difference for most cases but allows for i18n of event names and so on</li>
<li>ASCII: simple, but only English</li>
<li>avoid everything else IMO</li>
</ul>
</blockquote>
<ul>
<li>add a charset byte within the metadata packet header (e.g. 0 => ASCII, 1 => UTF-8, etc.), but this breaks binary compatibility</li>
</ul> LTTng - Feature #211 (New): Environment varshttps://bugs.lttng.org/issues/2112012-04-12T16:44:18ZMatthew Khouzam
Hi, I think it would be an interesting items to put in the environment variables.
<ul>
<li>Session creation time. ( To be sure you're dealing with the right trace) </li>
<li>Session destruction time. </li>
<li>Cpuinfo ( to understand performance, if you're running a 386 or an i7, the latencies are different)</li>
<li>Amount of Ram ( could quickly explain swapping)</li>
<li>hw info </li>
<li>network info</li>
<li>hostname (oops tracing the wrong machine)</li>
</ul> LTTng-tools - Feature #193 (Feedback): Find a more intelligent heuristics for the FD reservationhttps://bugs.lttng.org/issues/1932012-03-20T20:12:48ZYannick Brosseauyannick.brosseau@polymtl.ca
<p>We currently reserve 25% of FD in the session deamon.</p>
<p>We could compute a better estimation, especially if we have lots of FD authorized.</p> LTTng-tools - Feature #137 (Confirmed): No more tracing possible after consumerD dies (via kill c...https://bugs.lttng.org/issues/1372012-02-29T15:58:59ZTan le trantan.dung.le.tran@ericsson.com
<p>There are a couple of weird behavior after the consumerD got killed via "kill" command.</p>
<p>Commit used when this test got carried out:<br /> userspace-rcu : feb-08 fcf4348721903bed57027c7eb0ea13765895cd37<br /> lttng-ust : feb-23 (20:11) 64493e4fa019e9bdfe0d9c6a4738c9552f250f35<br /> lttng-tools : feb-24 (14:41) 6bf73bf53464ad309fcf7f02a4dc397d280b81f8<br /> babeltrace : feb-23 (13:59) 305c65e5d7156ae7936f07ad93dd45ac318b4ce2</p>
<p>Here is the sceanrio: <br />1)_ lttng-sessiond -vvv &<br />2)_ Run the instrumented app <br />3)_ lttng create feb29_ses1 -o /tmp/tdlt/feb29_ses1<br /> lttng enable-event the_tracepoint_name -u<br /> lttng start<br /> Note: There are a couple of printout, "TEST: lib_ring_buffer_reserve data_size..." <br /> Those printf lines have been added by Mathieu to troubleshoot<br /> Issue-24 yesterday.<br />4)_ ps -elf |grep ltt<br />5)_ kill <process ID of consumerD><br />6)_ ps -elf |grep ltt<br /> get: [lttng-consumerd] <defunct><br /> No new consumerD has been spawned<br />7)_ lttng stop<br />8)_ lttng start feb29_ses1 (start the session again with the hope that new<br /> consummerD will be spawned)<br /> Note: still no new consummerD</p>
<p>9)_ lttng stop<br />10)_ ls -lR /tmp/tdlt/feb29_ses1/ust/<br /> get: all files with 0 size</p>
<p>11)_ lttng destroy</p>
<p>12)_ lttng create feb29_ses2 -o /tmp/tdlt/feb29_ses2 (test if new session is effected by the<br /> fact that consumerD dies previously)</p>
<p>13)_ lttng enable-event tracepoint_name -s feb29_ses2 -u</p>
<pre><code>Note: This process hang. Can not get the prompt back from the terminal (from where<br /> this command is entered).</code></pre>
<pre><code>&lt;ctrl&gt;+&lt;z&gt;<br /> bg</code></pre>
<pre><code>ps -elf |grep ltt<br /> get: :<br /> [lttng-consumerd] &lt;defunct&gt;<br /> lttng enable-event .... (enable-event is hanging)<br /> lttng-consumerd --quiet -u .... (a new consumerD is spawned)<br /> :</code></pre>
<p>14)_ lttng start<br /> lttng stop<br /> ls -lR /tmp/tdlt/feb29_ses2/ust/<br /> get: no files being created for this session !</p>
<p>15)_ lttng enable-event a_new_tracepoint -s feb29_2 -u<br /> Note: this one is also hanging !</p>
<pre><code>&lt;ctrl&gt;+&lt;z&gt;<br /> bg<br /> ps -elf |grep ltt<br /> get: :<br /> [lttng-consumerd] &lt;defunct&gt;<br /> lttng enable-event .... (1st enable-event is hanging)<br /> lttng-consumerd --quiet -u .... (previous consumerD is spawned)<br /> lttng enable-event .... (2nd enable-event is hanging)<br /> lttng-consumerd --quiet -u .... (a new consumerD is spawned)<br /> :</code></pre>
<p>16)_ kill <process_id of sessiond><br /> ps -elf |grep ltt<br /> get: :<br /> [lttng-consumerd] <defunct><br /> lttng enable-event .... (1st enable-event is hanging)<br /> lttng-consumerd --quiet -u .... (previous consumerD is spawned)<br /> lttng enable-event .... (2nd enable-event is hanging)<br /> lttng-consumerd --quiet -u .... (previous consumerD is spawned)<br /> :</p>
<p>17)_ lttng-sessiond -vvv &</p>
<p>18)_ lttng list -u <br /> Get: no apps has been registered !<br /> That means the application that already ran (before the<br /> consumerD is killed), can no longer see the new sessionD.</p>
<pre><code>Therefore, no more tracing is possible !</code></pre>
<p>Please, find enclosed the logfile for the above steps.</p> LTTng-tools - Feature #106 (Confirmed): Missing context information in list commandhttps://bugs.lttng.org/issues/1062012-02-24T18:13:01ZBernd HufmannBernd.Hufmann@ericsson.com
<p>After adding a context to an event there is no way to retrieve the context information, i.e. lttng list <session name> doesn't show the context information. <br />I think, a general rule should be, that things that can be configured should be able to be retrieved.</p>
<p>I used commit aeb968923d1245f4cb2be1eed2ab7b8b5eb45cf8</p>
<p>Here is an example sequence. As you can see "lttng list my"-comand doesn't show the context information:</p>
<blockquote>
<p>lttng create my</p>
</blockquote>
<p>Session my created.</p>
<p>Traces will be written in /home/eedbhu/lttng-traces/my-20120224-130527</p>
<blockquote>
<p>lttng enable-event sched_switch -k</p>
</blockquote>
<p>kernel event sched_switch created in channel channel0</p>
<blockquote>
<p>lttng add-context -k -e sched_switch -t pid</p>
</blockquote>
<p>kernel context pid added to event sched_switch</p>
<blockquote>
<p>lttng list my</p>
</blockquote>
<p>Tracing session my: [inactive]<br /> Trace path: /home/eedbhu/lttng-traces/my-20120224-130527</p>
<p>=== Domain: Kernel ===</p>
<p>Channels:<br />-------------<br />- channel0: [enabled]</p>
<pre><code>Attributes:<br /> overwrite mode: 0<br /> subbufers size: 262144<br /> number of subbufers: 4<br /> switch timer interval: 0<br /> read timer interval: 200<br /> output: splice()</code></pre>
<pre><code>Events:<br /> sched_switch (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]</code></pre> LTTng-tools - Feature #35 (Confirmed): Remote administrationhttps://bugs.lttng.org/issues/352012-02-12T01:11:02ZAnonymous
<p>Allow trace control on remote machines via the "lttng" command.</p>
<p>We can do some of this through SSH, but we'd like to be able to set up sessions and trace destinations remotely.</p>