LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912023-09-27T20:38:17ZLTTng bugs repository
Redmine LTTng-tools - Bug #1392 (Feedback): lttng view: Stream 0 is not declared in metadatahttps://bugs.lttng.org/issues/13922023-09-27T20:38:17ZRicardo Nabinger Sanchezrnsanchez@gmail.com
<p>While trying to capture UST events from a simple program, even though kernel events seem to be properly captured, UST events are not. Trace Compass cannot decode the UST data (but it can decode from the kernel channel).<br />The application is single-threaded, performs only disk and stdout I/O, nothing fancy. Compiled with <code>-finstrument-functions</code>, no optimizations, and with debugging symbols.</p>
<p>Session setup:<br /><pre>
# lttng create
# lttng enable-channel -u --subbuf-size=64M --num-subbuf=8 chan_ust
# lttng enable-channel -k --subbuf-size=64M --num-subbuf=2 chan_kernel
# lttng add-context -u -t vpid -t ip -t procname -t vtid -c chan_ust
# lttng enable-event -u -a -c chan_ust
# lttng enable-event -k -c chan_kernel --syscall --all
# lttng enable-event -k -c chan_kernel lttng_statedump_block_device,lttng_statedump_file_descriptor,lttng_statedump_process_state,mm_page_alloc,mm_page_free,net_dev_xmit,netif_receive_skb,sched_pi_setprio,sched_process_exec,sched_process_fork,sched_switch,sched_wakeup,sched_waking,softirq_entry,softirq_exit,softirq_raise
# lttng start
(possibly excessive preloads, but these are from my notes, collecting data on Varnish Cache)
# LTTNG_UST_BLOCKING_RETRY_TIMEOUT=-1 LD_PRELOAD=liblttng-ust-fd.so:liblttng-ust-fork.so:liblttng-ust-dl.so:liblttng-ust-cyg-profile.so:liblttng-ust-libc-wrapper.so:liblttng-ust-pthread-wrapper.so tests/test-playlist tests/playlist/bug_ffmpeg_1000-x20.m3u
# lttng stop
# lttng view > /dev/null
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
[error] Stream 0 is not declared in metadata.
</pre></p>
Environment:
<ul>
<li>lttng-modules-2.13.10 (from tarball)</li>
<li>lttng-tools-2.13.11 (from tarball)</li>
<li>lttng-ust-2.13.6 (from tarball)</li>
<li>babeltrace (git; at <code>v1.5.11</code>)</li>
<li>Linux kernel 6.6.0-rc1</li>
<li>Slackware64-current, up-to-date as of 2023-09-26</li>
<li>Trace Compass 9.1.0.202309200833</li>
</ul>
<p>IRC snippet:<br /><pre>
Sep/27 16:13:54 rnsanchez I'm trying to find out what I'm doing wrong.. I keep getting "[error] Stream 0 is not declared in metadata." in my captures. I guess this is making tracecompass not being able to show UST events (simple function enter-exit instrumentation)
Sep/27 16:16:46 rnsanchez using my old notes, while checking https://lttng.org/docs/v2.13/#doc-liblttng-ust-cyg-profile and other sections
Sep/27 16:18:25 jgalar rnsanchez: shot in the dark, but are you making sure the session is stopped or destroyed before reading the trace?
Sep/27 16:18:28 rnsanchez if it helps knowing, events from kernel channel are fine and I can see them in tracecompass (the trace is huge)
Sep/27 16:18:52 rnsanchez jgalar: yes, with lttng stop, then I copy the data from root to my user
Sep/27 16:19:10 rnsanchez but I get that Stream 0 regardless.. it's from lttng view
Sep/27 16:20:11 jgalar it sounds like a metadata flushing issue, any chance you can share the trace? (or at least, the metadata files)
Sep/27 16:20:42 rnsanchez yes, sure. just a moment while I upload it
Sep/27 16:20:48 jgalar okay thanks
Sep/27 16:21:43 ebugden not sure if related, but it's triggering deja vu of this issue: https://github.com/efficios/barectf/commit/d024537859c1d869bfa1cedc8abe8e3f7a648faa
Sep/27 16:22:23 ebugden maybe a similar problem in a similar area in lttng view
Sep/27 16:25:28 rnsanchez jgalar: https://rnsanchez.wait4.org/auto-20230927-161138.tar.xz
Sep/27 16:27:33 rnsanchez jgalar: this last one has incomplete events as I was trying to investigate, but I can bundle another one with full events as I used to collect
Sep/27 16:31:51 rnsanchez ebugden: is there anything on my side I can do?
Sep/27 16:34:38 ebugden rnsanchez: i don't feel i'm familiar enough with this situation to say; i was tossing out the association as possible food for thought for jgalar
Sep/27 16:35:12 rnsanchez oh ok :-)
Sep/27 16:43:03 rnsanchez jgalar: just in case, here is with full events: https://rnsanchez.wait4.org/auto-20230927-164014.tar.xz
Sep/27 16:45:53 jgalar ebugden: yup, the errors are indeed similar
Sep/27 16:50:59 jgalar hmm, the user space trace's metadata doesn't declare any stream class, that's... unexpected...
Sep/27 16:51:14 jgalar which versions of lttng-ust and lttng-tools are you running?
Sep/27 16:55:03 rnsanchez ust 2.13.6 (tarball) and tools 2.13.11 (tarball too)
Sep/27 16:55:07 rnsanchez compiled today
Sep/27 16:55:13 jgalar okay
Sep/27 17:17:57 ebugden rnsanchez (cc: jgalar): would you open a bug report on https://bugs.lttng.org/ ?
Sep/27 17:18:16 rnsanchez ebugden: sure
Sep/27 17:19:21 ebugden thanks! (we have a few hypotheses, but we'd need more information about exactly how the trace is generated)
Sep/27 17:19:58 jgalar rnsanchez: it's weird, the user space trace's stream files just have the packet headers... if you can reproduce the problem while running lttng-sessiond with the `-vvv --verbose-consumer` options to get the logs, it would be helpful
Sep/27 17:20:47 rnsanchez I can try. I suppose I have to finish the sessiond already running and manually launch another?
Sep/27 17:21:06 jgalar yep
</pre></p> LTTng - Bug #1385 (Feedback): [lttng-live] error: recevie a message in the pasthttps://bugs.lttng.org/issues/13852023-08-16T18:53:52ZBin Yuan
<p>My program works well most of time. But in rare case, it reports the following hightling error.<br /><img src="https://bugs.lttng.org/attachments/download/572/clipboard-202308170250-vfley.jpg" alt="" loading="lazy" /></p>
<p>What will lead to those branch? any suggestion to debug?</p> LTTng - Bug #1369 (Feedback): Nano clock value overflows the signed 64-bit integer range on the b...https://bugs.lttng.org/issues/13692023-03-26T02:41:32ZBin Yuan
<p>I launched a program producing traces to the lttng consumerd and lttng relayd, also launching a babeltrace command as lttng live consumer to show the tracing real-timely.</p>
<p>It works well when the channel is set to per user mode. But when it is changed to per process per user without no other changes, the babeltrace2 client crashing showing a overflow error:</p>
<p>The last error message of babeltrace2 exception stack shows :</p>
<p>"Cannot convert cycle to nanoseconds from origin for given clock class: value overflows the signed 64-bit integer range: cc-addr=****, cc-name="monotonic", cc-freq=1000000000, ....."</p>
<p>What's the difference resulting this crash ? How to solve it since the perprocess per user mode is perfered for me.</p>
<p>version info:<br />babeltrace2: 2.0.4<br />lttng comands: 2.12.12</p> LTTng-tools - Bug #1342 (New): Rotation thread's handling of session consumed size notifications ...https://bugs.lttng.org/issues/13422021-12-09T21:40:35ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>Related to this change:<br /><a class="external" href="https://review.lttng.org/c/lttng-tools/+/6886/">https://review.lttng.org/c/lttng-tools/+/6886/</a></p>
The race described in that change has another consequence: the notification could refer to a different session instance of the same name:
<ul>
<li>notification emitted for session <code>foo</code>,</li>
<li>session <code>foo</code> is destroyed,</li>
<li>a new session named <code>foo</code> is created,</li>
<li>a rotation is triggered for <code>foo</code>.</li>
</ul>
There are multiple solutions for this:
<ul>
<li>Provide a unique session id as part the of the evaluation of <code>SESSION_CONSUMED_SIZE</code>,</li>
<li>Register an internal trigger that has the <code>rotate session</code> action (since ids are sampled during their handling).</li>
</ul>
<p>I can't work on a fix right now, but here is a placeholder patch that will eventually contain the fix:<br /><a class="external" href="https://review.lttng.org/c/lttng-tools/+/6907">https://review.lttng.org/c/lttng-tools/+/6907</a></p>
<p>This is hypothetical and I noticed it during a code review; nobody reported this problem.</p> LTTng-tools - Bug #1324 (New): lttng_enable_event() and lttng_enable_event_with_filter() cannot a...https://bugs.lttng.org/issues/13242021-08-24T19:48:46ZPhilippe Proulxeeppeliteloop@gmail.com
<p>Currently:</p>
<ul>
<li><code>lttng_enable_event()</code> calls <code>lttng_enable_event_with_exclusions()</code>, passing no filter expression and no event name exclusion patterns.</li>
<li><code>lttng_enable_event_with_filter()</code> calls <code>lttng_enable_event_with_exclusions()</code>, passing no event name exclusion patterns.</li>
</ul>
<p>This means that if you want to enable a recording event rule described with a filter expression and event name exclusion patterns, you need to:</p>
<ol>
<li>Get the filter expression from its descriptor with <code>lttng_event_get_filter_expression()</code>.</li>
<li>Build an array of event name exclusion patterns, getting them from its descriptor with <code>lttng_event_get_exclusion_name_count()</code> and <code>lttng_event_get_exclusion_name()</code>.</li>
<li>Call <code>lttng_enable_event_with_exclusions()</code>.</li>
</ol>
<p>This is what you would need to do to blindly enable a disabled recording event rule of which the descriptor comes from <code>lttng_list_events()</code>.</p>
<p><code>lttng_enable_event()</code> and <code>lttng_enable_event_with_filter()</code> could do the steps above for you, if this doesn't break any backward compatibility.</p> LTTng - Bug #1315 (Feedback): Kernel panics after `pkill lttng`; root session daemon has active t...https://bugs.lttng.org/issues/13152021-05-07T20:09:26ZPhilippe Proulxeeppeliteloop@gmail.com
<p>I don't know how to reproduce this bug.</p>
<p>I played with the <code>lttng add-trigger</code> command while writing usage examples for its manual page.</p>
<p>I started the root session daemon as such:</p>
<pre>
# lttng-sessiond --daemonize --group=eepp
</pre>
<p>I then ran commands such as:</p>
<pre>
$ lttng add-trigger --name user --condition=event-rule-matches --domain=user --action=notify
</pre>
<pre>
$ lttng add-trigger --condition=event-rule-matches \
--domain=user --action=notify \
--rate-policy=every:10
</pre>
<pre>
$ lttng add-trigger --owner-uid=33 \
--condition=event-rule-matches \
--domain=kernel --name='sched*' \
--action=notify
</pre>
<pre>
$ lttng add-trigger --condition=event-rule-matches \
--domain=kernel --type=syscall \
--filter='fd < 3' \
--action=start-session my-session \
--rate-policy=once-after:40
</pre>
<p>Note that I had no tracing session named <code>my-session</code>.</p>
<p>After a few minutes, Xorg froze. I managed to login again in a virtual console. I ran:</p>
<pre>
# pkill lttng
</pre>
<p>and got an instant kernel panic.</p>
<p>Attached is a photo of what was on the screen after running <code>pkill</code>.</p>
<p>Using:</p>
<ul>
<li>LTTng-tools <code>60860e547ce31ea629e846e00b66342425474b8d</code>.</li>
<li>LTTng-UST <code>a0f2513af262a19822d46f84cd5e34be0badc484</code></li>
<li>LTTng-modules <code>51ef453614a6db2b778595b16d93283d25db974a</code></li>
<li>liburcu <code>5e1b7c840a2b21b8442b322cedbb70a790e49520</code></li>
</ul> LTTng - Feature #1269 (New): Document clearly the versioning scheme of lttng-projects.https://bugs.lttng.org/issues/12692020-05-22T19:42:08ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Correct me if I am wrong but it is an "unwritten" rule that we follow semver [1]. Some organization might appreciate that we state clearly our versioning policy.</p>
<p>[1] <a class="external" href="https://semver.org/">https://semver.org/</a></p> LTTng - Feature #1268 (New): Adopting a Vulnerability Disclosure Policy for our projectshttps://bugs.lttng.org/issues/12682020-05-22T19:27:49ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Some organization might require/appreciate such policy.</p>
<p>We could probably base our policy on <a class="external" href="https://disclose.io/">https://disclose.io/</a>, considering they have "terms" [1] with canada in mind.</p>
<p>[1] <a class="external" href="https://github.com/disclose/disclose/tree/master/terms">https://github.com/disclose/disclose/tree/master/terms</a></p>
<p>IANAL</p> LTTng-tools - Feature #1180 (New): SDT tracing does not work when the probes are compiled with se...https://bugs.lttng.org/issues/11802019-03-27T15:41:21ZNaser Ez
<p>Tracing SDT probes does not work when the probes are compiled with semaphores.</p>
<p>For example, when we have a probe like:</p>
<pre>
Provider: nginx
Name: http__subrequest__start
Location: 0x0000000000429e9c, Base: 0x0000000000473810, Semaphore: 0x00000000006920ba
Arguments: 8@%rbx__
</pre>
<p>then running the following command:<br /><pre>
lttng enable-event -k nginx:http__subrequest__start --userspace-probe=sdt:/usr/local/nginx/sbin/nginx:nginx:http__subrequest__start
</pre></p>
<p>will generate this error:<br /><pre>
Error: Event nginx:http__subrequest__start: Invalid userspace probe location (channel channel0, session auto-20190327-105118)
</pre></p> LTTng-tools - Bug #994 (Confirmed): Configured metadata data channel not listed for UST domainhttps://bugs.lttng.org/issues/9942016-01-27T16:17:44ZBernd HufmannBernd.Hufmann@ericsson.com
<p>I noticed this for LTTng 2.7 and 2.6.</p>
<p>After configuring the metadata channel for UST, the metadata channel is not listed when listing the session. By the way, it works for the kernel domain.</p>
<p>To reproduce:</p>
<blockquote>
<p>lttng create my<br />lttng enable-channel metadata -u -s my --subbuf-size 65536<br />lttng list my</p>
</blockquote>
<p>Tracing session my: [inactive]<br /> Trace path: /home/user/lttng-traces/my-20160127-105230</p>
<p>=== Domain: UST global ===</p>
<p>Buffer type: per UID</p> LTTng-tools - Bug #987 (Confirmed): Incomplete string comparison patternhttps://bugs.lttng.org/issues/9872016-01-06T23:23:31ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>In a few instances in lttng-tools, a string comparison pattern involving strncmp and strlen() to limit the length of the comparison is used.</p>
<p>One can quickly find them with:</p>
<p>grep -r -A 3 strncmp . | less -> search for strlen or strnlen</p>
<p>Many of those are OK: they indeed only want to match the beginning of the strings. However, this pattern is incorrectly used in cases where a full match is required, for instance:</p>
<p>ust-registry.c:ht_match_event()<br />./src/bin/lttng/commands/enable_channels.c: if (!strncmp(output_mmap, opt_output, strlen(output_mmap))) {<br />./src/bin/lttng/commands/enable_channels.c: } else if (!strncmp(output_splice, opt_output, strlen(output_splice))) {<br />./src/bin/lttng-sessiond/snapshot.c: if (!strncmp(output->name, name, strlen(name))) {</p> LTTng-tools - Feature #986 (New): Warn when unreasonably short timer values are usedhttps://bugs.lttng.org/issues/9862016-01-05T22:35:59ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>Most timer configuration options are expressed in microseconds which makes it easy for users to specify an unreasonably short intervals of time. These configuration mistakes result in unusually high tracing overhead and may be quite challenging to figure out for inexperienced users.</p>
<p>The lttng client should warn when timers are set to very low values.</p> LTTng-modules - Feature #950 (New): Exclude specific kernel events when enabling all of themhttps://bugs.lttng.org/issues/9502015-10-09T22:55:29ZJulien Desfossezjdesfossez@efficios.com
<p>It would be nice to be able to do that (like in UST):<br />lttng enable-event -k -a --exclude sched_switch</p>
<p>My current workaround:<br />lttng enable-event <del>k $(lttng list -k | grep -v -E "Kernel|---</del>" | grep -v sched_switch | awk '{print $1}' | tr "\n" ",")</p> LTTng-tools - Feature #932 (Confirmed): LTTng client error code semantics are ill-definedhttps://bugs.lttng.org/issues/9322015-09-05T16:34:01ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>The LTTNG (1) man page mentions that</p>
<pre>
On success 0 is returned and a positive value on error. Value of 1 means a command error, 2 an undefined command, 3 a fatal error and 4 a command warning meaning that something went wrong during the command.
Any other value above 10, please refer to <lttng/lttng-error.h> for a detailed list or use lttng_strerror() to get a human readable string of the error code.
</pre>
<p>There are a number of things wrong with this. First of all, asking script authors to check some obscure error header deep in the bowels of their system is questionable, at best.</p>
<p>Second, as remarked by Michael Jeanson, Philippe Proulx, most command-line utilities don't return an error code to indicate a warning. A non-zero error code means an error as occurred.</p>
<p>We will have to go through commands and determine which errors are in fact harmless warnings (e.g. enabling the same event twice), and which ones shall signal an error (e.g. invalid channel configuration).</p> LTTng-tools - Bug #901 (In Progress): Some liblttng-ctl don't return LTTNG_OK on successhttps://bugs.lttng.org/issues/9012015-08-05T18:26:10ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>It appears that some liblttng-ctl functions, such as lttng_create_session_snapshot() have conflicting return code conventions.</p>
<p>In this specific case, the header under lttng/session.h asserts that the function will<br /><pre>
/*
[...]
* Return 0 on success else a negative LTTng error code.
*/
</pre></p>
<p>while the header in lttng-ctl.c affirms that it<br /><pre>
/*
[...]
* Returns LTTNG_OK on success or a negative error code.
*/
</pre></p>
<p>and the function actually returns<br /><pre>
ret = lttng_ctl_ask_sessiond_varlen(&lsm, uris, ...
</pre></p>
<p>which, itself will<br /><pre>
/*
[...]
Return size of data (only payload, not header) or a negative error code.
*/
</pre></p>
<p>This pattern is used in multiple places which breaks code which checks for "ret == LTTNG_OK" instead of "ret < 0".</p>