LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912024-02-21T17:04:43ZLTTng bugs repository
Redmine Babeltrace - Feature #1410 (New): Add pretty-print sink options to override how to format floatin...https://bugs.lttng.org/issues/14102024-02-21T17:04:43ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Babeltrace 2, just like its predecessor babeltrace 1.5, uses "%g" to print floating point numbers.</p>
<p>It would be useful to let users optionally override the formatting for floating point numbers, perhaps with a pretty print sink option. They could then choose if exponent notation should be used or not, choose the precision, etc.</p> LTTng-tools - Bug #1379 (New): Document behavior of live mode with per-pid UST buffershttps://bugs.lttng.org/issues/13792023-06-01T18:48:58ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>There is a design limitation of per-pid UST buffers when used in live mode which should be documented.</p>
<p>Basically, because the per-pid buffers are reclaimed soon after the application exits, it may cause the events traced by a short-lived application to never appear in the output of a live trace when per-pid buffers are used.</p>
<p>This is caused by the fact that the live client periodically checks for new buffers, but there is no inherent reference kept on the streams before they disappear.</p>
<p>This means that the information will be available in the disk output of the trace, but it will be missing from the live mode output.</p>
<p>This could eventually be mitigated by keeping references to the streams received by the relay daemon which are of interest to a live client until they are referenced by the live client.</p> LTTng-tools - Bug #1378 (New): Document behavior of snapshot mode with per-pid UST buffershttps://bugs.lttng.org/issues/13782023-06-01T18:01:12ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should document a design limitation of the per-pid UST buffers which can lead to unexpected results when used with the "snapshot" feature.</p>
<p>Basically, because the per-pid buffers lifetime is bound by the application using them (and the consumer daemon extracting data from them), they are freed almost immediately after the traced application exits. However, in a scenario where we are interested in capturing a snapshot of the content of flight recorder buffers soon after application exits (e.g. caused by a crash), those buffers are not available anymore a few milliseconds after the application exits.</p>
<p>We should document this design limitation, and eventually think about improvements (feature request ?) to allow a configurable delay after application exit during which the consumer daemon could keep references on the buffers so they are available for snapshot.</p> LTTng-tools - Bug #1340 (New): Document behavior of per-uid UST buffers with respect to asynchron...https://bugs.lttng.org/issues/13402021-12-07T20:22:28ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should document a known design limitation of the per-uid UST buffers:</p>
<p>When using lttng-ust with per-uid buffers, there is a known design limitation<br />regarding process terminated by non-handled kill signals: if this occurs while<br />the buffer is being written to (between reserve and commit), it will cause the<br />rest of the per-cpu buffer to be unreadable. This applies to all events<br />recorded by all applications of the same user id for that particular CPU.<br />Recovering from this involves destroying and re-creating the tracing session.</p>
<p>If this kind of scenario is likely for you, you might want to consider using per-<br />pid buffers (lttng enable-channel -u --buffers-pid), which do not suffer from<br />this design limitation. The downside of per-pid buffers is that it allocates<br />more memory on your system for buffering, and adds extra overhead when<br />used with processes that have a short life-time.</p> Babeltrace - Feature #1300 (New): babeltrace2 lacks knowledge of bitwise enum produced by lttng-m...https://bugs.lttng.org/issues/13002021-03-23T17:41:47ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>This is a follow up on <a class="external" href="https://review.lttng.org/c/babeltrace/+/3045">https://review.lttng.org/c/babeltrace/+/3045</a> now that end users are starting to contact us wondering why up-to-date lttng and babeltrace2 show incomplete information for bitwise enums.</p>
<p>I think we should implement a better stop-gap solution while awaiting CTF2. I recommend that we specialize the babeltrace2 CTF input plugin to detect traces produced by the lttng-modules kernel tracer, for specific known (hardcoded) events and fields which have a bitwise enum semantic, and use this hardcoded knowledge to print the correct bitwise enum to the end user rather than "<unknown>".</p>
<p>This should ensure the current producer/consumer tools work well together (show complete information about those bitwise enums) without requiring to modify tons of documentation about the specific flags needed for bt2 to handle lttng traces appropriately. And this would not lead to misleading scenarios where we print an unknown value as a bitwise enum by mistake.</p>
<p>Hardcoded producer detection/event/fields is of course a last resort solution awaiting proper self-description in the upcoming CTF2 spec.</p> Babeltrace - Bug #1299 (Confirmed): babeltrace2 re-uses prior event string rather than expected e...https://bugs.lttng.org/issues/12992021-03-04T18:49:50ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Trying to reproduce the page fault scenario for an openat() event. Note that the behavior of generating a page fault may be toolchain-specific (linker-specific):</p>
<p>Test program "do-pagefault-open":</p>
<p>#include <stdio.h><br />#include <sys/types.h><br />#include <sys/stat.h><br />#include <fcntl.h></p>
<p>const char *str = "/tmp/blah";<br />const char *str2 = "/tmp/blah2";</p>
<p>int main()
{<br /> open(str, O_RDWR);<br /> perror("first");<br /> open(str2, O_RDWR);<br /> perror("second");<br /> return 0;<br />}</p>
<p>then trace with:</p>
<p>lttng-trace ./do-pagefault-open</p>
<p>Relevant events with babeltrace2 (grepping for 'openat'):</p>
<p>[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296960451] (+0.000006223) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }</p>
<p>Same trace with babeltrace 1:</p>
<p>[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 5 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 3 in enum.<br />[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }</p>
<p>Notice how filename = "/lib/x86_64-linux-gnu/libc.so.6" appears twice in the bt2 output, where we would expect the empty string (as provided by bt1).</p>
<p>Tested with babeltrace2 commit c429f86d0.</p> LTTng - Feature #1288 (New): Combine all per-cpu shm areas for a channel into a single filehttps://bugs.lttng.org/issues/12882020-10-13T15:39:03ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>On the reason for having one shm file per cpu (per channel/per uid/per session): having one file per cpu is mainly done to facilitate NUMA-local memory allocation. That being said, I wonder if as a future improvement we could combine all per-cpu buffers into a single shm file, and issue numa_set_preferred piecewise when mmapping regions of the files. This would lessen the number of file descriptors needed by a significant amount. I suspect it would work, but we'd need to do some prototyping to validate this approach first.</p> LTTng-tools - Bug #1256 (New): Check babeltrace1 stderr output in testshttps://bugs.lttng.org/issues/12562020-04-08T19:23:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Currently, babeltrace1 stderr's output contains warnings which are not checked by the tests. Those do not fail the overall babeltrace execution, but end up being unexpected regressions when they appear between releases.</p>
<p>We could modify all babeltrace invocations to add a helper which saves the stderr output, and check that it is empty after babeltrace has completed execution. If non-empty, a test error would be reported.</p> Babeltrace - Bug #1221 (New): Babeltrace 2 sometimes fail on live trace with lttng-clear featurehttps://bugs.lttng.org/issues/12212020-02-17T20:52:19ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Babeltrace 2 sometimes fail on live trace with lttng-clear feature</p>
<p>This may happen in a scenario where per-pid buffers are used, an application has just quit, and babeltrace 2 observes a stream in make_viewer_streams, but then fails to open the associated index or data file on the first get_next_index because it has been unlinked by clear.</p>
<p>The babeltrace output is:</p>
<pre>
12-04 14:01:48.455 11884 11884 E PLUGIN/CTF/MSG-ITER create_msg_stream_end@msg-iter.c:2464 [lttng-live] Cannot create stream for stream message: notit-addr=0x55b89f957980
12-04 14:01:48.455 11884 11884 W LIB/MSG-ITER bt_self_component_port_input_message_iterator_next@iterator.c:906 Component input port message iterator's "next" method failed: iter-addr=0x55b89f93f510, iter-upstream-comp-name="lttng-live", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE, iter-upstream-comp-class-name="lttng-live", iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER muxer_upstream_msg_iter_next@muxer.c:444 [muxer] Error or unsupported status code: status-code=-1
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER validate_muxer_upstream_msg_iters@muxer.c:975 [muxer] Cannot validate muxer's upstream message iterator wrapper: muxer-msg-iter-addr=0x55b89f950950, muxer-upstream-msg-iter-wrap-addr=0x55b89f9415d0
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER muxer_msg_iter_next@muxer.c:1371 [muxer] Cannot get next message: comp-addr=0x55b89f93ebc0, muxer-comp-addr=0x55b89f93ec40, muxer-msg-iter-addr=0x55b89f950950, msg-iter-addr=0x55b89f93f3a0, status=ERROR
12-04 14:01:48.455 11884 11884 W LIB/MSG-ITER bt_self_component_port_input_message_iterator_next@iterator.c:906 Component input port message iterator's "next" method failed: iter-addr=0x55b89f93f3a0, iter-upstream-comp-name="muxer", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER, iter-upstream-comp-class-name="muxer", iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
12-04 14:01:48.455 11884 11884 W LIB/GRAPH consume_graph_sink@graph.c:596 Component's "consume" method failed: status=ERROR, comp-addr=0x55b89f93ed30, comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK, comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages (`text` fo", comp-class-is-frozen=0, comp-class-so-handle-addr=0x55b89f94c740, comp-class-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-text.so", comp-input-port-count=1, comp-output-port-count=0
12-04 14:01:48.456 11884 11884 E CLI cmd_run@babeltrace2.c:2574 Graph failed to complete successfully
ERROR: [Babeltrace CLI] (babeltrace2.c:2574)
Graph failed to complete successfully
CAUSED BY [Babeltrace library] (graph.c:596)
Component's "consume" method failed: status=ERROR, comp-addr=0x55b89f93ed30,
comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK,
comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages
(`text` fo", comp-class-is-frozen=0, comp-class-so-handle-addr=0x55b89f94c740,
comp-class-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-text.so",
comp-input-port-count=1, comp-output-port-count=0
CAUSED BY [Babeltrace library] (iterator.c:906)
Component input port message iterator's "next" method failed:
iter-addr=0x55b89f93f3a0, iter-upstream-comp-name="muxer",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER,
iter-upstream-comp-class-name="muxer",
iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
CAUSED BY [Babeltrace library] (iterator.c:906)
Component input port message iterator's "next" method failed:
iter-addr=0x55b89f93f510, iter-upstream-comp-name="lttng-live",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE,
iter-upstream-comp-class-name="lttng-live",
iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
</pre>
<p>notit->stream is expected not to be NULL when reaching the state STATE_EMIT_MSG_STREAM_END, but if no packet was received, it is indeed still NULL, because it is the packet header being read that moves to the STATE_CHECK_EMIT_MSG_STREAM_BEGINNING state (after_packet_context_state()).</p> LTTng-modules - Bug #975 (Confirmed): execve compat syscall exit syscall value issuehttps://bugs.lttng.org/issues/9752015-11-08T16:03:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>syscall_exit for execve changing the 32/64-bit compat mode for a process has wrong system call number on exit:</p>
<pre>
[19:36:57.188066018] (+0.000000616) sinkpad syscall_entry_execve: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", argv = 0x7FFD275BBF40, envp = 0x7FFD275BBF50 }
[19:36:57.188162705] (+0.000000851) sinkpad module_get: { cpu_id = 0 }, { ip = 18446744071581118857, refcnt = 3, name = "binfmt_misc" }
[19:36:57.188170506] (+0.000000517) sinkpad module_put: { cpu_id = 0 }, { ip = 18446744071581118912, refcnt = 2, name = "binfmt_misc" }
[19:36:57.188630250] (+0.000000911) sinkpad random_get_random_bytes: { cpu_id = 0 }, { nbytes = 16, IP = 18446744071581461559 }
[19:36:57.188630781] (+0.000000531) sinkpad random_extract_entropy: { cpu_id = 0 }, { pool_name = "nonblocking", nbytes = 16, entropy_count = 0, IP = 18446744071583984742 }
[19:36:57.188634640] (+0.000001773) sinkpad sched_waking: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188637855] (+0.000003215) sinkpad sched_stat_sleep: { cpu_id = 0 }, { comm = "rngd", tid = 2177, delay = 2656341 }
[19:36:57.188639681] (+0.000001826) sinkpad sched_wakeup: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188641483] (+0.000000180) sinkpad power_cpu_idle: { cpu_id = 2 }, { state = 4294967295, cpu_id = 2 }
[19:36:57.188646080] (+0.000000644) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649456] (+0.000000788) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649704] (+0.000000248) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188650149] (+0.000000182) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188652253] (+0.000001864) sinkpad sched_stat_wait: { cpu_id = 2 }, { comm = "rngd", tid = 2177, delay = 0 }
[19:36:57.188654080] (+0.000001827) sinkpad sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "rngd", next_tid = 2177, next_prio = 20 }
[19:36:57.188658382] (+0.000000567) sinkpad sched_process_exec: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", tid = 29058, old_tid = 29058 }
[19:36:57.188661040] (+0.000000020) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188661415] (+0.000000375) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188662327] (+0.000000409) sinkpad sched_stat_runtime: { cpu_id = 2 }, { comm = "rngd", tid = 2177, runtime = 25827, vruntime = 1365908673 }
[19:36:57.188664216] (+0.000001266) sinkpad compat_syscall_exit_olduname: { cpu_id = 0 }, { ret = 0, name = 0 }
</pre> LTTng-modules - Feature #964 (New): Implement support for persistent memory buffershttps://bugs.lttng.org/issues/9642015-10-22T20:41:43ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #962 (New): add x86 exceptions.h and irq_vectors.h instrumentation (and m...https://bugs.lttng.org/issues/9622015-10-22T19:47:19ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-UST - Feature #527 (New): Add git version information to project versionhttps://bugs.lttng.org/issues/5272013-05-10T12:54:46ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-UST - Feature #520 (New): Allow override of /var/run directoryhttps://bugs.lttng.org/issues/5202013-05-04T13:59:36ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Allow override of /var/run directory, as will as per-user $HOME/.lttng/ directory at configure time, and by environment variables. This should match a similar feature in lttng-tools.</p> LTTng-UST - Feature #327 (On pause): Implement missing hostname contexthttps://bugs.lttng.org/issues/3272012-08-26T23:22:47ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>To match features of lttng-modules.</p>