LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912023-06-01T18:48:58ZLTTng bugs repository
Redmine LTTng-tools - Bug #1379 (New): Document behavior of live mode with per-pid UST buffershttps://bugs.lttng.org/issues/13792023-06-01T18:48:58ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>There is a design limitation of per-pid UST buffers when used in live mode which should be documented.</p>
<p>Basically, because the per-pid buffers are reclaimed soon after the application exits, it may cause the events traced by a short-lived application to never appear in the output of a live trace when per-pid buffers are used.</p>
<p>This is caused by the fact that the live client periodically checks for new buffers, but there is no inherent reference kept on the streams before they disappear.</p>
<p>This means that the information will be available in the disk output of the trace, but it will be missing from the live mode output.</p>
<p>This could eventually be mitigated by keeping references to the streams received by the relay daemon which are of interest to a live client until they are referenced by the live client.</p> LTTng-tools - Feature #1341 (New): Introduce "--shm-path-ust" to clarify use vs documentationhttps://bugs.lttng.org/issues/13412021-12-07T20:53:52ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Considering that the "--shm-path" argument is documented as applying to all domains, but that currently the kernel domain does not support this argument, it leads us to a situation where users may budget their NVRAM for the space needed only for UST buffers, and eventually upgrade to a new LTTng version which would implement kernel support for shm-path. They would then face issues given the lack of available space.</p>
<p>I would recommend to add a new "--shm-path-ust" argument as an alias to "--shm-path", but document that it only applies to the UST domain. This way, users wishing to budget their NVRAM space only for UST tracing, but still perform kernel tracing in the same session, will be able to do so.</p> Babeltrace - Bug #1299 (Confirmed): babeltrace2 re-uses prior event string rather than expected e...https://bugs.lttng.org/issues/12992021-03-04T18:49:50ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Trying to reproduce the page fault scenario for an openat() event. Note that the behavior of generating a page fault may be toolchain-specific (linker-specific):</p>
<p>Test program "do-pagefault-open":</p>
<p>#include <stdio.h><br />#include <sys/types.h><br />#include <sys/stat.h><br />#include <fcntl.h></p>
<p>const char *str = "/tmp/blah";<br />const char *str2 = "/tmp/blah2";</p>
<p>int main()
{<br /> open(str, O_RDWR);<br /> perror("first");<br /> open(str2, O_RDWR);<br /> perror("second");<br /> return 0;<br />}</p>
<p>then trace with:</p>
<p>lttng-trace ./do-pagefault-open</p>
<p>Relevant events with babeltrace2 (grepping for 'openat'):</p>
<p>[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296960451] (+0.000006223) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }</p>
<p>Same trace with babeltrace 1:</p>
<p>[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 5 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 3 in enum.<br />[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }</p>
<p>Notice how filename = "/lib/x86_64-linux-gnu/libc.so.6" appears twice in the bt2 output, where we would expect the empty string (as provided by bt1).</p>
<p>Tested with babeltrace2 commit c429f86d0.</p> LTTng-tools - Feature #1287 (New): Use abstract sockets for lttng-consumerd UST shared memory fileshttps://bugs.lttng.org/issues/12872020-10-13T15:35:32ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Abstract sockets (unix(7)) are not tied to the filesystem, and are available since Linux 2.2.</p>
<p>Those are Linux-specific.</p>
<p>Those are the same as regular unix domain but their first character of path is NULL. They have the benefit of not requiring unlinking of files left behind.</p>
<p>We could use those abstract sockets in lttng-consumerd on Linux.</p> LTTng-tools - Bug #987 (Confirmed): Incomplete string comparison patternhttps://bugs.lttng.org/issues/9872016-01-06T23:23:31ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>In a few instances in lttng-tools, a string comparison pattern involving strncmp and strlen() to limit the length of the comparison is used.</p>
<p>One can quickly find them with:</p>
<p>grep -r -A 3 strncmp . | less -> search for strlen or strnlen</p>
<p>Many of those are OK: they indeed only want to match the beginning of the strings. However, this pattern is incorrectly used in cases where a full match is required, for instance:</p>
<p>ust-registry.c:ht_match_event()<br />./src/bin/lttng/commands/enable_channels.c: if (!strncmp(output_mmap, opt_output, strlen(output_mmap))) {<br />./src/bin/lttng/commands/enable_channels.c: } else if (!strncmp(output_splice, opt_output, strlen(output_splice))) {<br />./src/bin/lttng-sessiond/snapshot.c: if (!strncmp(output->name, name, strlen(name))) {</p> LTTng-modules - Bug #975 (Confirmed): execve compat syscall exit syscall value issuehttps://bugs.lttng.org/issues/9752015-11-08T16:03:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>syscall_exit for execve changing the 32/64-bit compat mode for a process has wrong system call number on exit:</p>
<pre>
[19:36:57.188066018] (+0.000000616) sinkpad syscall_entry_execve: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", argv = 0x7FFD275BBF40, envp = 0x7FFD275BBF50 }
[19:36:57.188162705] (+0.000000851) sinkpad module_get: { cpu_id = 0 }, { ip = 18446744071581118857, refcnt = 3, name = "binfmt_misc" }
[19:36:57.188170506] (+0.000000517) sinkpad module_put: { cpu_id = 0 }, { ip = 18446744071581118912, refcnt = 2, name = "binfmt_misc" }
[19:36:57.188630250] (+0.000000911) sinkpad random_get_random_bytes: { cpu_id = 0 }, { nbytes = 16, IP = 18446744071581461559 }
[19:36:57.188630781] (+0.000000531) sinkpad random_extract_entropy: { cpu_id = 0 }, { pool_name = "nonblocking", nbytes = 16, entropy_count = 0, IP = 18446744071583984742 }
[19:36:57.188634640] (+0.000001773) sinkpad sched_waking: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188637855] (+0.000003215) sinkpad sched_stat_sleep: { cpu_id = 0 }, { comm = "rngd", tid = 2177, delay = 2656341 }
[19:36:57.188639681] (+0.000001826) sinkpad sched_wakeup: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188641483] (+0.000000180) sinkpad power_cpu_idle: { cpu_id = 2 }, { state = 4294967295, cpu_id = 2 }
[19:36:57.188646080] (+0.000000644) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649456] (+0.000000788) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649704] (+0.000000248) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188650149] (+0.000000182) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188652253] (+0.000001864) sinkpad sched_stat_wait: { cpu_id = 2 }, { comm = "rngd", tid = 2177, delay = 0 }
[19:36:57.188654080] (+0.000001827) sinkpad sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "rngd", next_tid = 2177, next_prio = 20 }
[19:36:57.188658382] (+0.000000567) sinkpad sched_process_exec: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", tid = 29058, old_tid = 29058 }
[19:36:57.188661040] (+0.000000020) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188661415] (+0.000000375) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188662327] (+0.000000409) sinkpad sched_stat_runtime: { cpu_id = 2 }, { comm = "rngd", tid = 2177, runtime = 25827, vruntime = 1365908673 }
[19:36:57.188664216] (+0.000001266) sinkpad compat_syscall_exit_olduname: { cpu_id = 0 }, { ret = 0, name = 0 }
</pre> LTTng - Feature #968 (Feedback): lttng-modules kernel and user callstack contexthttps://bugs.lttng.org/issues/9682015-10-25T20:14:40ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Implementation from Francis Giraldeau reviewed and cleaned up available here:</p>
<p><a class="external" href="https://github.com/compudj/lttng-tools-dev/tree/callstack">https://github.com/compudj/lttng-tools-dev/tree/callstack</a><br /><a class="external" href="https://github.com/compudj/lttng-modules-dev/tree/callstack">https://github.com/compudj/lttng-modules-dev/tree/callstack</a></p>
<p>Documentation and tests are missing.</p> LTTng-UST - Feature #965 (New): Implement UST statedumphttps://bugs.lttng.org/issues/9652015-10-22T20:43:37ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Initial implementation: <a class="external" href="https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier">https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier</a></p>
<p>Missing tests in lttng-tools for this feature before we can merge it into lttng-ust.</p> LTTng-modules - Feature #964 (New): Implement support for persistent memory buffershttps://bugs.lttng.org/issues/9642015-10-22T20:41:43ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #963 (New): Implement user-space stack dump contexthttps://bugs.lttng.org/issues/9632015-10-22T20:41:10ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #962 (New): add x86 exceptions.h and irq_vectors.h instrumentation (and m...https://bugs.lttng.org/issues/9622015-10-22T19:47:19ZMathieu Desnoyersmathieu.desnoyers@efficios.comUserspace RCU - Feature #940 (New): Wire up sys membarrier on each architecturehttps://bugs.lttng.org/issues/9402015-09-26T16:00:41ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-tools - Bug #833 (Confirmed): memcpy of non-packed struct into packed struct (possible layo...https://bugs.lttng.org/issues/8332014-09-09T13:17:10ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>lttng-tools src/lib/lttng-ctl/lttng-ctl.c:</p>
<p>lttng_enable_event_with_exclusions()</p>
<p>memcpy(&lsm.u.enable.event, ev, sizeof(lsm.u.enable.event));</p>
<p>copy "ev" (non-packed) into a packed structure.</p>
<p>We should copy each field one by one (create a copy_event_to_event_packed() helper to do so).</p> LTTng-UST - Bug #525 (Confirmed): new "notifications" from UST do not strictly respect LTTNG_UST_...https://bugs.lttng.org/issues/5252013-05-07T15:25:54ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should eventually find a way to improve notifications from UST to sessiond so they don't delay .so loading (and thus application startup) when env. var. specify a LTTNG_UST_REGISTER_TIMEOUT=0. Currently, we work around this issue by setting the notification socket with a timeout of 100ms as minimum timeout if the LTTNG_UST_REGISTER_TIMEOUT value is below 100.</p>
<p>A cleaner fix could involve handling these notifications from a (possibly new) separate thread, and use a semaphore-based scheme to handle optional wait from the application.</p> LTTng-UST - Feature #327 (On pause): Implement missing hostname contexthttps://bugs.lttng.org/issues/3272012-08-26T23:22:47ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>To match features of lttng-modules.</p>