LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912020-10-12T19:16:13ZLTTng bugs repository
Redmine LTTng-UST - Bug #1286 (Resolved): session daemon should validate credentials received from applic...https://bugs.lttng.org/issues/12862020-10-12T19:16:13ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Looking at ustctl_recv_reg_msg() , I notice that the session daemon fails to validate the pid and uid credentials it receives from the application, thus trusting them blindly. This means a non-root application could theoretically impersonate a root application from a tracing perspective, and thus access root tracing buffers in a per-uid configuration, which is unwanted. I remember that initially we had no validation of the pid provided by the application because original lttng 2.0 only supported per-pid buffers and had per application tracing buffers only, so it did not cause any issue other than mislabeling the trace directory. However, now that the buffers can be shared between processes belonging to the same uid, this needs to be validated by the session daemon, and it's not.</p>
<p>So the quick fix here would be to validate on the session daemon side that the credentials provided by the application match those from a sessiond perspective through unix socket credentials (getsockopt(2) SO_PEERCRED on Linux and LOCAL_PEERCRED on BSD). That would however mean that sessiond would refuse applications that come from separate namespaces if the credentials don't match.</p>
<p>Tweaking liblttng-ust-comm/lttng-ust-comm.c:ustcomm_send_reg_msg() to send dummy credentials shows that the session daemon indeed trusts the application blindly.</p> LTTng - Feature #968 (Feedback): lttng-modules kernel and user callstack contexthttps://bugs.lttng.org/issues/9682015-10-25T20:14:40ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Implementation from Francis Giraldeau reviewed and cleaned up available here:</p>
<p><a class="external" href="https://github.com/compudj/lttng-tools-dev/tree/callstack">https://github.com/compudj/lttng-tools-dev/tree/callstack</a><br /><a class="external" href="https://github.com/compudj/lttng-modules-dev/tree/callstack">https://github.com/compudj/lttng-modules-dev/tree/callstack</a></p>
<p>Documentation and tests are missing.</p> LTTng - Feature #966 (Resolved): Merge Java early filtering featurehttps://bugs.lttng.org/issues/9662015-10-22T20:47:44ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>UST: <a class="external" href="https://github.com/alexmonthy/lttng-ust/commits/java-filter-notifications">https://github.com/alexmonthy/lttng-ust/commits/java-filter-notifications</a></p>
<p>Tools: <a class="external" href="https://github.com/alexmonthy/lttng-tools/tree/java-filter-notifications">https://github.com/alexmonthy/lttng-tools/tree/java-filter-notifications</a></p>
<p>Needs to be merged in both projects in locked-step.</p> LTTng-UST - Feature #965 (New): Implement UST statedumphttps://bugs.lttng.org/issues/9652015-10-22T20:43:37ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Initial implementation: <a class="external" href="https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier">https://github.com/compudj/lttng-ust-dev/tree/statedump-notifier</a></p>
<p>Missing tests in lttng-tools for this feature before we can merge it into lttng-ust.</p> LTTng-modules - Feature #964 (New): Implement support for persistent memory buffershttps://bugs.lttng.org/issues/9642015-10-22T20:41:43ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-modules - Feature #963 (New): Implement user-space stack dump contexthttps://bugs.lttng.org/issues/9632015-10-22T20:41:10ZMathieu Desnoyersmathieu.desnoyers@efficios.comLTTng-tools - Bug #861 (Resolved): UST subbuffers silently dropped on moderate trace traffichttps://bugs.lttng.org/issues/8612014-11-18T16:31:15ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>lttng-tools commit 02b3d1769d5f8a33e4109b1e681141c9295dfda6 introduced an important regression for lttng-ust tracing in the consumer daemon: after reading a sub-buffer, a check has been added to see whether there are more sub-buffers available to read, and if it is the case, it ensures the wakeup pipe will be awakened again.</p>
<p>The issue lies in the use of ustctl_put_next_subbuf() in this check. This acts as if the sub-buffer has been read, when in reality it has not been read. It therefore trashes the data contained by this sub-buffer.</p>
<p>This check should use ustctl_put_subbuf(), which does not move the consumer position.</p> LTTng-tools - Bug #773 (Resolved): LTTng JUL JNI handler does not distinguish between sessionshttps://bugs.lttng.org/issues/7732014-03-25T19:53:30ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>If, for a given e.g. root session daemon, we start 2 sessions (sessions A and B), each with JUL events (targeting different loggers), those events will be logged into both sessions, rather than having the events enabled for session A into session A, and likewise for B.</p>
<p>We should filter by logger name at the JUL JNI interface at UST level to fix this issue. This fix should not require changes to UST, only changes to session daemon.</p> LTTng-UST - Bug #772 (Resolved): LTTng JUL Agent should clear state when sessiond disconnectshttps://bugs.lttng.org/issues/7722014-03-25T19:49:53ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>LTTngTCPSessiondClient.java<br /> } catch (IOException ioe) {<br /> tryReleaseSem();<br /> Thread.sleep(3000);</p>
<p>We should clear the enabled loggers for this thread when the<br />sessiond disconnects. Failure to do so will cause an<br />inconsistency between the JUL state and the state of a<br />new sessiond that would be started afterward.</p> LTTng-UST - Bug #745 (Resolved): base address dump triggers deadlockshttps://bugs.lttng.org/issues/7452014-02-27T21:24:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>It appears that the daemon() test case of lttng-tools deadlocks when base address is dumped.<br />(lttng-tools tests/regression/ust/daemon)</p>
<p>From preliminary analysis, it seems to be a hang in the child process, likely caused by use of the dynamic linker lock within a lttng-ust thread without protection of ust_lock().</p>
<p>We'll need to figure out a way to rework locking to make use of the dynamic linked safe wrt fork/clone/daemon. It's possible that we need to disable the base address feature for 2.4, as it is late in the rc cycle.</p> LTTng-UST - Bug #695 (Resolved): deadlock with baddr and JUL tracinghttps://bugs.lttng.org/issues/6952013-11-24T08:17:52ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p><a class="external" href="http://lists.lttng.org/pipermail/lttng-dev/2013-November/021949.html">http://lists.lttng.org/pipermail/lttng-dev/2013-November/021949.html</a></p>
<p>Appeared in 2.4.0-rc1.</p> LTTng-modules - Bug #631 (Resolved): hard lockup with lttng-modules and kernel 3.10 +https://bugs.lttng.org/issues/6312013-09-11T16:13:45ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Starting from Linux kernel commit<br />06c017fdd4dc48451a29ac37fc1db4a3f86b7f40 "timekeeping: Hold<br />timekeepering locks in do_adjtimex and hardpps" (3.10 kernels), the<br />xtime write seqlock is held across calls to __do_adjtimex(), which<br />includes a call to notify_cmos_timer(), and hence<br />schedule_delayed_work().</p>
<p>This introduces a side-effect for a set of tracepoints, including mainly<br />the workqueue tracepoints: a tracer hooking on those tracepoints and<br />reading current time with ktime_get() will cause hard system LOCKUP such<br />as:</p>
<pre>
WARNING: CPU: 6 PID: 2258 at kernel/watchdog.c:245 watchdog_overflow_callback+0x93/0x9e()
Watchdog detected hard LOCKUP on cpu 6
Modules linked in: lttng_probe_workqueue(O) lttng_probe_vmscan(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_s]
CPU: 6 PID: 2258 Comm: ntpd Tainted: G O 3.11.0 #158
Hardware name: Supermicro X7DAL/X7DAL, BIOS 6.00 12/03/2007
0000000000000000 ffffffff814f83eb ffffffff813b206a ffff88042fd87c78
ffffffff8106a07c 0000000000000000 ffffffff810c94c2 0000000000000000
ffff88041f31bc00 0000000000000000 ffff88042fd87d68 ffff88042fd87ef8
Call Trace:
<NMI> [<ffffffff813b206a>] ? dump_stack+0x41/0x51
[<ffffffff8106a07c>] ? warn_slowpath_common+0x79/0x92
[<ffffffff810c94c2>] ? watchdog_overflow_callback+0x93/0x9e
[<ffffffff8106a12d>] ? warn_slowpath_fmt+0x45/0x4a
[<ffffffff810c94c2>] ? watchdog_overflow_callback+0x93/0x9e
[<ffffffff810c942f>] ? watchdog_enable_all_cpus.part.2+0x31/0x31
[<ffffffff810ecc66>] ? __perf_event_overflow+0x12c/0x1ae
[<ffffffff810eab60>] ? perf_event_update_userpage+0x13/0xc2
[<ffffffff81016820>] ? intel_pmu_handle_irq+0x26a/0x2fd
[<ffffffff813b7a0b>] ? perf_event_nmi_handler+0x24/0x3d
[<ffffffff813b728f>] ? nmi_handle.isra.3+0x58/0x12f
[<ffffffff813b7a59>] ? perf_ibs_nmi_handler+0x35/0x35
[<ffffffff813b7404>] ? do_nmi+0x9e/0x2bc
[<ffffffff813b6af7>] ? end_repeat_nmi+0x1e/0x2e
[<ffffffff810a2a33>] ? read_seqcount_begin.constprop.4+0x8/0xf
[<ffffffff810a2a33>] ? read_seqcount_begin.constprop.4+0x8/0xf
[<ffffffff810a2a33>] ? read_seqcount_begin.constprop.4+0x8/0xf
<<EOE>> [<ffffffff810a2d6c>] ? ktime_get+0x23/0x5e
[<ffffffffa0314670>] ? lib_ring_buffer_clock_read.isra.28+0x1f/0x21 [lttng_ring_buffer_client_discard]
[<ffffffffa0314786>] ? lttng_event_reserve+0x112/0x3f3 [lttng_ring_buffer_client_discard]
[<ffffffffa045b1c5>] ? __event_probe__workqueue_queue_work+0x72/0xe0 [lttng_probe_workqueue]
[<ffffffff812ef7e9>] ? sock_aio_read.part.10+0x110/0x124
[<ffffffff81133a36>] ? do_sync_readv_writev+0x50/0x76
[<ffffffff8107d514>] ? __queue_work+0x1ab/0x265
[<ffffffff8107da7e>] ? queue_delayed_work_on+0x3f/0x4e
[<ffffffff810a473d>] ? __do_adjtimex+0x408/0x413
[<ffffffff810a3e9a>] ? do_adjtimex+0x98/0xee
[<ffffffff8106cec6>] ? SYSC_adjtimex+0x32/0x5d
[<ffffffff813bb74b>] ? tracesys+0xdd/0xe2
</pre> LTTng-tools - Bug #582 (Resolved): lttng-sessiond asserts on multi-channel UST for pid-uid channelshttps://bugs.lttng.org/issues/5822013-07-01T16:13:25ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Error:<br />lttng-sessiond: ust-app.c:4520: reply_ust_register_channel: Assertion `chan_reg' failed.<br />Aborted</p>
<p>How to reproduce:</p>
<p>lttng create<br />lttng enable-channel mychan1 -u --buffers-uid <br />lttng enable-channel mychan2 -u --buffers-uid <br />lttng enable-event -u -a -c mychan1<br />lttng enable-event -u -a -c mychan2<br />lttng start</p>
<p>use UST hello test program, run twice:</p>
<p>./hello<br />./hello</p>
<p>At the second execution of "hello", we get the assert in sessiond. It's possibly a mixup in the channel IDs given to per-UID buffers within sessiond.</p> LTTng-UST - Bug #498 (Resolved): Can not successfully launch instrumented app (that are compiled ...https://bugs.lttng.org/issues/4982013-04-10T00:56:40ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<pre>
Problem Description:
====================
* When launching an instrumented app (which is compiled with lttng 2.1) on a
system that has lttng latest origin/master loaded (and an active session
is present), then the instrumented app either segfault or gives the following
error message:
"error while loading shared libraries: liburcu-bp.so.1: cannot open shared
object file: No such file or directory"
The instrumented app segfault when the following commits were used:
userspace : 8a97780 (HEAD, origin/stable-0.7) Fix: tests/api.h use cpuset.h
lttng-tools : 8f3f773 (HEAD, origin/master, origin/HEAD, globalBuffer) Fix: lttng UI ...
lttng-ust : d3db6a4 (HEAD, origin/master, origin/HEAD) Performance: add unlikely ...
The instrumented app fails to launch with error "error while loading shared libraries:
liburcu-bp.so.1: cannot open shared object file: No such file or directory"
when the following commit were used:
userspace : d107390 (HEAD, origin/master, origin/HEAD, globalBuffer) Add tab to output ...
lttng-tools : e2ad3b4 (HEAD, origin/master, origin/HEAD, globalBuffer) Add demo README
lttng-ust : 9f9ee9c (HEAD, origin/master, origin/HEAD, globalBuffer) Fix: UST context activation
Is problem reproducible ?
=========================
Yes
How to reproduce (if reproducible):
===================================
Scenario1 (when usersapce 0.7 is used)
0)_ Use the following commits:
userspace : 8a97780 (HEAD, origin/stable-0.7) Fix: tests/api.h use cpuset.h
lttng-tools : 8f3f773 (HEAD, origin/master, origin/HEAD, globalBuffer) Fix: lttng UI ...
lttng-ust : d3db6a4 (HEAD, origin/master, origin/HEAD) Performance: add unlikely ...
1)_ Compile an instrumented app with lttng 2.1 library
2)_ Load lttng from origin/master branch
3)_ Launch the instr app (from step-1)
lttng list -u #-- shows that the app is registered to sessionD
4)_ lttng create s
lttng enable-event -u "com*"
lttng start
5)_ The app will segfault as soon as the session is started.
6)_ Relaunch the app again
The app will segfault right away.
Scenario2 (when latest origin/master userspace is used)
0)_ Use the following commits:
userspace : d107390 (HEAD, origin/master, origin/HEAD, globalBuffer) Add tab to output ...
lttng-tools : e2ad3b4 (HEAD, origin/master, origin/HEAD, globalBuffer) Add demo README
lttng-ust : 9f9ee9c (HEAD, origin/master, origin/HEAD, globalBuffer) Fix: UST context activation
1)_ Compile an instrumented app with lttng 2.1 library
2)_ Load lttng from origin/master branch
3)_ Launch the instr app (from step-1)
The app fail to launch with the following error:
"error while loading shared libraries: liburcu-bp.so.1: cannot
open shared object file: No such file or directory"
Commit used:
============
Please see description above.
</pre> LTTng-UST - Bug #10 (Resolved): tor instrumentation only works partlyhttps://bugs.lttng.org/issues/102012-02-10T01:25:58ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>When instrumenting tor, only some of the tracepoints are fired, event though all events are enabled.</p>
<p>(skipping bug reproduction details, since the bug is already solved)</p>
<p>Solved by commit:</p>
<p>commit 51920067af7b1049413c1b8c30ee254afbd4e448<br />Author: Mathieu Desnoyers <<a class="email" href="mailto:mathieu.desnoyers@efficios.com">mathieu.desnoyers@efficios.com</a>><br />Date: Thu Feb 9 18:55:44 2012 -0500</p>
<pre><code>Fix tracepoint.h multiple .o within module/core exec linkage bug</code></pre>
<pre><code>We need all symbols looked up with dlopen to share the same linkage<br /> property as the __tracepoint_registered variable (shared across .o in a<br /> module/executable), otherwise only the first .o which runs its<br /> constructor will have those defined.</code></pre>
<pre><code>Caused some tracepoints not to be traced in non-_LGPL_SOURCE<br /> applications, due to the check:</code></pre>
<pre><code>if (!TP_RCU_LINK_TEST()) \<br /> return;</code></pre>
<pre><code>Signed-off-by: Mathieu Desnoyers &lt;<a class="email" href="mailto:mathieu.desnoyers@efficios.com">mathieu.desnoyers@efficios.com</a>&gt;</code></pre>