LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912023-04-25T11:44:50ZLTTng bugs repository
Redmine LTTng-tools - Bug #1372 (In Progress): Consumer crashes during rotation (write to bad file descri...https://bugs.lttng.org/issues/13722023-04-25T11:44:50ZMaximilian Fickert
<p>Hi,</p>
<p>We are observing seemingly random crashes in the LTTng consumer daemon when tracing a C++ application with LTTng-UST. Our workload has a single <code>printf</code>-like tracepoint, where each string is in the order of 1kb and the total output is around 30MB/s. LTTng is set up with a single session and channel enabling this tracepoint, and we enabled rotation with a maximum size of 100MB or every 30 seconds. We are periodically starting new traced processes and the system runs close to 100% CPU load. This ran on an AWS Graviton2 (ARM) instance with CentOS 7 and a 5.4 kernel, using LTTng-UST 2.13.5 and LTTng-tools 2.13.8.</p>
<p>The first reported error is a write to a bad file descriptor (-1), apparently when waking up the metadata poll thread during a rotation.</p>
<p>I looked through the LTTng logs with <code>-vvv --verbose-consumer</code> and I suspect this might be caused by some sort of race condition; looking at the following log fragments (I inserted two additional debug lines in LTTng so the line numbers may not exactly match those in the 2.13.8 source code):<br /><pre>
DBG1 - 15:12:12.865621802 [Rotation]: Consumer rotate channel key 574 (in consumer_rotate_channel() at consumer.c:1694)
</pre><br />A rotation is requested for the channel with key 574. Then the metadata pipe of this channel is closed:<br /><pre>
DBG3 - 15:12:13.269805401 [UST application management]: Buffer registry per PID find id: 288 (in buffer_reg_pid_find() at buffer-registry.c:308)
DBG3 - 15:12:13.269811366 [UST application management]: No metadata to push for metadata key 574 (in ust_app_push_metadata() at ust-app.c:689)
DBG2 - 15:12:13.269816979 [UST application management]: Consumer close metadata channel key 574 (in consumer_close_metadata() at consumer.c:1394)
DBG1 - 15:12:13.269826899 [6593/6605]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:3283)
DBG1 - 15:12:13.269835259 [6593/6605]: UST consumer close metadata key 574 (in close_metadata() at ust-consumer.c:804)
DBG1 - 15:12:13.269838591 [6593/6605]: Closing metadata channel key 574 (in lttng_ustconsumer_close_metadata() at ust-consumer.c:3235)
DBG3 - 15:12:13.269841167 [6593/6605]: close() fd = 603 (in lttng_ustconsumer_close_metadata() at ust-consumer.c:3250)
DBG1 - 15:12:13.269847075 [6593/6605]: Received command on sock (in consumer_thread_sessiond_poll() at consumer.c:3299)
</pre><br />But then LTTng unsuccessfully attempts to wake up the metadata poll thread by writing to the file descriptor that was just closed (the <code>channel key = 574</code> in the metadata poll thread message below is <code>channel->key</code>):<br /><pre>
DBG1 - 15:12:13.271001175 [6593/6605]: Waking up metadata poll thread (writing to pipe): channel name = 'metadata', channel key = 574 (in consumer_metadata_wakeup_pipe() at consumer.c:888)
DBG3 - 15:12:13.271010093 [6593/6605]: write() fd = -1 (in consumer_metadata_wakeup_pipe() at consumer.c:892)
PERROR - 15:12:13.271014655 [6593/6605]: Failed to write to UST metadata pipe while attempting to wake-up the metadata poll thread: Bad file descriptor (in consumer_metadata_wakeup_pipe() at consumer.c:907)
Error: Failed to dump the metadata cache
Error: Rotate channel failed
</pre><br />After this, there are a bunch more errors in the logs, but I guess they are ultimately caused by the issue above:<br /><pre>
DBG1 - 15:12:13.273572566 [6593/6604]: Consumer mmap write() ret 446464 (len 446464) (in lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1726)
lttng-consumerd: consumer.c:1639: lttng_consumer_on_read_subbuffer_mmap: Assertion `stream->net_seq_idx != (uint64_t) -1ULL || stream->trace_chunk' failed.
...
DBG1 - 15:12:13.475559588 [Consumer management]: Error when receiving data from the consumer socket 61 (in consumer_socket_recv() at consumer.c:159)
Error: Handling metadata request
DBG1 - 15:12:13.476382551 [Rotation]: Error when receiving data from the consumer socket 60 (in consumer_socket_recv() at consumer.c:159)
Error: Error pushing metadata to consumer
DBG1 - 15:12:13.476418727 [Rotation]: Consumer rotate channel key 604 (in consumer_rotate_channel() at consumer.c:1694)
DBG1 - 15:12:13.476424962 [Rotation]: Setting trace chunk close command to "move to completed chunk folder" (in lttng_trace_chunk_set_close_command() at trace-chunk.c:1797)
DBG1 - 15:12:13.476428671 [Rotation]: lttng_trace_chunk_rename_path from .tmp_new_chunk to (null) (in lttng_trace_chunk_rename_path_no_lock() at trace-chunk.c:755)
DBG3 - 15:12:13.476437516 [Rotation]: renameat() old_dirfd = 946, old_name = .tmp_new_chunk, new_dirfd = 946, new_name = 20230421T151158+0000-5, uid = 1004, gid = 1004 (in run_as_renameat() at runas.c:1883)
DBG1 - 15:12:13.476444999 [Rotation]: Using run_as worker (in run_as() at runas.c:1646)
Error: Error pushing metadata to consumer
DBG2 - 15:12:13.476569797 [UST application management]: Consumer close metadata channel key 624 (in consumer_close_metadata() at consumer.c:1394)
DBG1 - 15:12:13.476585469 [UST application management]: PID 7729 unregistering with sock 1060 (in ust_app_unregister() at ust-app.c:4253)
DBG1 - 15:12:13.476589957 [UST application management]: Flushing app session buffers for ust app pid 7729 (in ust_app_flush_app_session() at ust-app.c:5315)
DBG2 - 15:12:13.476611364 [UST application management]: Consumer flush channel key 651 (in consumer_flush_channel() at consumer.c:1328)
Error: Error flushing consumer channel
DBG3 - 15:12:13.476616188 [UST application management]: Buffer registry per PID find id: 327 (in buffer_reg_pid_find() at buffer-registry.c:308)
DBG3 - 15:12:13.476620406 [UST application management]: No metadata to push for metadata key 652 (in ust_app_push_metadata() at ust-app.c:689)
DBG2 - 15:12:13.476623277 [UST application management]: Consumer close metadata channel key 652 (in consumer_close_metadata() at consumer.c:1394)
DBG1 - 15:12:13.476633312 [UST application management]: PID 11467 unregistering with sock 457 (in ust_app_unregister() at ust-app.c:4253)
DBG1 - 15:12:13.476636996 [UST application management]: Flushing app session buffers for ust app pid 11467 (in ust_app_flush_app_session() at ust-app.c:5315)
DBG2 - 15:12:13.476640508 [UST application management]: Consumer flush channel key 661 (in consumer_flush_channel() at consumer.c:1328)
Error: Error flushing consumer channel
DBG3 - 15:12:13.476643946 [UST application management]: Buffer registry per PID find id: 332 (in buffer_reg_pid_find() at buffer-registry.c:308)
DBG2 - 15:12:13.476649501 [UST application management]: Consumer push metadata to consumer socket -1 (in consumer_push_metadata() at consumer.c:1462)
Error: Error pushing metadata to consumer
...
</pre></p>
<p>We currently do not have simple steps to reliably reproduce the issue, but this happens somewhat irregularly after 5-10 minutes (sometimes earlier, sometimes later) of running our workload. I attached a longer fragment of the logs around the time the error happens, let us know if we can provide any more information to help.</p> Babeltrace - Bug #1299 (Confirmed): babeltrace2 re-uses prior event string rather than expected e...https://bugs.lttng.org/issues/12992021-03-04T18:49:50ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Trying to reproduce the page fault scenario for an openat() event. Note that the behavior of generating a page fault may be toolchain-specific (linker-specific):</p>
<p>Test program "do-pagefault-open":</p>
<p>#include <stdio.h><br />#include <sys/types.h><br />#include <sys/stat.h><br />#include <fcntl.h></p>
<p>const char *str = "/tmp/blah";<br />const char *str2 = "/tmp/blah2";</p>
<p>int main()
{<br /> open(str, O_RDWR);<br /> perror("first");<br /> open(str2, O_RDWR);<br /> perror("second");<br /> return 0;<br />}</p>
<p>then trace with:</p>
<p>lttng-trace ./do-pagefault-open</p>
<p>Relevant events with babeltrace2 (grepping for 'openat'):</p>
<p>[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296960451] (+0.000006223) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }</p>
<p>Same trace with babeltrace 1:</p>
<p>[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 5 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 3 in enum.<br />[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }</p>
<p>Notice how filename = "/lib/x86_64-linux-gnu/libc.so.6" appears twice in the bt2 output, where we would expect the empty string (as provided by bt1).</p>
<p>Tested with babeltrace2 commit c429f86d0.</p> LTTng - Bug #1298 (On pause): LTTng build reprudibility for OE-core (yocto)https://bugs.lttng.org/issues/12982021-03-01T14:52:11ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Currently the installable part of LTTng are "reproducible" for the Debian packages.</p>
<p>Still, oe-core are having issues since they seems to be also shipping the tests for ptest.</p>
<p>See <a class="external" href="https://autobuilder.yocto.io/pub/repro-fail/2021-2-25-rp/lttng-tools/diff/">https://autobuilder.yocto.io/pub/repro-fail/2021-2-25-rp/lttng-tools/diff/</a></p>
<p>As of today this is not a high priority for us. This will be revisited after 2.13 is released.</p> Babeltrace - Bug #1277 (Confirmed): The `ctf` plugin does not support a negative TSDL clock class...https://bugs.lttng.org/issues/12772020-07-23T14:08:36ZSeongab Kim
<p>Hi,</p>
<p>I have a trace which cannot be opened by babeltrace as below, but I can open it with Trace Compass.</p>
<pre>
skim@d54030999178:/mnt/ssd/work/skim/traces$ babeltrace2 ./kernel/
</pre>
<pre>
07-22 07:49:20.099 5264 5264 E PLUGIN/CTF/META/IR-VISITOR get_unary_unsigned@visitor-generate-ir.c:800 [auto-disc-source-ctf-fs] At line 40 in metadata stream: Invalid constant unsigned integer.
07-22 07:49:20.099 5264 5264 E PLUGIN/CTF/META/IR-VISITOR visit_clock_decl_entry@visitor-generate-ir.c:4357 [auto-disc-source-ctf-fs] At line 40 in metadata stream: Unexpected unary expression for clock class's `offset` attribute.
07-22 07:49:20.099 5264 5264 E PLUGIN/CTF/META/IR-VISITOR visit_clock_decl@visitor-generate-ir.c:4532 [auto-disc-source-ctf-fs] At line 40 in metadata stream: Cannot visit clock class's entry: ret=-22
07-22 07:49:20.099 5264 5264 E PLUGIN/CTF/META/IR-VISITOR ctf_visitor_generate_ir_visit_node@visitor-generate-ir.c:4775 [auto-disc-source-ctf-fs] At line 41 in metadata stream: Cannot visit clock class: ret=-22
07-22 07:49:20.099 5264 5264 E PLUGIN/CTF/META/DECODER ctf_metadata_decoder_append_content@decoder.c:337 [auto-disc-source-ctf-fs] Failed to visit AST node to create CTF IR objects: mdec-addr=0x22a0d90, ret=-22
07-22 07:49:20.099 5264 5264 E PLUGIN/SRC.CTF.FS/META ctf_fs_metadata_set_trace_class@metadata.c:128 [auto-disc-source-ctf-fs] Cannot update metadata decoder's content.
07-22 07:49:20.122 5264 5264 E PLUGIN/SRC.CTF.FS ctf_fs_component_create_ctf_fs_trace_one_path@fs.c:1206 [auto-disc-source-ctf-fs] Cannot create trace for `/mnt/ssd/work/skim/traces/kernel`.
07-22 07:49:20.123 5264 5264 W LIB/GRAPH add_component_with_init_method_data@graph.c:977 Component initialization method failed: status=ERROR, comp-addr=0x22a68d0, comp-name="auto-disc-source-ctf-fs", comp-log-level=WARNING, comp-class-type=SOURCE, comp-class-name="fs", comp-class-partial-descr="Read CTF traces from the file sy", comp-class-is-frozen=0, comp-class-so-handle-addr=0x22b0de0, comp-class-so-handle-path="/usr/lib/x86_64-linux-gnu/babeltrace2/plugins/babeltrace-plugin-ctf.so", comp-input-port-count=0, comp-output-port-count=0
07-22 07:49:20.123 5264 5264 E CLI cmd_run_ctx_create_components_from_config_components@babeltrace2.c:2301 Cannot create component: plugin-name="ctf", comp-cls-name="fs", comp-cls-type=1, comp-name="auto-disc-source-ctf-fs"
07-22 07:49:20.123 5264 5264 E CLI cmd_run@babeltrace2.c:2480 Cannot create components.
ERROR: [Babeltrace CLI] (babeltrace2.c:2480)
Cannot create components.
CAUSED BY [Babeltrace CLI] (babeltrace2.c:2301)
Cannot create component: plugin-name="ctf", comp-cls-name="fs", comp-cls-type=1, comp-name="auto-disc-source-ctf-fs"
CAUSED BY [libbabeltrace2] (graph.c:977)
Component initialization method failed: status=ERROR, comp-addr=0x22a68d0, comp-name="auto-disc-source-ctf-fs", comp-log-level=WARNING,
comp-class-type=SOURCE, comp-class-name="fs", comp-class-partial-descr="Read CTF traces from the file sy", comp-class-is-frozen=0,
comp-class-so-handle-addr=0x22b0de0, comp-class-so-handle-path="/usr/lib/x86_64-linux-gnu/babeltrace2/plugins/babeltrace-plugin-ctf.so",
comp-input-port-count=0, comp-output-port-count=0 CAUSED BY [auto-disc-source-ctf-fs: 'source.ctf.fs'] (fs.c:1206)
Cannot create trace for `/mnt/ssd/work/skim/traces/kernel`.
</pre>
<p>I'm using below version.</p>
<pre>
skim@d54030999178:~/work/tmp$ babeltrace2 -V
Babeltrace 2.0.4 "Amqui"
Amqui (/_mkwi_/) is a town in eastern Qu_bec, Canada, at the base of the Gasp_ peninsula in Bas-Saint-Laurent. Located at the confluence of the Humqui and Matap_dia Rivers, its proximity to woodlands makes it a great destination for outdoor activities such as camping, hiking, and mountain biking.
</pre>
<p>Here is the test result which Philippe Proulx requested.</p>
<pre>
skim@d54030999178:~/ssd_work/traces$ babeltrace2 -o ctf-metadata ./kernel | grep -A10 '^clock {'
</pre>
<pre>
clock {
name = "monotonic";
uuid = "e00bcef2-1ef1-4f02-a241-8561834511fd";
description = "Monotonic Clock";
freq = 1000000000; /* Frequency, in Hz */
/* clock value offset from Epoch is: offset * (1/freq) */
offset = -48;
};
</pre> Babeltrace - Bug #1225 (On pause): src.ctf.lttng-live: `sessions` query error reportinghttps://bugs.lttng.org/issues/12252020-02-17T21:07:37ZFrancis Deslauriersfrancis.deslauriers@efficios.com
<p>The right format to query the list of sessions is:<br /><pre>
babeltrace2 query -p 'url="net://127.0.0.1"' src.ctf.lttng-live sessions
</pre></p>
<p>When adding a `/` at the end of the URL (by mistake maybe) you get the following error:<br /><pre>
10-23 10:11:25.193 2096 2096 E PLUGIN/SRC.CTF.LTTNG-LIVE/VIEWER parse_url@viewer-connection.c:127 [lttng-live] Invalid LTTng live URL format: Expecting `/host/` after hostname or port
10-23 10:11:25.193 2096 2096 E PLUGIN/SRC.CTF.LTTNG-LIVE/VIEWER lttng_live_connect_viewer@viewer-connection.c:253 [lttng-live] Failed to parse URL
10-23 10:11:25.193 2096 2096 E PLUGIN/SRC.CTF.LTTNG-LIVE/VIEWER live_viewer_connection_create@viewer-connection.c:1629 [lttng-live] Failure to establish connection: url="net://127.0.0.1/"
10-23 10:11:25.193 2096 2096 E PLUGIN/SRC.CTF.LTTNG-LIVE lttng_live_query_list_sessions@lttng-live.c:1589 [lttng-live] Failed to create viewer connection
10-23 10:11:25.193 2096 2096 W LIB/QUERY-EXECUTOR bt_query_executor_query@query-executor.c:246 Component class's "query" method failed: query-exec-addr=0x5625255fd7a0, cc-addr=0x5625255f80a0, cc-type=SOURCE, cc-name="lttng-live", cc-partial-descr="Connect to an LTTng relay daemon", cc-is-frozen=0, cc-so-handle-addr=0x5625255f7270, cc-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-ctf.so", object="sessions", params-addr=0x5625255e4540, params-type=MAP, params-element-count=1, log-level=WARNING
10-23 10:11:25.193 2096 2096 E CLI cmd_query@babeltrace2.c:727 Failed to query component class: unknown error: plugin-name="ctf", comp-cls-name="lttng-live", comp-cls-type=1 object="sessions"
ERROR: [Babeltrace CLI] (babeltrace2.c:727)
Failed to query component class: unknown error: plugin-name="ctf", comp-cls-name="lttng-live", comp-cls-type=1
object="sessions"
CAUSED BY [Babeltrace library] (query-executor.c:246)
Component class's "query" method failed: query-exec-addr=0x5625255fd7a0, cc-addr=0x5625255f80a0, cc-type=SOURCE,
cc-name="lttng-live", cc-partial-descr="Connect to an LTTng relay daemon", cc-is-frozen=0,
cc-so-handle-addr=0x5625255f7270, cc-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-ctf.so",
object="sessions", params-addr=0x5625255e4540, params-type=MAP, params-element-count=1, log-level=WARNING
CAUSED BY ['source.ctf.lttng-live'] (lttng-live.c:1589)
Failed to create viewer connection
CAUSED BY ['source.ctf.lttng-live'] (viewer-connection.c:1629)
Failure to establish connection: url="net://127.0.0.1/"
CAUSED BY ['source.ctf.lttng-live'] (viewer-connection.c:253)
Failed to parse URL
CAUSED BY ['source.ctf.lttng-live'] (viewer-connection.c:127)
Invalid LTTng live URL format: Expecting `/host/` after hostname or port
</pre><br />Suggesting to add the `/host/` part of the URL as if we were trying to connect to a session.<br />When in fact, since I am in a query, I would want it to be suggesting me to remove the extra `/` if anything.</p> LTTng - Bug #1207 (Confirmed): Tools 2.11 fails on destroy for lttng-modules 2.9https://bugs.lttng.org/issues/12072019-11-05T22:19:08ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>The following test from lttng-ivc is failing: test_modules_base_tracing[lttng-modules-2.9-lttng-tools-2.11]</p>
<p>This can be reproduced with head of lttng-ivc and running:</p>
<pre>
sudo tox -- -k test_modules_base_tracing[lttng-modules-2.9-lttng-tools-2.11
</pre>
<p>Relevant part so far from lttng-sessiond verbose mode:<br /><pre>
DEBUG1 - 16:20:40.713886603 [10006/10153]: Begin destroy session trace (id 0) (in cmd_destroy_session() at cmd.c:3176)
DEBUG1 - 16:20:40.713892241 [10006/10153]: Rotate kernel session trace started (session 0) (in kernel_rotate_session() at kernel.c:1445)
DEBUG1 - 16:20:40.713896285 [10006/10153]: Rotate kernel channel 1, session trace (in kernel_rotate_session() at kernel.c:1460)
DEBUG1 - 16:20:40.713899898 [10006/10153]: Consumer rotate channel key 1 (in consumer_rotate_channel() at consumer.c:1694)
DEBUG1 - 16:20:40.713913220 [10164/10173]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:3434)
DEBUG1 - 16:20:40.713925589 [10164/10173]: Consumer rotate channel 1 (in lttng_kconsumer_recv_cmd() at kernel-consumer.c:1124)
DEBUG1 - 16:20:40.713932035 [10164/10173]: Consumer sample rotate position for channel 1 (in lttng_consumer_rotate_channel() at consumer.c:4064)
Error: Failed to sample snapshot position during channel rotation
Error: Rotate channel failed
DEBUG1 - 16:20:40.713947543 [10164/10173]: Consumer rotate ready streams in channel 1 (in lttng_consumer_rotate_ready_streams() at consumer.c:4400)
DEBUG1 - 16:20:40.713951227 [10006/10153]: Consumer ret code -121 (in consumer_recv_status_reply() at consumer.c:208)
DEBUG1 - 16:20:40.713953876 [10164/10173]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:3450)
DEBUG1 - 16:20:40.713964554 [10006/10153]: Sending consumer close trace chunk command: relayd_id = -1, session_id = 0, chunk_id = 0, close command = "none" (in consumer_close_trace_chunk() at consumer.c:1970)
DEBUG1 - 16:20:40.713976173 [10164/10173]: Incoming command on sock (in consumer_thread_sessiond_poll() at consumer.c:3434)
DEBUG1 - 16:20:40.713985029 [10164/10173]: Consumer close trace chunk command: relayd_id = (none), session_id = 0, chunk_id = 0, close command = none (in lttng_consumer_close_trace_chunk() at consumer.c:4677)
DEBUG1 - 16:20:40.713999876 [10164/10173]: received command on sock (in consumer_thread_sessiond_poll() at consumer.c:3450)
Error: Failed to perform a quiet rotation as part of the destruction of session "trace": Unknown error code
DEBUG1 - 16:20:40.714017305 [10006/10153]: Tearing down kernel session (in kernel_destroy_session() at kernel.c:1199)
DEBUG1 - 16:20:40.714021825 [10006/10153]: [trace] Closing session fd 66 (in trace_kernel_destroy_session() at trace-kernel.c:689)
DEBUG1 - 16:20:40.714026143 [10006/10153]: [trace] Closing metadata stream fd 82 (in trace_kernel_destroy_session() at trace-kernel.c:699)
DEBUG1 - 16:20:40.714030026 [10006/10153]: [trace] Closing metadata fd 81 (in trace_kernel_destroy_metadata() at trace-kernel.c:662)
DEBUG1 - 16:20:40.714038288 [10006/10153]: [trace] Closing channel fd 73 (in trace_kernel_destroy_channel() at trace-kernel.c:616)
DEBUG1 - 16:20:40.714042757 [10006/10153]: [trace] Closing stream fd 77 (in trace_kernel_destroy_stream() at trace-kernel.c:544)
DEBUG1 - 16:20:40.714046626 [10006/10153]: [trace] Closing stream fd 76 (in trace_kernel_destroy_stream() at trace-kernel.c:544)
DEBUG1 - 16:20:40.714050245 [10006/10153]: [trace] Closing stream fd 75 (in trace_kernel_destroy_stream() at trace-kernel.c:544)
DEBUG1 - 16:20:40.714053833 [10006/10153]: [trace] Closing stream fd 65 (in trace_kernel_destroy_stream() at trace-kernel.c:544)
DEBUG1 - 16:20:40.714057474 [10006/10153]: [trace] Closing event fd 74 (in trace_kernel_destroy_event() at trace-kernel.c:570)
</pre></p>
<p>See attached tar.gz for more context.</p> LTTng-tools - Bug #1128 (Confirmed): Contexts can be added multiple times to a kernel channelhttps://bugs.lttng.org/issues/11282017-08-03T02:48:03ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>The session daemon checks for duplicate user space contexts being enabled on a user space channel (using trace_ust_match_context()). A similar check is missing for kernel channels, allowing users to add the same context multiple times.</p>
<p><em>Extract from a session profile demonstrating the problem:</em><br /><pre>
<domain>
<type>KERNEL</type>
<buffer_type>GLOBAL</buffer_type>
<channels>
<channel>
<name>lol</name>
<enabled>true</enabled>
<overwrite_mode>DISCARD</overwrite_mode>
<subbuffer_size>1048576</subbuffer_size>
<subbuffer_count>4</subbuffer_count>
<switch_timer_interval>0</switch_timer_interval>
<read_timer_interval>200000</read_timer_interval>
<output_type>SPLICE</output_type>
<tracefile_size>0</tracefile_size>
<tracefile_count>0</tracefile_count>
<live_timer_interval>4294967295</live_timer_interval>
<monitor_timer_interval>1000000</monitor_timer_interval>
<blocking_timeout>0</blocking_timeout>
<events/>
<contexts>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
<context>
<type>PID</type>
</context>
</contexts>
</channel>
</channels>
<trackers/>
</domain>
</pre></p> LTTng-tools - Bug #1102 (In Progress): Trigger conditions are not evaluated on subscriptionhttps://bugs.lttng.org/issues/11022017-05-11T22:43:22ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>Trigger conditions are not evaluated on subscription of a client. In the case of buffer usage conditions, this provides no way for clients to know the current state of the buffers.</p> LTTng - Bug #1060 (Confirmed): Document the extra reading subbuffer always allocatedhttps://bugs.lttng.org/issues/10602016-08-30T14:04:00ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>On channel creation a user might only consider the size of the passed argument instead we should either warn or document that there is always an extra sub buffer allocated for reading purpose.</p>
<p>See lib_ring_buffer_backend_allocate on lttng-modules.</p>
<p>This also apply to userspace.</p> LTTng-tools - Bug #1059 (Confirmed): The save and load commands do not use the same default home ...https://bugs.lttng.org/issues/10592016-08-25T13:28:24ZPhilippe Proulxeeppeliteloop@gmail.com
<p>The <code>save</code> command does not consider the <code>LTTNG_HOME</code> environment variable (nor <code>HOME</code>) because it uses <code>utils_get_user_home_dir()</code>, whereas the <code>load</code> command uses <code>utils_get_home_dir()</code>.</p>
<p>Therefore if <code>LTTNG_HOME</code> is set (or if <code>$HOME</code> has a different value than the entry in <code>/etc/passwd</code>), a <code>save</code> and <code>load</code> sequence does not find the session configuration file.</p>
<p>To remain consistent, I think the value of <code>utils_get_home_dir()</code> should be sent from the client to the session daemon at save time, so that, from the user's perspective, both commands are synchronized on the same environment variables.</p>
<p>Use case: this bug makes the save/load operations impossible to do in a virtual environment.</p> LTTng - Bug #1034 (Confirmed): KPROBE & KREPROBE support for saved/loaded sessions is not workinghttps://bugs.lttng.org/issues/10342016-06-22T21:01:13ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Reproduce:</p>
<p>lttng create<br />lttng enable-event -k --probe={a valid address from System.map}<br />lttng save<br />lttng load -i {the path to the previously save session}</p>
<p>Same for --function</p> LTTng - Bug #1030 (Confirmed): Ci: unit test are not running for stable v2.8 and greaterhttps://bugs.lttng.org/issues/10302016-06-15T17:09:58ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>Since 2.8 the unit_test test descriptor under the test/ folder was removed since we moved to the autotools tap harness.</p>
<p>The ci jobs did not get updated and simply fail to run unit test.</p>
<p>19:15:40 + prove --merge -v --exec '' - --archive /home/jenkins/workspace/lttng-tools_master_build/arch/x86-32/babeltrace_version/master/build/std/conf/std/liburcu_version/master/tap/unit/<br />19:15:40 /tmp/hudson6487886163425871557.sh: line 247: /home/jenkins/workspace/lttng-tools_master_build/arch/x86-32/babeltrace_version/master/build/std/conf/std/liburcu_version/master/tests/unit_tests: No such file or directory</p> LTTng - Bug #1023 (Confirmed): Error message on enabling event/all event for -j/-l/-phttps://bugs.lttng.org/issues/10232016-06-14T14:38:53ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<p>To reproduce:<br /><pre>
lttng create
lttng enable-event -p -a (success)
lttng enable-event -p -a (fail)
Error: Events: UST event already enabled (channel channel0, session auto-20160614-103428)
Error: Command error
</pre></p>
<p>This message should be:</p>
<pre>
Error: Events: Python event already enabled (channel channel0, session auto-20160614-103428)
Error: Command error
</pre>
<p>Same behavior for -l and -j</p> LTTng-modules - Bug #938 (Confirmed): Enabling two events with same name but not the same event t...https://bugs.lttng.org/issues/9382015-09-15T17:10:43ZJonathan Rajotte Julienjonathan.rajotte-julien@efficios.com
<pre>
sudo lttng-sessiond -d
lttng create mysession
lttng enable-event --kernel kmem_kfree --probe sys_open+0xE
lttng enable-event --kernel kmem_kfree
lttng start
sleep 0.1
lttng stop
lttng view
</pre>
<p>Only the probe is considered when tracing. The metadata only shows the probe event.</p>
<p>This is a kernel tracer side problem (name as event key). A solution could be to prefix certain type of event on enable thus creating namespace e.g: probe_kmem_kfree.</p> LTTng - Bug #934 (Confirmed): Session in snapshot mode does not indicate a path when listedhttps://bugs.lttng.org/issues/9342015-09-06T03:23:30ZJérémie Galarneaujeremie.galarneau@efficios.com
<p>The LTTng client does not indicate the default snapshot output path when listing a snapshot session.<br />It should either omit the "Trace path:", or indicate the default snapshot output path.</p>
<pre>
$ lttng create snap --snapshot
Session snap created.
Default snapshot output set to: /home/jgalar/lttng-traces/snap-20150905-232023
Snapshot mode set. Every channel enabled for that session will be set in overwrite mode and mmap output.
$ lttng list snap
Tracing session snap: [inactive snapshot]
Trace path:
</pre>