LTTng bugs repository: Issueshttps://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912023-06-01T18:48:58ZLTTng bugs repository
Redmine LTTng-tools - Bug #1379 (New): Document behavior of live mode with per-pid UST buffershttps://bugs.lttng.org/issues/13792023-06-01T18:48:58ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>There is a design limitation of per-pid UST buffers when used in live mode which should be documented.</p>
<p>Basically, because the per-pid buffers are reclaimed soon after the application exits, it may cause the events traced by a short-lived application to never appear in the output of a live trace when per-pid buffers are used.</p>
<p>This is caused by the fact that the live client periodically checks for new buffers, but there is no inherent reference kept on the streams before they disappear.</p>
<p>This means that the information will be available in the disk output of the trace, but it will be missing from the live mode output.</p>
<p>This could eventually be mitigated by keeping references to the streams received by the relay daemon which are of interest to a live client until they are referenced by the live client.</p> LTTng-tools - Bug #1378 (New): Document behavior of snapshot mode with per-pid UST buffershttps://bugs.lttng.org/issues/13782023-06-01T18:01:12ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should document a design limitation of the per-pid UST buffers which can lead to unexpected results when used with the "snapshot" feature.</p>
<p>Basically, because the per-pid buffers lifetime is bound by the application using them (and the consumer daemon extracting data from them), they are freed almost immediately after the traced application exits. However, in a scenario where we are interested in capturing a snapshot of the content of flight recorder buffers soon after application exits (e.g. caused by a crash), those buffers are not available anymore a few milliseconds after the application exits.</p>
<p>We should document this design limitation, and eventually think about improvements (feature request ?) to allow a configurable delay after application exit during which the consumer daemon could keep references on the buffers so they are available for snapshot.</p> LTTng-tools - Bug #1340 (New): Document behavior of per-uid UST buffers with respect to asynchron...https://bugs.lttng.org/issues/13402021-12-07T20:22:28ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should document a known design limitation of the per-uid UST buffers:</p>
<p>When using lttng-ust with per-uid buffers, there is a known design limitation<br />regarding process terminated by non-handled kill signals: if this occurs while<br />the buffer is being written to (between reserve and commit), it will cause the<br />rest of the per-cpu buffer to be unreadable. This applies to all events<br />recorded by all applications of the same user id for that particular CPU.<br />Recovering from this involves destroying and re-creating the tracing session.</p>
<p>If this kind of scenario is likely for you, you might want to consider using per-<br />pid buffers (lttng enable-channel -u --buffers-pid), which do not suffer from<br />this design limitation. The downside of per-pid buffers is that it allocates<br />more memory on your system for buffering, and adds extra overhead when<br />used with processes that have a short life-time.</p> Babeltrace - Bug #1299 (Confirmed): babeltrace2 re-uses prior event string rather than expected e...https://bugs.lttng.org/issues/12992021-03-04T18:49:50ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Trying to reproduce the page fault scenario for an openat() event. Note that the behavior of generating a page fault may be toolchain-specific (linker-specific):</p>
<p>Test program "do-pagefault-open":</p>
<p>#include <stdio.h><br />#include <sys/types.h><br />#include <sys/stat.h><br />#include <fcntl.h></p>
<p>const char *str = "/tmp/blah";<br />const char *str2 = "/tmp/blah2";</p>
<p>int main()
{<br /> open(str, O_RDWR);<br /> perror("first");<br /> open(str2, O_RDWR);<br /> perror("second");<br /> return 0;<br />}</p>
<p>then trace with:</p>
<p>lttng-trace ./do-pagefault-open</p>
<p>Relevant events with babeltrace2 (grepping for 'openat'):</p>
<p>[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296960451] (+0.000006223) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }</p>
<p>Same trace with babeltrace 1:</p>
<p>[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 5 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 129 in enum.<br />[warning] Unknown value 3 in enum.<br />[warning] Unknown value 3 in enum.<br />[13:47:12.296015570] (+0.000015421) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/etc/ld.so.cache", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296029004] (+0.000013434) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[13:47:12.296116608] (+0.000058899) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/lib/x86_64-linux-gnu/libc.so.6", flags = ( "O_CLOEXEC" : container = 524288 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296126244] (+0.000009636) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = 3 }<br />[warning] Unknown value 0 in enum.<br />[warning] Unknown value 0 in enum.<br />[13:47:12.296706313] (+0.000096888) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }<br />[13:47:12.296717797] (+0.000011484) thinkos syscall_exit_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { ret = -2 }<br />[13:47:12.296954228] (+0.000005481) thinkos syscall_entry_openat: { cpu_id = 3 }, { procname = "do-pagefault-op", vpid = 31576, vtid = 31576 }, { dfd = -100, filename = "/tmp/blah2", flags = ( "O_RDWR" : container = 2 ), mode = ( <unknown> : container = 0 ) }</p>
<p>Notice how filename = "/lib/x86_64-linux-gnu/libc.so.6" appears twice in the bt2 output, where we would expect the empty string (as provided by bt1).</p>
<p>Tested with babeltrace2 commit c429f86d0.</p> LTTng-tools - Bug #1256 (New): Check babeltrace1 stderr output in testshttps://bugs.lttng.org/issues/12562020-04-08T19:23:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Currently, babeltrace1 stderr's output contains warnings which are not checked by the tests. Those do not fail the overall babeltrace execution, but end up being unexpected regressions when they appear between releases.</p>
<p>We could modify all babeltrace invocations to add a helper which saves the stderr output, and check that it is empty after babeltrace has completed execution. If non-empty, a test error would be reported.</p> Babeltrace - Bug #1221 (New): Babeltrace 2 sometimes fail on live trace with lttng-clear featurehttps://bugs.lttng.org/issues/12212020-02-17T20:52:19ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Babeltrace 2 sometimes fail on live trace with lttng-clear feature</p>
<p>This may happen in a scenario where per-pid buffers are used, an application has just quit, and babeltrace 2 observes a stream in make_viewer_streams, but then fails to open the associated index or data file on the first get_next_index because it has been unlinked by clear.</p>
<p>The babeltrace output is:</p>
<pre>
12-04 14:01:48.455 11884 11884 E PLUGIN/CTF/MSG-ITER create_msg_stream_end@msg-iter.c:2464 [lttng-live] Cannot create stream for stream message: notit-addr=0x55b89f957980
12-04 14:01:48.455 11884 11884 W LIB/MSG-ITER bt_self_component_port_input_message_iterator_next@iterator.c:906 Component input port message iterator's "next" method failed: iter-addr=0x55b89f93f510, iter-upstream-comp-name="lttng-live", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE, iter-upstream-comp-class-name="lttng-live", iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER muxer_upstream_msg_iter_next@muxer.c:444 [muxer] Error or unsupported status code: status-code=-1
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER validate_muxer_upstream_msg_iters@muxer.c:975 [muxer] Cannot validate muxer's upstream message iterator wrapper: muxer-msg-iter-addr=0x55b89f950950, muxer-upstream-msg-iter-wrap-addr=0x55b89f9415d0
12-04 14:01:48.455 11884 11884 E PLUGIN/FLT.UTILS.MUXER muxer_msg_iter_next@muxer.c:1371 [muxer] Cannot get next message: comp-addr=0x55b89f93ebc0, muxer-comp-addr=0x55b89f93ec40, muxer-msg-iter-addr=0x55b89f950950, msg-iter-addr=0x55b89f93f3a0, status=ERROR
12-04 14:01:48.455 11884 11884 W LIB/MSG-ITER bt_self_component_port_input_message_iterator_next@iterator.c:906 Component input port message iterator's "next" method failed: iter-addr=0x55b89f93f3a0, iter-upstream-comp-name="muxer", iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER, iter-upstream-comp-class-name="muxer", iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu", iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
12-04 14:01:48.455 11884 11884 W LIB/GRAPH consume_graph_sink@graph.c:596 Component's "consume" method failed: status=ERROR, comp-addr=0x55b89f93ed30, comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK, comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages (`text` fo", comp-class-is-frozen=0, comp-class-so-handle-addr=0x55b89f94c740, comp-class-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-text.so", comp-input-port-count=1, comp-output-port-count=0
12-04 14:01:48.456 11884 11884 E CLI cmd_run@babeltrace2.c:2574 Graph failed to complete successfully
ERROR: [Babeltrace CLI] (babeltrace2.c:2574)
Graph failed to complete successfully
CAUSED BY [Babeltrace library] (graph.c:596)
Component's "consume" method failed: status=ERROR, comp-addr=0x55b89f93ed30,
comp-name="pretty", comp-log-level=WARNING, comp-class-type=SINK,
comp-class-name="pretty", comp-class-partial-descr="Pretty-print messages
(`text` fo", comp-class-is-frozen=0, comp-class-so-handle-addr=0x55b89f94c740,
comp-class-so-handle-path="/usr/local/lib/babeltrace2/plugins/babeltrace-plugin-text.so",
comp-input-port-count=1, comp-output-port-count=0
CAUSED BY [Babeltrace library] (iterator.c:906)
Component input port message iterator's "next" method failed:
iter-addr=0x55b89f93f3a0, iter-upstream-comp-name="muxer",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=FILTER,
iter-upstream-comp-class-name="muxer",
iter-upstream-comp-class-partial-descr="Sort messages from multiple inpu",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
CAUSED BY [Babeltrace library] (iterator.c:906)
Component input port message iterator's "next" method failed:
iter-addr=0x55b89f93f510, iter-upstream-comp-name="lttng-live",
iter-upstream-comp-log-level=WARNING, iter-upstream-comp-class-type=SOURCE,
iter-upstream-comp-class-name="lttng-live",
iter-upstream-comp-class-partial-descr="Connect to an LTTng relay daemon",
iter-upstream-port-type=OUTPUT, iter-upstream-port-name="out", status=ERROR
</pre>
<p>notit->stream is expected not to be NULL when reaching the state STATE_EMIT_MSG_STREAM_END, but if no packet was received, it is indeed still NULL, because it is the packet header being read that moves to the STATE_CHECK_EMIT_MSG_STREAM_BEGINNING state (after_packet_context_state()).</p> LTTng-UST - Bug #1203 (New): Improve documentation of lttng-ust with daemonshttps://bugs.lttng.org/issues/12032019-10-07T15:51:26ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>The lttng-ust(3) man page should be clarified regarding the scenarios where the liblttng-ust-fork.so wrapper is needed.</p>
<p>It should document that it is needed in scenarios where applications using fork(2), clone(2) (with CLONE_VM flag cleared), BSD's rfork(2), daemon(3), without <strong>immediately</strong> following those by an exec(3) family call. This includes scenarios where close(2) is used to close file descriptors before invoking exec(3), even though close(2) is listed as an async-signal-safe function.</p> LTTng-tools - Bug #1182 (New): relayd mishandles channels configured with tracefile-count == 0 an...https://bugs.lttng.org/issues/11822019-04-02T17:25:50ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Relayd mistakenly considers that the criterion tracefile-count == 0 means no tracefile rotation should happen for the stream.</p>
<p>However, it's possible that tracefile-size is set to non-zero, but tracefile-count is left at its default value of 0.</p>
<p>The expected behavior would be to have an ever increasing counter (no limit) of tracefiles each "size" in size at most.</p>
<p>I suspect the relayd receiving side may also be broken, because its rotation handling code also has the same incorrect assumptions.</p>
<p>When connecting a live client to such a session, we get on the client side:<br />[error] Remote side has closed connection</p>
<p>And a PERROR on relayd:<br />PERROR - 13:16:28.097800421 [8061/8068]: opening index in read-only: No such file or directory (in lttng_index_file_open() at index.c:261)</p>
<p>Likely caused by confusion between tracefile rotation suffixes in the filename.</p> LTTng-tools - Bug #1117 (New): lttng-relayd should use specific major.minor versions for relay-co...https://bugs.lttng.org/issues/11172017-06-02T01:36:05ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Currently, it uses RELAYD_VERSION_COMM_MAJOR/MINOR, which are increased at each LTTng minor release, and therefore automatically bump the minor version even though there are no new features added.</p> LTTng-tools - Bug #1116 (New): MI XSD minor version should have been bumped once or twice (?) bet...https://bugs.lttng.org/issues/11162017-06-02T00:42:28ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Looking at the diff of the xsd, we see that its content changed, which should have required minor xsd version bumps. What should we do now ?</p> LTTng-tools - Bug #1115 (New): LTTng sessiond should accept different agent minor version numberhttps://bugs.lttng.org/issues/11152017-06-02T00:40:03ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>Currently, it refuses to talk to agents with different minor number, which would not allow compatibility between minor bumps (feature addition).</p>
<p>Given that the agents only sent their major/minor version (and sessiond do not send its own), we should only accept agent minor versions that are <= to the server minor version.</p>
<p>If we ever want to support letting the sessiond talk to agents with higher minor version number, we should change the protocol to allow the agent handling component in the sessiond to publish its own minor version number to the application agents, so they would know they should only use a subset of their feature set.</p> LTTng-tools - Bug #987 (Confirmed): Incomplete string comparison patternhttps://bugs.lttng.org/issues/9872016-01-06T23:23:31ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>In a few instances in lttng-tools, a string comparison pattern involving strncmp and strlen() to limit the length of the comparison is used.</p>
<p>One can quickly find them with:</p>
<p>grep -r -A 3 strncmp . | less -> search for strlen or strnlen</p>
<p>Many of those are OK: they indeed only want to match the beginning of the strings. However, this pattern is incorrectly used in cases where a full match is required, for instance:</p>
<p>ust-registry.c:ht_match_event()<br />./src/bin/lttng/commands/enable_channels.c: if (!strncmp(output_mmap, opt_output, strlen(output_mmap))) {<br />./src/bin/lttng/commands/enable_channels.c: } else if (!strncmp(output_splice, opt_output, strlen(output_splice))) {<br />./src/bin/lttng-sessiond/snapshot.c: if (!strncmp(output->name, name, strlen(name))) {</p> LTTng-modules - Bug #975 (Confirmed): execve compat syscall exit syscall value issuehttps://bugs.lttng.org/issues/9752015-11-08T16:03:18ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>syscall_exit for execve changing the 32/64-bit compat mode for a process has wrong system call number on exit:</p>
<pre>
[19:36:57.188066018] (+0.000000616) sinkpad syscall_entry_execve: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", argv = 0x7FFD275BBF40, envp = 0x7FFD275BBF50 }
[19:36:57.188162705] (+0.000000851) sinkpad module_get: { cpu_id = 0 }, { ip = 18446744071581118857, refcnt = 3, name = "binfmt_misc" }
[19:36:57.188170506] (+0.000000517) sinkpad module_put: { cpu_id = 0 }, { ip = 18446744071581118912, refcnt = 2, name = "binfmt_misc" }
[19:36:57.188630250] (+0.000000911) sinkpad random_get_random_bytes: { cpu_id = 0 }, { nbytes = 16, IP = 18446744071581461559 }
[19:36:57.188630781] (+0.000000531) sinkpad random_extract_entropy: { cpu_id = 0 }, { pool_name = "nonblocking", nbytes = 16, entropy_count = 0, IP = 18446744071583984742 }
[19:36:57.188634640] (+0.000001773) sinkpad sched_waking: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188637855] (+0.000003215) sinkpad sched_stat_sleep: { cpu_id = 0 }, { comm = "rngd", tid = 2177, delay = 2656341 }
[19:36:57.188639681] (+0.000001826) sinkpad sched_wakeup: { cpu_id = 0 }, { comm = "rngd", tid = 2177, prio = 120, target_cpu = 2 }
[19:36:57.188641483] (+0.000000180) sinkpad power_cpu_idle: { cpu_id = 2 }, { state = 4294967295, cpu_id = 2 }
[19:36:57.188646080] (+0.000000644) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649456] (+0.000000788) sinkpad random_mix_pool_bytes_nolock: { cpu_id = 0 }, { pool_name = "nonblocking", bytes = 20, IP = 18446744071583981062 }
[19:36:57.188649704] (+0.000000248) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188650149] (+0.000000182) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188652253] (+0.000001864) sinkpad sched_stat_wait: { cpu_id = 2 }, { comm = "rngd", tid = 2177, delay = 0 }
[19:36:57.188654080] (+0.000001827) sinkpad sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "rngd", next_tid = 2177, next_prio = 20 }
[19:36:57.188658382] (+0.000000567) sinkpad sched_process_exec: { cpu_id = 0 }, { filename = "/usr/bin/burnP6", tid = 29058, old_tid = 29058 }
[19:36:57.188661040] (+0.000000020) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "Start context switch" }
[19:36:57.188661415] (+0.000000375) sinkpad rcu_utilization: { cpu_id = 2 }, { s = "End context switch" }
[19:36:57.188662327] (+0.000000409) sinkpad sched_stat_runtime: { cpu_id = 2 }, { comm = "rngd", tid = 2177, runtime = 25827, vruntime = 1365908673 }
[19:36:57.188664216] (+0.000001266) sinkpad compat_syscall_exit_olduname: { cpu_id = 0 }, { ret = 0, name = 0 }
</pre> LTTng-tools - Bug #833 (Confirmed): memcpy of non-packed struct into packed struct (possible layo...https://bugs.lttng.org/issues/8332014-09-09T13:17:10ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>lttng-tools src/lib/lttng-ctl/lttng-ctl.c:</p>
<p>lttng_enable_event_with_exclusions()</p>
<p>memcpy(&lsm.u.enable.event, ev, sizeof(lsm.u.enable.event));</p>
<p>copy "ev" (non-packed) into a packed structure.</p>
<p>We should copy each field one by one (create a copy_event_to_event_packed() helper to do so).</p> LTTng-UST - Bug #525 (Confirmed): new "notifications" from UST do not strictly respect LTTNG_UST_...https://bugs.lttng.org/issues/5252013-05-07T15:25:54ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<p>We should eventually find a way to improve notifications from UST to sessiond so they don't delay .so loading (and thus application startup) when env. var. specify a LTTNG_UST_REGISTER_TIMEOUT=0. Currently, we work around this issue by setting the notification socket with a timeout of 100ms as minimum timeout if the LTTNG_UST_REGISTER_TIMEOUT value is below 100.</p>
<p>A cleaner fix could involve handling these notifications from a (possibly new) separate thread, and use a semaphore-based scheme to handle optional wait from the application.</p>