Project

General

Profile

Bug #1169

Possible race-condition in lttng-ust-statedump.c cleanup code during process exit.

Added by Rob Maher about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Target version:
-
Start date:
09/19/2018
Due date:
% Done:

100%

Estimated time:

Description

We have a short-lived program called 'testcmd' running on a Broadwell x86 processor with 8 logical cores (4 physical)
During cleanup/exit of the process while the system is under load (stress -c 6) we are seeing occasional core-dumps with similar traceback.

After inspecting the code it does not appear that there is locking within lttng-ust-statedump.c for the lttng_ust_statedump_destroy() function when it destroys the list. This patch seems to have resolved the issue for us.

diff --git a/liblttng-ust/lttng-ust-statedump.c b/liblttng-ust/lttng-ust-statedump.c
index efa8a55..1b135de 100644
--- a/liblttng-ust/lttng-ust-statedump.c
+++ b/liblttng-ust/lttng-ust-statedump.c
@@ -630,6 +630,8 @@ void ust_dl_state_destroy(void)
 {
     unsigned int i;

+    ust_lock_nocheck();
+
     for (i = 0; i < UST_DL_STATE_TABLE_SIZE; i++) {
         struct cds_hlist_head *head;
         struct lttng_ust_dl_node *e, *tmp;
@@ -639,6 +641,8 @@ void ust_dl_state_destroy(void)
             free_dl_node(e);
         CDS_INIT_HLIST_HEAD(head);
     }
+
+    ust_unlock();
 }

 void lttng_ust_statedump_destroy(void)
-- 
2.1.1

Backtrace + some additional details are provided below. Please let me know if you need additional information.

#VERSION: This is using the source from the official lttng-ust 2.10.1 tarball. 
#https://lttng.org/files/lttng-ust/lttng-ust-2.10.1.tar.bz2

# Here is the test program.  Typically by the 40th iteration of the loop the program has crashed and dumped core at least once.
COUNTER=0; while [ 1 ]; do echo "ITER ${COUNTER}"; testcmd show c; ((COUNTER=COUNTER+1)); usleep 500000; done

# After setting the environment variable and re-running the above test, the program no longer crashes after >300 iterations.
export LTTNG_UST_DEBUG=1

### Here is the full backtrace of the threads present at the time of the core dump.
### This has been reproduced several times.  As mentioned above, it usually happens by the 40th iteration.

Thread 3 (LWP 22540):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1  0x000000344e21a246 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=0, op=0, uaddr=<optimized out>) at /usr/include/urcu/futex.h:65
No locals.
#2  futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, uaddr=<optimized out>) at /usr/include/urcu/futex.h:97
No locals.
#3  wait_for_sessiond (sock_info=0x344e4709e0 <local_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1311
No locals.
#4  ust_listener_thread (arg=0x344e4709e0 <local_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1373
        sock_info = 0x344e4709e0 <local_apps>
        sock = <optimized out>
        ret = <optimized out>
        prev_connect_failed = 1
        has_waited = <optimized out>
        timeout = <optimized out>
        __func__ = "ust_listener_thread" 
        __PRETTY_FUNCTION__ = "ust_listener_thread" 
#5  0x00000034460073e7 in start_thread (arg=0x7fcff3fde700) at /usr/src/debug/glibc/2.26-r0/git/nptl/pthread_create.c:465
        pd = 0x7fcff3fde700
        now = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {140531128461056, -6091954114337190402, 140734590213134, 140734590213135, 8396800, 140734590213312, 6118966899948899838, -6116587572263268866}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 0
            }
          }
        }
        not_first_call = <optimized out>
#6  0x00000034458f0bcf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 2 (LWP 22538):
#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
No locals.
#1  0x000000344587d204 in _int_free (av=0x3445baac20 <main_arena>, p=0x606160, have_lock=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:4273
        size = 4192
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        __func__ = "_int_free" 
#2  0x000000344e22f2a1 in free_dl_node (e=0x606170) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:122
No locals.
#3  ust_dl_state_destroy () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:639
        e = 0x606170
        tmp = 0x0
        i = <optimized out>
#4  lttng_ust_statedump_destroy () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:649
No locals.
#5  0x000000344e218f4b in lttng_ust_cleanup (exiting=1) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1815
No locals.
#6  0x000000344e20f077 in lttng_ust_exit () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1874
        ret = <optimized out>
        __func__ = "lttng_ust_exit" 
#7  0x000000344540f943 in _dl_fini () at /usr/src/debug/glibc/2.26-r0/git/elf/dl-fini.c:235
        array = 0x344e469e60
        i = <optimized out>
        l = 0x7fcff47e4000
        maps = 0x7fff5341dfa0
        i = <optimized out>
        l = <optimized out>
        nmaps = <optimized out>
        nloaded = <optimized out>
        ns = 0
        do_audit = <optimized out>
        __PRETTY_FUNCTION__ = "_dl_fini" 
#8  0x0000003445836608 in __run_exit_handlers (status=0, listp=0x3445baa6f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at /usr/src/debug/glibc/2.26-r0/git/stdlib/exit.c:83
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
        f = <optimized out>
#9  0x000000344583665a in __GI_exit (status=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/stdlib/exit.c:105
No locals.
#10 0x0000003445820f11 in __libc_start_main (main=0x400940 <main>, argc=3, argv=0x7fff5341e188, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff5341e178) at /usr/src/debug/glibc/2.26-r0/git/csu/libc-start.c:342
        result = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {0, -6091953564717691394, 4197168, 140734590214528, 0, 0, 6092154793035550206, -6116588655236100610}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x7fff5341e1a8, 0x3445626110}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 1396826536
            }
          }
        }
        not_first_call = <optimized out>
#11 0x0000000000400b5a in _start () at ../sysdeps/x86_64/start.S:120
No locals.

Thread 1 (LWP 22539):
#0  __GI_raise (sig=sig@entry=6) at /usr/src/debug/glibc/2.26-r0/git/sysdeps/unix/sysv/linux/raise.c:51
        set = {
          __val = {18446744067266838239, 18446744073709551615 <repeats 13 times>, 18446744067266838271, 0}
        }
        pid = <optimized out>
        tid = <optimized out>
#1  0x0000003445835157 in __GI_abort () at /usr/src/debug/glibc/2.26-r0/git/stdlib/abort.c:90
        save_stage = 2
        act = {
          __sigaction_handler = {
            sa_handler = 0xffffffffffffffff, 
            sa_sigaction = 0xffffffffffffffff
          }, 
          sa_mask = {
            __val = {18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744067266838271, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744067266838271, 140531136848272, 140531136848560}
          }, 
          sa_flags = -193076848, 
          sa_restorer = 0x1000
        }
        sigs = {
          __val = {32, 0 <repeats 15 times>}
        }
#2  0x00000034458751b8 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x34459750fc "%s\n") at /usr/src/debug/glibc/2.26-r0/git/sysdeps/posix/libc_fatal.c:181
        ap = {{
            gp_offset = 24, 
            fp_offset = 0, 
            overflow_arg_area = 0x7fcff47de2c0, 
            reg_save_area = 0x7fcff47de250
          }}
        fd = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x000000344587b33a in malloc_printerr (str=str@entry=0x3445976d30 "double free or corruption (!prev)") at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:5368
No locals.
#4  0x000000344587cffc in _int_free (av=0x3445baac20 <main_arena>, p=0x610940, have_lock=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:4288
        size = 4192
        fb = <optimized out>
        nextchunk = 0x6119a0
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        __func__ = "_int_free" 
#5  0x000000344e22eb30 in free_dl_node (e=0x610950) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:122
No locals.
#6  iter_end (data=<optimized out>, ip=<optimized out>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:440
        e = 0x610950
#7  lttng_ust_dl_update (ip=0x344e224429 <lttng_handle_pending_statedump+9>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:575
        data = {
          exec_found = 0, 
          first = false, 
          cancel = true
        }
        ip = 0x344e224429 <lttng_handle_pending_statedump+9>
#8  0x000000344e22f107 in do_baddr_statedump (owner=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:588
No locals.
#9  do_lttng_ust_statedump (owner=owner@entry=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:611
No locals.
#10 0x000000344e224429 in lttng_handle_pending_statedump (owner=owner@entry=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-events.c:871
        session = <optimized out>
#11 0x000000344e21aa4d in handle_pending_statedump (sock_info=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:623
        ctor_passed = 1
#12 handle_message (lum=0x7fcff47de8b0, sock=<optimized out>, sock_info=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:981
        ret = <optimized out>
        lur = {
          handle = 0, 
          cmd = 68, 
          ret_code = 0, 
          ret_val = 0, 
          padding = '\000' <repeats 31 times>, 
          u = {
            channel = {
              memory_map_size = 0
            }, 
            stream = {
              memory_map_size = 0
            }, 
            version = {
              major = 0, 
              minor = 0, 
              patchlevel = 0
            }, 
            tracepoint = {
              name = '\000' <repeats 255 times>, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>
            }, 
            padding = '\000' <repeats 31 times>
          }
        }
        ctxstr = "\005", '\000' <repeats 254 times>
        len = <optimized out>
        ops = <optimized out>
        args = {
          channel = {
            chan_data = 0x1500000014, 
            wakeup_fd = 5
          }, 
          stream = {
            shm_fd = 20, 
            wakeup_fd = 21
          }, 
          field_list = {
            entry = {
              event_name = "\024\000\000\000\025\000\000\000\005", '\000' <repeats 246 times>, 
              field_name = '\000' <repeats 255 times>, 
              type = LTTNG_UST_FIELD_OTHER, 
              loglevel = 0, 
              nowrite = 0, 
              padding = '\000' <repeats 283 times>
            }
          }, 
          app_context = {
            ctxname = 0x1500000014 <error: Cannot access memory at address 0x1500000014>
          }
        }
#13 ust_listener_thread (arg=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1589
        len = <optimized out>
        lum = {
          handle = 0, 
          cmd = 68, 
          padding = '\000' <repeats 31 times>, 
          u = {
            channel = {
              len = 0, 
              type = LTTNG_UST_CHAN_PER_CPU, 
              padding = '\000' <repeats 287 times>, 
              data = 0x7fcff47dea04 "" 
            }, 
            stream = {
              len = 0, 
              stream_nr = 0, 
              padding = '\000' <repeats 287 times>
            }, 
            event = {
              instrumentation = LTTNG_UST_TRACEPOINT, 
              name = '\000' <repeats 255 times>, 
              loglevel_type = LTTNG_UST_LOGLEVEL_ALL, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>, 
              u = {
                padding = '\000' <repeats 287 times>
              }
            }, 
            context = {
              ctx = LTTNG_UST_CONTEXT_VTID, 
              padding = '\000' <repeats 15 times>, 
              u = {
                perf_counter = {
                  type = 0, 
                  config = 0, 
                  name = '\000' <repeats 255 times>
                }, 
                app_ctx = {
                  provider_name_len = 0, 
                  ctx_name_len = 0
                }, 
                padding = '\000' <repeats 287 times>
              }
            }, 
            version = {
              major = 0, 
              minor = 0, 
              patchlevel = 0
            }, 
            tracepoint = {
              name = '\000' <repeats 255 times>, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>
            }, 
            filter = {
              data_size = 0, 
              reloc_offset = 0, 
              seqnum = 0
            }, 
            exclusion = {
              count = 0
            }, 
            padding = '\000' <repeats 31 times>
          }
        }
        sock_info = 0x344e472a20 <global_apps>
        sock = <optimized out>
        ret = <optimized out>
        prev_connect_failed = 0
        has_waited = <optimized out>
        timeout = <optimized out>
        __func__ = "ust_listener_thread" 
        __PRETTY_FUNCTION__ = "ust_listener_thread" 
#14 0x00000034460073e7 in start_thread (arg=0x7fcff47df700) at /usr/src/debug/glibc/2.26-r0/git/nptl/pthread_create.c:465
        pd = 0x7fcff47df700
        now = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {140531136853760, -6091954114337190402, 140734590213134, 140734590213135, 8396800, 140734590213312, 6118950406737612286, -6116587572263268866}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 0
            }
          }
        }
        not_first_call = <optimized out>
#15 0x00000034458f0bcf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
(gdb) thread apply all bt full

Thread 3 (LWP 22540):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1  0x000000344e21a246 in futex (val3=0, uaddr2=0x0, timeout=0x0, val=0, op=0, uaddr=<optimized out>) at /usr/include/urcu/futex.h:65
No locals.
#2  futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, uaddr=<optimized out>) at /usr/include/urcu/futex.h:97
No locals.
#3  wait_for_sessiond (sock_info=0x344e4709e0 <local_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1311
No locals.
#4  ust_listener_thread (arg=0x344e4709e0 <local_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1373
        sock_info = 0x344e4709e0 <local_apps>
        sock = <optimized out>
        ret = <optimized out>
        prev_connect_failed = 1
        has_waited = <optimized out>
        timeout = <optimized out>
        __func__ = "ust_listener_thread" 
        __PRETTY_FUNCTION__ = "ust_listener_thread" 
#5  0x00000034460073e7 in start_thread (arg=0x7fcff3fde700) at /usr/src/debug/glibc/2.26-r0/git/nptl/pthread_create.c:465
        pd = 0x7fcff3fde700
        now = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {140531128461056, -6091954114337190402, 140734590213134, 140734590213135, 8396800, 140734590213312, 6118966899948899838, -6116587572263268866}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 0
            }
          }
        }
        not_first_call = <optimized out>
#6  0x00000034458f0bcf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Thread 2 (LWP 22538):
#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
No locals.
#1  0x000000344587d204 in _int_free (av=0x3445baac20 <main_arena>, p=0x606160, have_lock=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:4273
        size = 4192
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        __func__ = "_int_free" 
#2  0x000000344e22f2a1 in free_dl_node (e=0x606170) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:122
No locals.
#3  ust_dl_state_destroy () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:639
        e = 0x606170
        tmp = 0x0
        i = <optimized out>
#4  lttng_ust_statedump_destroy () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:649
No locals.
#5  0x000000344e218f4b in lttng_ust_cleanup (exiting=1) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1815
No locals.
#6  0x000000344e20f077 in lttng_ust_exit () at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1874
        ret = <optimized out>
        __func__ = "lttng_ust_exit" 
#7  0x000000344540f943 in _dl_fini () at /usr/src/debug/glibc/2.26-r0/git/elf/dl-fini.c:235
        array = 0x344e469e60
        i = <optimized out>
        l = 0x7fcff47e4000
        maps = 0x7fff5341dfa0
        i = <optimized out>
        l = <optimized out>
        nmaps = <optimized out>
        nloaded = <optimized out>
        ns = 0
        do_audit = <optimized out>
        __PRETTY_FUNCTION__ = "_dl_fini" 
#8  0x0000003445836608 in __run_exit_handlers (status=0, listp=0x3445baa6f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at /usr/src/debug/glibc/2.26-r0/git/stdlib/exit.c:83
        atfct = <optimized out>
        onfct = <optimized out>
        cxafct = <optimized out>
        f = <optimized out>
#9  0x000000344583665a in __GI_exit (status=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/stdlib/exit.c:105
No locals.
#10 0x0000003445820f11 in __libc_start_main (main=0x400940 <main>, argc=3, argv=0x7fff5341e188, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff5341e178) at /usr/src/debug/glibc/2.26-r0/git/csu/libc-start.c:342
        result = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {0, -6091953564717691394, 4197168, 140734590214528, 0, 0, 6092154793035550206, -6116588655236100610}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x7fff5341e1a8, 0x3445626110}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 1396826536
            }
          }
        }
        not_first_call = <optimized out>
#11 0x0000000000400b5a in _start () at ../sysdeps/x86_64/start.S:120
No locals.

Thread 1 (LWP 22539):
#0  __GI_raise (sig=sig@entry=6) at /usr/src/debug/glibc/2.26-r0/git/sysdeps/unix/sysv/linux/raise.c:51
        set = {
          __val = {18446744067266838239, 18446744073709551615 <repeats 13 times>, 18446744067266838271, 0}
        }
        pid = <optimized out>
        tid = <optimized out>
#1  0x0000003445835157 in __GI_abort () at /usr/src/debug/glibc/2.26-r0/git/stdlib/abort.c:90
        save_stage = 2
        act = {
          __sigaction_handler = {
            sa_handler = 0xffffffffffffffff, 
            sa_sigaction = 0xffffffffffffffff
          }, 
          sa_mask = {
            __val = {18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744067266838271, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744073709551615, 18446744067266838271, 140531136848272, 140531136848560}
          }, 
          sa_flags = -193076848, 
          sa_restorer = 0x1000
        }
        sigs = {
          __val = {32, 0 <repeats 15 times>}
        }
#2  0x00000034458751b8 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x34459750fc "%s\n") at /usr/src/debug/glibc/2.26-r0/git/sysdeps/posix/libc_fatal.c:181
        ap = {{
            gp_offset = 24, 
            fp_offset = 0, 
            overflow_arg_area = 0x7fcff47de2c0, 
            reg_save_area = 0x7fcff47de250
          }}
        fd = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3  0x000000344587b33a in malloc_printerr (str=str@entry=0x3445976d30 "double free or corruption (!prev)") at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:5368
No locals.
#4  0x000000344587cffc in _int_free (av=0x3445baac20 <main_arena>, p=0x610940, have_lock=<optimized out>) at /usr/src/debug/glibc/2.26-r0/git/malloc/malloc.c:4288
        size = 4192
        fb = <optimized out>
        nextchunk = 0x6119a0
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        __func__ = "_int_free" 
#5  0x000000344e22eb30 in free_dl_node (e=0x610950) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:122
No locals.
#6  iter_end (data=<optimized out>, ip=<optimized out>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:440
        e = 0x610950
#7  lttng_ust_dl_update (ip=0x344e224429 <lttng_handle_pending_statedump+9>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:575
        data = {
          exec_found = 0, 
          first = false, 
          cancel = true
        }
        ip = 0x344e224429 <lttng_handle_pending_statedump+9>
#8  0x000000344e22f107 in do_baddr_statedump (owner=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:588
No locals.
#9  do_lttng_ust_statedump (owner=owner@entry=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-statedump.c:611
No locals.
#10 0x000000344e224429 in lttng_handle_pending_statedump (owner=owner@entry=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-events.c:871
        session = <optimized out>
#11 0x000000344e21aa4d in handle_pending_statedump (sock_info=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:623
        ctor_passed = 1
#12 handle_message (lum=0x7fcff47de8b0, sock=<optimized out>, sock_info=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:981
        ret = <optimized out>
        lur = {
          handle = 0, 
          cmd = 68, 
          ret_code = 0, 
          ret_val = 0, 
          padding = '\000' <repeats 31 times>, 
          u = {
            channel = {
              memory_map_size = 0
            }, 
            stream = {
              memory_map_size = 0
            }, 
            version = {
              major = 0, 
              minor = 0, 
              patchlevel = 0
            }, 
            tracepoint = {
              name = '\000' <repeats 255 times>, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>
            }, 
            padding = '\000' <repeats 31 times>
          }
        }
        ctxstr = "\005", '\000' <repeats 254 times>
        len = <optimized out>
        ops = <optimized out>
        args = {
          channel = {
            chan_data = 0x1500000014, 
            wakeup_fd = 5
          }, 
          stream = {
            shm_fd = 20, 
            wakeup_fd = 21
          }, 
          field_list = {
            entry = {
              event_name = "\024\000\000\000\025\000\000\000\005", '\000' <repeats 246 times>, 
              field_name = '\000' <repeats 255 times>, 
              type = LTTNG_UST_FIELD_OTHER, 
              loglevel = 0, 
              nowrite = 0, 
              padding = '\000' <repeats 283 times>
            }
          }, 
          app_context = {
            ctxname = 0x1500000014 <error: Cannot access memory at address 0x1500000014>
          }
        }
#13 ust_listener_thread (arg=0x344e472a20 <global_apps>) at /usr/src/debug/lttng-ust/2_2.10.1-r0/lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1589
        len = <optimized out>
        lum = {
          handle = 0, 
          cmd = 68, 
          padding = '\000' <repeats 31 times>, 
          u = {
            channel = {
              len = 0, 
              type = LTTNG_UST_CHAN_PER_CPU, 
              padding = '\000' <repeats 287 times>, 
              data = 0x7fcff47dea04 "" 
            }, 
            stream = {
              len = 0, 
              stream_nr = 0, 
              padding = '\000' <repeats 287 times>
            }, 
            event = {
              instrumentation = LTTNG_UST_TRACEPOINT, 
              name = '\000' <repeats 255 times>, 
              loglevel_type = LTTNG_UST_LOGLEVEL_ALL, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>, 
              u = {
                padding = '\000' <repeats 287 times>
              }
            }, 
            context = {
              ctx = LTTNG_UST_CONTEXT_VTID, 
              padding = '\000' <repeats 15 times>, 
              u = {
                perf_counter = {
                  type = 0, 
                  config = 0, 
                  name = '\000' <repeats 255 times>
                }, 
                app_ctx = {
                  provider_name_len = 0, 
                  ctx_name_len = 0
                }, 
                padding = '\000' <repeats 287 times>
              }
            }, 
            version = {
              major = 0, 
              minor = 0, 
              patchlevel = 0
            }, 
            tracepoint = {
              name = '\000' <repeats 255 times>, 
              loglevel = 0, 
              padding = '\000' <repeats 15 times>
            }, 
            filter = {
              data_size = 0, 
              reloc_offset = 0, 
              seqnum = 0
            }, 
            exclusion = {
              count = 0
            }, 
            padding = '\000' <repeats 31 times>
          }
        }
        sock_info = 0x344e472a20 <global_apps>
        sock = <optimized out>
        ret = <optimized out>
        prev_connect_failed = 0
        has_waited = <optimized out>
        timeout = <optimized out>
        __func__ = "ust_listener_thread" 
        __PRETTY_FUNCTION__ = "ust_listener_thread" 
#14 0x00000034460073e7 in start_thread (arg=0x7fcff47df700) at /usr/src/debug/glibc/2.26-r0/git/nptl/pthread_create.c:465
        pd = 0x7fcff47df700
        now = <optimized out>
        unwind_buf = {
          cancel_jmp_buf = {{
              jmp_buf = {140531136853760, -6091954114337190402, 140734590213134, 140734590213135, 8396800, 140734590213312, 6118950406737612286, -6116587572263268866}, 
              mask_was_saved = 0
            }}, 
          priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {
              prev = 0x0, 
              cleanup = 0x0, 
              canceltype = 0
            }
          }
        }
        not_first_call = <optimized out>
#15 0x00000034458f0bcf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
(gdb) 

### This is the output of a single execution of the above command with LTTNG_UST_DEBUG environment variable set.
### (Program does not crash and exits normally.)
root@hal-dnx:/# testcmd show c
liblttng_ust_tracepoint[6722/6722]: Your compiler treats weak symbols with hidden visibility for integer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:916)
liblttng_ust_tracepoint[6722/6722]: Your compiler treats weak symbols with hidden visibility for pointer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:920)
liblttng_ust_tracepoint[6722/6722]: Your compiler treats weak symbols with hidden visibility for 24-byte structure objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:924)
liblttng_ust_tracepoint[6722/6722]: just registered a tracepoints section from 0x344e4751e0 and having 25 tracepoints (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:867)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_statedump:end (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_statedump:debug_link (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_statedump:build_id (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_statedump:bin_info (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_statedump:start (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_lib:unload (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_lib:debug_link (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_lib:build_id (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_lib:load (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracef:event (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_LINE (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_UNIT (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_MODULE (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROCESS (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_INFO (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_NOTICE (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_WARNING (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_ERR (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_CRIT (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_ALERT (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
liblttng_ust_tracepoint[6722/6722]: registered tracepoint: lttng_ust_tracelog:TRACE_EMERG (in tracepoint_register_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:872)
libust[6722/6722]: Provider "lttng_ust_statedump" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: adding probe lttng_ust_statedump containing 5 events to lazy registration list (in lttng_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:219)
libust[6722/6722]: LTT : ltt ring buffer client "relay-metadata-mmap" init
 (in lttng_ring_buffer_metadata_client_init() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-metadata-client.h:350)
libust[6722/6722]: LTT : ltt ring buffer client "relay-overwrite-mmap" init
 (in lttng_ring_buffer_client_overwrite_init() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:825)
libust[6722/6722]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init
 (in lttng_ring_buffer_client_overwrite_rt_init() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:825)
libust[6722/6722]: LTT : ltt ring buffer client "relay-discard-mmap" init
 (in lttng_ring_buffer_client_discard_init() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:825)
libust[6722/6722]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init
 (in lttng_ring_buffer_client_discard_rt_init() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:825)
libust[6722/6724]: Info: sessiond not accepting connections to local apps socket (in ust_listener_thread() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1424)
libust[6722/6724]: Waiting for local apps sessiond (in wait_for_sessiond() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:1306)
libust[6722/6723]: Message Received "Get Tracer Version" (65), Handle "root" (0) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Session" (64), Handle "root" (0) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 1 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Channel" (81), Handle "session" (1) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: channel data received (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:811)
libust[6722/6723]: Return value: 2 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Stream" (96), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 3 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (3) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 4 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (4) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 5 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (5) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 6 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (6) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 7 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (7) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 8 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (8) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 9 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (9) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 10 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (10) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 11 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (11) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 12 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (12) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 13 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (13) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 14 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (14) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 15 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (15) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 16 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (16) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 17 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (17) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 18 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (18) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 19 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (19) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 20 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (20) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 21 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (21) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 22 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (22) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 23 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (23) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Create Event" (97), Handle "channel" (2) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 24 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "enabler" (24) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Enable" (128), Handle "session" (1) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Sent register channel notification: chan_id 0, header_type 1
 (in ustcomm_register_channel() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1511)
libust[6722/6723]: just registered probe lttng_ust_statedump containing 5 events (in lttng_lazy_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:116)
liblttng_ust_tracepoint[6722/6723]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:702)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Wait for Quiescent State" (67), Handle "root" (0) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6723]: message successfully sent (in send_reply() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:571)
libust[6722/6723]: Message Received "Registration Done" (68), Handle "root" (0) (in print_cmd() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:430)
libust[6722/6722]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6723]: Return value: 0 (in handle_message() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ust-comm.c:972)
libust[6722/6722]: adding probe lttng_ust_lib containing 4 events to lazy registration list (in lttng_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:219)
libust[6722/6722]: just registered probe lttng_ust_lib containing 4 events (in lttng_lazy_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:116)
liblttng_ust_tracepoint[6722/6722]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:702)
libust[6722/6722]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: adding probe lttng_ust_tracef containing 1 events to lazy registration list (in lttng_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:219)
libust[6722/6722]: just registered probe lttng_ust_tracef containing 1 events (in lttng_lazy_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:116)
liblttng_ust_tracepoint[6722/6722]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:702)
libust[6722/6722]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: adding probe lttng_ust_tracelog containing 15 events to lazy registration list (in lttng_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:219)
libust[6722/6722]: just registered probe lttng_ust_tracelog containing 15 events (in lttng_lazy_probe_register() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:116)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_EMERG": ret_code 0, event_id 0
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_ALERT": ret_code 0, event_id 1
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_CRIT": ret_code 0, event_id 2
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_ERR": ret_code 0, event_id 3
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_WARNING": ret_code 0, event_id 4
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_NOTICE": ret_code 0, event_id 5
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
libust[6722/6722]: Sent register event notification for name "lttng_ust_tracelog:TRACE_INFO": ret_code 0, event_id 6
 (in ustcomm_register_event() at ../../lttng-ust-2.10.1/liblttng-ust-comm/lttng-ust-comm.c:1293)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_INFO. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_NOTICE. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_WARNING. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_ERR. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_CRIT. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_ALERT. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Registering probe to tracepoint lttng_ust_tracelog:TRACE_EMERG. Queuing release. (in __tracepoint_probe_register_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:611)
liblttng_ust_tracepoint[6722/6722]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:702)
libust[6722/6722]: Provider "lttng_ust_tracelog" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: just unregistered probe lttng_ust_tracelog (in lttng_probe_unregister() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:250)
libust[6722/6722]: Provider "lttng_ust_tracef" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: just unregistered probe lttng_ust_tracef (in lttng_probe_unregister() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:250)
libust[6722/6722]: Provider "lttng_ust_lib" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: just unregistered probe lttng_ust_lib (in lttng_probe_unregister() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:250)
|liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_INFO. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_NOTICE. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_WARNING. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_ERR. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_CRIT. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_ALERT. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Un-registering probe from tracepoint lttng_ust_tracelog:TRACE_EMERG. Queuing release. (in __tracepoint_probe_unregister_queue_release() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:682)
liblttng_ust_tracepoint[6722/6722]: Release queue of unregistered tracepoint probes. (in __tracepoint_probe_prune_release_queue() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:702)
libust[6722/6722]: LTT : ltt ring buffer client "relay-discard-rt-mmap" exit
 (in lttng_ring_buffer_client_discard_rt_exit() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:832)
libust[6722/6722]: LTT : ltt ring buffer client "relay-discard-mmap" exit
 (in lttng_ring_buffer_client_discard_exit() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:832)
libust[6722/6722]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" exit
 (in lttng_ring_buffer_client_overwrite_rt_exit() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:832)
libust[6722/6722]: LTT : ltt ring buffer client "relay-overwrite-mmap" exit
 (in lttng_ring_buffer_client_overwrite_exit() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-client.h:832)
libust[6722/6722]: LTT : ltt ring buffer client "relay-metadata-mmap" exit
 (in lttng_ring_buffer_metadata_client_exit() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-ring-buffer-metadata-client.h:357)
libust[6722/6722]: Provider "lttng_ust_statedump" accepted, version 1.0 is compatible with LTTng UST provider version 1.0. (in check_provider_version() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:175)
libust[6722/6722]: just unregistered probe lttng_ust_statedump (in lttng_probe_unregister() at ../../lttng-ust-2.10.1/liblttng-ust/lttng-probes.c:250)
liblttng_ust_tracepoint[6722/6722]: just unregistered a tracepoints section from 0x344e4751e0 (in tracepoint_unregister_lib() at ../../lttng-ust-2.10.1/liblttng-ust/tracepoint.c:897)
root@hal-dnx:/#
root@hal-dnx:/# 


Files

#1

Updated by Jonathan Rajotte Julien about 2 years ago

  • Description updated (diff)
  • Status changed from New to Feedback
  • Assignee set to Mathieu Desnoyers

Mathieu can you take a look at this?

Thanks

#2

Updated by Mathieu Desnoyers about 2 years ago

Can you try with the attached patch ?

#3

Updated by Mathieu Desnoyers about 2 years ago

  • Status changed from Confirmed to Resolved
  • % Done changed from 0 to 100
#4

Updated by Rob Maher almost 2 years ago

Mathieu Desnoyers wrote:

Can you try with the attached patch ?

After reproducing the original issue I downloaded the patch from the git repo and applied it.
Still no crashes after ~25000 iterations.
Thank you for the fix!

Also available in: Atom PDF