Project

General

Profile

Bug #45

lttng core dump when doing lttng list -u

Added by Yannick Brosseau about 8 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Critical
Assignee:
Target version:
Start date:
02/13/2012
Due date:
% Done:

0%

Estimated time:

Description

I start an application instrumented with UST, I then do
lttng create
lttng list -u
and then lttng stop with a core dump

#1

Updated by Yannick Brosseau about 8 years ago

Output:
scientist@atwood:~/projets/lttng-tools/tests$ lttng create
Spawning a session daemon
Session auto-20120213-143531 created.
Traces will be written in /home/scientist/lttng-traces/auto-20120213-143531
scientist@atwood:~/projets/lttng-tools/tests$ lttng list -u
  • glibc detected * lttng-sessiond: free(): invalid pointer: 0x00007fd8ad22dadf * ======= Backtrace: =========
    /lib/x86_64-linux-gnu/libc.so.6(+0x75ab6)[0x7fd8acf15ab6]
    /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fd8acf1a7ec]
    lttng-sessiond[0x409f70]
    /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fd8ad22db50]
    /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd8acf7890d] ======= Memory map: ========
    00400000-0041a000 r-xp 00000000 fe:00 1573472 /usr/local/bin/lttng-sessiond
    0061a000-00621000 rw-p 0001a000 fe:00 1573472 /usr/local/bin/lttng-sessiond
    00621000-00624000 rw-p 00000000 00:00 0
    02533000-02554000 rw-p 00000000 00:00 0 [heap]
    7fd8a4000000-7fd8a4021000 rw-p 00000000 00:00 0
    7fd8a4021000-7fd8a8000000 ---p 00000000 00:00 0
    7fd8a9c4e000-7fd8a9c63000 r-xp 00000000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7fd8a9c63000-7fd8a9e63000 ---p 00015000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7fd8a9e63000-7fd8a9e64000 rw-p 00015000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7fd8a9e64000-7fd8a9e65000 ---p 00000000 00:00 0
    7fd8a9e65000-7fd8aa665000 rw-p 00000000 00:00 0
    7fd8aa665000-7fd8aa666000 ---p 00000000 00:00 0
    7fd8aa666000-7fd8aae66000 rw-p 00000000 00:00 0
    7fd8aae66000-7fd8aae67000 ---p 00000000 00:00 0
    7fd8aae67000-7fd8ab667000 rw-p 00000000 00:00 0
    7fd8ab667000-7fd8ab668000 ---p 00000000 00:00 0
    7fd8ab668000-7fd8abe68000 rw-p 00000000 00:00 0
    7fd8abe68000-7fd8abe69000 ---p 00000000 00:00 0
    7fd8abe69000-7fd8ac669000 rw-p 00000000 00:00 0
    7fd8ac669000-7fd8ac674000 r-xp 00000000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7fd8ac674000-7fd8ac873000 ---p 0000b000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7fd8ac873000-7fd8ac874000 r--p 0000a000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7fd8ac874000-7fd8ac875000 rw-p 0000b000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7fd8ac875000-7fd8ac87f000 r-xp 00000000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7fd8ac87f000-7fd8aca7e000 ---p 0000a000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7fd8aca7e000-7fd8aca7f000 r--p 00009000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7fd8aca7f000-7fd8aca80000 rw-p 0000a000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7fd8aca80000-7fd8aca95000 r-xp 00000000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7fd8aca95000-7fd8acc94000 ---p 00015000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7fd8acc94000-7fd8acc95000 r--p 00014000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7fd8acc95000-7fd8acc96000 rw-p 00015000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7fd8acc96000-7fd8acc98000 rw-p 00000000 00:00 0
    7fd8acc98000-7fd8acc9f000 r-xp 00000000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7fd8acc9f000-7fd8ace9e000 ---p 00007000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7fd8ace9e000-7fd8ace9f000 r--p 00006000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7fd8ace9f000-7fd8acea0000 rw-p 00007000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7fd8acea0000-7fd8ad01d000 r-xp 00000000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7fd8ad01d000-7fd8ad21d000 ---p 0017d000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7fd8ad21d000-7fd8ad221000 r--p 0017d000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7fd8ad221000-7fd8ad222000 rw-p 00181000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7fd8ad222000-7fd8ad227000 rw-p 00000000 00:00 0
    7fd8ad227000-7fd8ad23e000 r-xp 00000000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7fd8ad23e000-7fd8ad43d000 ---p 00017000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7fd8ad43d000-7fd8ad43e000 r--p 00016000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7fd8ad43e000-7fd8ad43f000 rw-p 00017000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7fd8ad43f000-7fd8ad443000 rw-p 00000000 00:00 0
    7fd8ad443000-7fd8ad44e000 r-xp 00000000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7fd8ad44e000-7fd8ad64e000 ---p 0000b000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7fd8ad64e000-7fd8ad64f000 rw-p 0000b000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7fd8ad64f000-7fd8ad653000 r-xp 00000000 fe:00 1576060 /usr/local/lib/liburcu.so.1.0.0
    7fd8ad653000-7fd8ad853000 ---p 00004000 fe:00 1576060 /usr/local/lib/liburcu.so.1.0.0
    7fd8ad853000-7fd8ad854000 rw-p 00004000 fe:00 1576060 /usr/local/lib/liburcu.so.1.0.0
    7fd8ad854000-7fd8ad856000 r-xp 00000000 fe:00 1576059 /usr/local/lib/liburcu-common.so.1.0.0
    7fd8ad856000-7fd8ada55000 ---p 00002000 fe:00 1576059 /usr/local/lib/liburcu-common.so.1.0.0
    7fd8ada55000-7fd8ada56000 rw-p 00001000 fe:00 1576059 /usr/local/lib/liburcu-common.so.1.0.0
    7fd8ada56000-7fd8ada5b000 r-xp 00000000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7fd8ada5b000-7fd8adc5b000 ---p 00005000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7fd8adc5b000-7fd8adc5c000 rw-p 00005000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7fd8adc5c000-7fd8adc5d000 rw-p 00000000 00:00 0
    7fd8adc5d000-7fd8adc64000 r-xp 00000000 fe:00 1709022 /lib/x86_64-linux-gnu/librt-2.13.so
    7fd8adc64000-7fd8ade63000 ---p 00007000 fe:00 1709022 /lib/x86_64-linux-gnu/librt-2.13.so
    7fd8ade63000-7fd8ade64000 r--p 00006000 fe:00 1709022 /lib/x86_64-linux-gnu/librt-2.13.so
    7fd8ade64000-7fd8ade65000 rw-p 00007000 fe:00 1709022 /lib/x86_64-linux-gnu/librt-2.13.so
    7fd8ade65000-7fd8ade84000 r-xp 00000000 fe:00 1709027 /lib/x86_64-linux-gnu/ld-2.13.so
    7fd8ae053000-7fd8ae058000 rw-p 00000000 00:00 0
    7fd8ae081000-7fd8ae082000 rw-s 00000000 00:13 186442 /run/shm/lttng-ust-apps-wait-1000UST events:
    -------------
    None
  • glibc detected * lttng: free(): invalid pointer: 0x00007f1f8dd3d4be * ======= Backtrace: =========
    /lib/x86_64-linux-gnu/libc.so.6(+0x75ab6)[0x7f1f8d600ab6]
    /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f1f8d6057ec]
    lttng[0x403c69]
    lttng[0x4023f2]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f1f8d5a9ead]
    lttng[0x402721] ======= Memory map: ========
    00400000-00410000 r-xp 00000000 fe:00 1573247 /usr/local/bin/lttng
    0060f000-00611000 rw-p 0000f000 fe:00 1573247 /usr/local/bin/lttng
    0111d000-0113e000 rw-p 00000000 00:00 0 [heap]
    7f1f88000000-7f1f88021000 rw-p 00000000 00:00 0
    7f1f88021000-7f1f8c000000 ---p 00000000 00:00 0
    7f1f8cb3e000-7f1f8cb53000 r-xp 00000000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f1f8cb53000-7f1f8cd53000 ---p 00015000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f1f8cd53000-7f1f8cd54000 rw-p 00015000 fe:00 1706455 /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f1f8cd54000-7f1f8cd5f000 r-xp 00000000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7f1f8cd5f000-7f1f8cf5e000 ---p 0000b000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7f1f8cf5e000-7f1f8cf5f000 r--p 0000a000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7f1f8cf5f000-7f1f8cf60000 rw-p 0000b000 fe:00 1706992 /lib/x86_64-linux-gnu/libnss_files-2.13.so
    7f1f8cf60000-7f1f8cf6a000 r-xp 00000000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7f1f8cf6a000-7f1f8d169000 ---p 0000a000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7f1f8d169000-7f1f8d16a000 r--p 00009000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7f1f8d16a000-7f1f8d16b000 rw-p 0000a000 fe:00 1706272 /lib/x86_64-linux-gnu/libnss_nis-2.13.so
    7f1f8d16b000-7f1f8d180000 r-xp 00000000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7f1f8d180000-7f1f8d37f000 ---p 00015000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7f1f8d37f000-7f1f8d380000 r--p 00014000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7f1f8d380000-7f1f8d381000 rw-p 00015000 fe:00 1705749 /lib/x86_64-linux-gnu/libnsl-2.13.so
    7f1f8d381000-7f1f8d383000 rw-p 00000000 00:00 0
    7f1f8d383000-7f1f8d38a000 r-xp 00000000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7f1f8d38a000-7f1f8d589000 ---p 00007000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7f1f8d589000-7f1f8d58a000 r--p 00006000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7f1f8d58a000-7f1f8d58b000 rw-p 00007000 fe:00 1705036 /lib/x86_64-linux-gnu/libnss_compat-2.13.so
    7f1f8d58b000-7f1f8d708000 r-xp 00000000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7f1f8d708000-7f1f8d908000 ---p 0017d000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7f1f8d908000-7f1f8d90c000 r--p 0017d000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7f1f8d90c000-7f1f8d90d000 rw-p 00181000 fe:00 1704030 /lib/x86_64-linux-gnu/libc-2.13.so
    7f1f8d90d000-7f1f8d912000 rw-p 00000000 00:00 0
    7f1f8d912000-7f1f8d929000 r-xp 00000000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7f1f8d929000-7f1f8db28000 ---p 00017000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7f1f8db28000-7f1f8db29000 r--p 00016000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7f1f8db29000-7f1f8db2a000 rw-p 00017000 fe:00 1707370 /lib/x86_64-linux-gnu/libpthread-2.13.so
    7f1f8db2a000-7f1f8db2e000 rw-p 00000000 00:00 0
    7f1f8db2e000-7f1f8db39000 r-xp 00000000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7f1f8db39000-7f1f8dd39000 ---p 0000b000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7f1f8dd39000-7f1f8dd3a000 rw-p 0000b000 fe:00 1706109 /lib/x86_64-linux-gnu/libpopt.so.0.0.0
    7f1f8dd3a000-7f1f8dd3f000 r-xp 00000000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7f1f8dd3f000-7f1f8df3f000 ---p 00005000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7f1f8df3f000-7f1f8df40000 rw-p 00005000 fe:00 1572965 /usr/local/lib/liblttng-ctl.so.0.0.0
    7f1f8df40000-7f1f8df41000 rw-p 00000000 00:00 0
    7f1f8df41000-7f1f8df60000 r-xp 00000000 fe:00 1709027 /lib/x86_64-linux-gnu/ld-2.13.so
    7f1f8e130000-7f1f8e134000 rw-p 00000000 00:00 0
    7f1f8e15d000-7f1f8e160000 rw-p 00000000 00:00 0
    7f1f8e160000-7f1f8e161000 r--p 0001f000 fe:00 1709027 /lib/x86_64-linux-gnu/ld-2.13.so
    7f1f8e161000-7f1f8e162000 rw-p 00020000 fe:00 1709027 /lib/x86_64-linux-gnu/ld-2.13.so
    7f1f8e162000-7f1f8e163000 rw-p 00000000 00:00 0
    7fff16f47000-7fff16f68000 rw-p 00000000 00:00 0 [stack]
    7fff16f8f000-7fff16f90000 r-xp 00000000 00:00 0 [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
    Abandon (core dumped)
#2

Updated by Yannick Brosseau about 8 years ago

The backtrace in the core:
Core was generated by `lttng list -u'.
Program terminated with signal 6, Aborted.
#0 0x00007f1f8d5bd475 in *GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: Aucun fichier ou dossier de ce type.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0 0x00007f1f8d5bd475 in *_GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f1f8d5c06f0 in *
_GI_abort () at abort.c:92
#2 0x00007f1f8d5f726b in _libc_message (do_abort=<optimized out>, fmt=<optimized out>)
at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3 0x00007f1f8d600ab6 in malloc_printerr (action=3, str=0x7f1f8d6d486a "free(): invalid pointer", ptr=<optimized out>)
at malloc.c:6283
#4 0x00007f1f8d6057ec in *
_GI
_libc_free (mem=<optimized out>) at malloc.c:3738
#5 0x0000000000403c69 in list_ust_events () at commands/list.c:286
#6 cmd_list (argc=<optimized out>, argv=<optimized out>) at commands/list.c:634
#7 0x00000000004023f2 in handle_command (argv=0x7fff16f663f0, argc=2) at lttng.c:268
#8 parse_args (argv=<optimized out>, argc=3) at lttng.c:486
#9 main (argc=3, argv=<optimized out>) at lttng.c:537

#3

Updated by David Goulet about 8 years ago

  • Status changed from New to Feedback

Can you clean everything in /usr/local/bin and lib/ and tell me if you hit the problem again... This is most unlikely since the pointer being free is either valid (pointing to allocated memory) or NULL so it should not fail..

If you can reproduce it after a cleanup, try this small patch and see if it works:

src/bin/lttng/commands/list.c +249

- struct lttng_event *event_list;
+ struct lttng_event *event_list = NULL;

#4

Updated by Yannick Brosseau about 8 years ago

I cannot reproduce this problem for now, but I have another problem with UST. I'll open another bug

#5

Updated by David Goulet about 8 years ago

  • Status changed from Feedback to New
#6

Updated by Anonymous about 8 years ago

  • Target version changed from 2.0 RC1 to 2.0 stable
#7

Updated by Yannick Brosseau about 8 years ago

  • Assignee changed from David Goulet to Mathieu Desnoyers
#8

Updated by Mathieu Desnoyers about 8 years ago

  • Assignee changed from Mathieu Desnoyers to David Goulet
#9

Updated by Mathieu Desnoyers about 8 years ago

  • Status changed from New to Resolved

lots of improvement to stability since report, probably good now. Cannot reproduce for the past 29 days.

Also available in: Atom PDF