LTTng » Babeltrace

I've been getting core dumps in semi-random fashion (first run okay, second and later runs fail in the exact same place, or sometimes thefirst run is okay, the second fails in a different spot, the third fails again in the "stable" spot...), and I finally caught one from within PyCharm's terminal:

~/PycharmProjects/untitled$ python3 ./ltrace_100.py > ./ltrace_100_analysis-085-096.5

(process:3567): GLib-CRITICAL **: g_hash_table_insert_internal: assertion `hash_table != NULL' failed
*** glibc detected *** python3: free(): invalid pointer: 0x00000000015462e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7da26)[0x7f1039139a26]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x372fa)[0x7f10384472fa]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_hash_table_destroy+0xe)[0x7f10384482ce]
/usr/local/lib/libbabeltrace.so.1(bt_context_put+0x1f)[0x7f103aa2277f]
/usr/local/lib/libbabeltrace-ctf.so.1(bt_ctf_iter_destroy+0xc5)[0x7f1038840a75]
/usr/local/lib/python3.2/dist-packages/_babeltrace.so(+0xc4c5)[0x7f103a8434c5]
python3(PyEval_EvalFrameEx+0x681)[0x458121]
python3[0x51b219]
python3(PyEval_EvalFrameEx+0x954)[0x4583f4]
python3[0x51b219]
python3(PyEval_EvalFrameEx+0x954)[0x4583f4]
python3(PyEval_EvalCodeEx+0x17d)[0x45fdfd]
python3(PyEval_EvalCode+0x3b)[0x46097b]
python3[0x480a65]
python3(PyRun_FileExFlags+0x91)[0x482511]
python3(PyRun_SimpleFileExFlags+0x199)[0x482f79]
python3(Py_Main+0x403)[0x493cd3]
python3(main+0xb7)[0x41d377]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f10390dd7ed]
python3[0x41ffc9]
======= Memory map: ========
00400000-00659000 r-xp 00000000 08:01 4861904                            /usr/bin/python3.2mu
00858000-00859000 r--p 00258000 08:01 4861904                            /usr/bin/python3.2mu
00859000-008d0000 rw-p 00259000 08:01 4861904                            /usr/bin/python3.2mu
008d0000-008ef000 rw-p 00000000 00:00 0 
011e7000-04307000 rw-p 00000000 00:00 0                                  [heap]
7f1037bb0000-7f1037bc5000 r-xp 00000000 08:01 4199886                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f1037bc5000-7f1037dc4000 ---p 00015000 08:01 4199886                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f1037dc4000-7f1037dc5000 r--p 00014000 08:01 4199886                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f1037dc5000-7f1037dc6000 rw-p 00015000 08:01 4199886                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f1037dc6000-7f1037dcd000 r-xp 00000000 08:01 4201994                    /lib/x86_64-linux-gnu/librt-2.15.so
7f1037dcd000-7f1037fcc000 ---p 00007000 08:01 4201994                    /lib/x86_64-linux-gnu/librt-2.15.so
7f1037fcc000-7f1037fcd000 r--p 00006000 08:01 4201994                    /lib/x86_64-linux-gnu/librt-2.15.so
7f1037fcd000-7f1037fce000 rw-p 00007000 08:01 4201994                    /lib/x86_64-linux-gnu/librt-2.15.so
7f1037fce000-7f103800a000 r-xp 00000000 08:01 4195390                    /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f103800a000-7f1038209000 ---p 0003c000 08:01 4195390                    /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f1038209000-7f103820a000 r--p 0003b000 08:01 4195390                    /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f103820a000-7f103820b000 rw-p 0003c000 08:01 4195390                    /lib/x86_64-linux-gnu/libpcre.so.3.12.1
7f103820b000-7f103820f000 r-xp 00000000 08:01 4194659                    /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f103820f000-7f103840e000 ---p 00004000 08:01 4194659                    /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f103840e000-7f103840f000 r--p 00003000 08:01 4194659                    /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f103840f000-7f1038410000 rw-p 00004000 08:01 4194659                    /lib/x86_64-linux-gnu/libuuid.so.1.3.0
7f1038410000-7f1038502000 r-xp 00000000 08:01 4194525                    /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4
7f1038502000-7f1038702000 ---p 000f2000 08:01 4194525                    /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4
7f1038702000-7f1038703000 r--p 000f2000 08:01 4194525                    /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4
7f1038703000-7f1038704000 rw-p 000f3000 08:01 4194525                    /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4
7f1038704000-7f1038746000 rw-p 00000000 00:00 0 
7f1038833000-7f103886b000 r-xp 00000000 08:01 5112342                    /usr/local/lib/libbabeltrace-ctf.so.1.0.0
7f103886b000-7f103886c000 ---p 00038000 08:01 5112342                    /usr/local/lib/libbabeltrace-ctf.so.1.0.0
7f103886c000-7f103886d000 r--p 00038000 08:01 5112342                    /usr/local/lib/libbabeltrace-ctf.so.1.0.0
7f103886d000-7f103886e000 rw-p 00039000 08:01 5112342                    /usr/local/lib/libbabeltrace-ctf.so.1.0.0
7f103888b000-7f10389d0000 rw-p 00000000 00:00 0 
7f10389d0000-7f10390bc000 r--p 00000000 08:01 4855197                    /usr/lib/locale/locale-archive
7f10390bc000-7f1039270000 r-xp 00000000 08:01 4202151                    /lib/x86_64-linux-gnu/libc-2.15.so
7f1039270000-7f103946f000 ---p 001b4000 08:01 4202151                    /lib/x86_64-linux-gnu/libc-2.15.so
7f103946f000-7f1039473000 r--p 001b3000 08:01 4202151                    /lib/x86_64-linux-gnu/libc-2.15.so
7f1039473000-7f1039475000 rw-p 001b7000 08:01 4202151                    /lib/x86_64-linux-gnu/libc-2.15.so
7f1039475000-7f103947a000 rw-p 00000000 00:00 0 
7f103947a000-7f1039575000 r-xp 00000000 08:01 4202214                    /lib/x86_64-linux-gnu/libm-2.15.so
7f1039575000-7f1039774000 ---p 000fb000 08:01 4202214                    /lib/x86_64-linux-gnu/libm-2.15.so
7f1039774000-7f1039775000 r--p 000fa000 08:01 4202214                    /lib/x86_64-linux-gnu/libm-2.15.so
7f1039775000-7f1039776000 rw-p 000fb000 08:01 4202214                    /lib/x86_64-linux-gnu/libm-2.15.so
7f1039776000-7f103979d000 r-xp 00000000 08:01 4195393                    /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f103979d000-7f103999d000 ---p 00027000 08:01 4195393                    /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f103999d000-7f103999f000 r--p 00027000 08:01 4195393                    /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f103999f000-7f10399a0000 rw-p 00029000 08:01 4195393                    /lib/x86_64-linux-gnu/libexpat.so.1.5.2
7f10399a0000-7f10399b6000 r-xp 00000000 08:01 4199976                    /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f10399b6000-7f1039bb5000 ---p 00016000 08:01 4199976                    /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f1039bb5000-7f1039bb6000 r--p 00015000 08:01 4199976                    /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f1039bb6000-7f1039bb7000 rw-p 00016000 08:01 4199976                    /lib/x86_64-linux-gnu/libz.so.1.2.3.4
7f1039bb7000-7f1039d6a000 r-xp 00000000 08:01 4195402                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f1039d6a000-7f1039f69000 ---p 001b3000 08:01 4195402                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f1039f69000-7f1039f84000 r--p 001b2000 08:01 4195402                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f1039f84000-7f1039f8f000 rw-p 001cd000 08:01 4195402                    /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f1039f8f000-7f1039f93000 rw-p 00000000 00:00 0 
7f1039f93000-7f1039fe8000 r-xp 00000000 08:01 4195409                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f1039fe8000-7f103a1e7000 ---p 00055000 08:01 4195409                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f103a1e7000-7f103a1ea000 r--p 00054000 08:01 4195409                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f103a1ea000-7f103a1f1000 rw-p 00057000 08:01 4195409                    /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f103a1f1000-7f103a1f3000 r-xp 00000000 08:01 4202147                    /lib/x86_64-linux-gnu/libutil-2.15.so
7f103a1f3000-7f103a3f2000 ---p 00002000 08:01 4202147                    /lib/x86_64-linux-gnu/libutil-2.15.so
7f103a3f2000-7f103a3f3000 r--p 00001000 08:01 4202147                    /lib/x86_64-linux-gnu/libutil-2.15.so
7f103a3f3000-7f103a3f4000 rw-p 00002000 08:01 4202147                    /lib/x86_64-linux-gnu/libutil-2.15.so
7f103a3f4000-7f103a3f6000 r-xp 00000000 08:01 4195424                    /lib/x86_64-linux-gnu/libdl-2.15.so
7f103a3f6000-7f103a5f6000 ---p 00002000 08:01 4195424                    /lib/x86_64-linux-gnu/libdl-2.15.so
7f103a5f6000-7f103a5f7000 r--p 00002000 08:01 4195424                    /lib/x86_64-linux-gnu/libdl-2.15.so
7f103a5f7000-7f103a5f8000 rw-p 00003000 08:01 4195424                    /lib/x86_64-linux-gnu/libdl-2.15.so
7f103a5f8000-7f103a610000 r-xp 00000000 08:01 4202167                    /lib/x86_64-linux-gnu/libpthread-2.15.so
7f103a610000-7f103a80f000 ---p 00018000 08:01 4202167                    /lib/x86_64-linux-gnu/libpthread-2.15.so
7f103a80f000-7f103a810000 r--p 00017000 08:01 4202167                    /lib/x86_64-linux-gnu/libpthread-2.15.so
7f103a810000-7f103a811000 rw-p 00018000 08:01 4202167                    /lib/x86_64-linux-gnu/libpthread-2.15.so
7f103a811000-7f103a815000 rw-p 00000000 00:00 0 
7f103a815000-7f103a837000 r-xp 00000000 08:01 4202180                    /lib/x86_64-linux-gnu/ld-2.15.so
7f103a837000-7f103a856000 r-xp 00000000 08:01 5124211                    /usr/local/lib/python3.2/dist-packages/_babeltrace.so.0.0.0
7f103a856000-7f103a857000 r--p 0001e000 08:01 5124211                    /usr/local/lib/python3.2/dist-packages/_babeltrace.so.0.0.0
7f103a857000-7f103a85b000 rw-p 0001f000 08:01 5124211                    /usr/local/lib/python3.2/dist-packages/_babeltrace.so.0.0.0
7f103a85b000-7f103aa18000 rw-p 00000000 00:00 0 
7f103aa1e000-7f103aa2a000 r-xp 00000000 08:01 5112285                    /usr/local/lib/libbabeltrace.so.1.0.0
7f103aa2a000-7f103aa2b000 r--p 0000b000 08:01 5112285                    /usr/local/lib/libbabeltrace.so.1.0.0
7f103aa2b000-7f103aa2c000 rw-p 0000c000 08:01 5112285                    /usr/local/lib/libbabeltrace.so.1.0.0
7f103aa2d000-7f103aa2e000 rw-p 00000000 00:00 0 
7f103aa2e000-7f103aa35000 r--s 00000000 08:01 5127521                    /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f103aa35000-7f103aa37000 rw-p 00000000 00:00 0 
7f103aa37000-7f103aa38000 r--p 00022000 08:01 4202180                    /lib/x86_64-linux-gnu/ld-2.15.so
7f103aa38000-7f103aa3a000 rw-p 00023000 08:01 4202180                    /lib/x86_64-linux-gnu/ld-2.15.so
7fffbd6f6000-7fffbd717000 rw-p 00000000 00:00 0                          [stack]
7fffbd7fe000-7fffbd800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Abandon (core dumped)

Python 3.2.3 (default, Jun 18 2015, 21:46:58) [GCC 4.6.3] on linux2
Ubuntu 12.04.5 LTS Precise
3.9.3 kernel
PyCharm Community Edition 2016.1.3
Build #PC-145.971, built on May 3, 2016
JRE: 1.8.0_76-release-b32 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

The ltrace_100.py is attached, I can supply the trace dataset too though that's some 6.3 GB. The core dump regularly occurs with the ffx-session-20160713-165412.0 trace, while outputting TID 10018, just before the syscall_exit_getpriority report. The one caught here occurred with the ffx-session-20160713-171045.0 trace, TID 10575, syscall_entry_madvise report. It seems to be triggerred by the Python garbage collector, tripping the bt_ctf iterator destructor. This may explain its semi-regularity.