login as: root Using keyboard-interactive authentication. Password: Last login: Mon Mar 11 09:06:09 2013 from 142.133.180.91 SC-1:~ # source /cluster/temp/tdlt/bashrc_showtime [SC-1:Wed Mar 13 15:12:19~] # [SC-1:Wed Mar 13 15:12:19~] # alias dir='ls -ltr' [SC-1:Wed Mar 13 15:12:28~] # cd /cluster/dumps/ [SC-1:Wed Mar 13 15:12:34/cluster/dumps] # dir total 18068 -rw------- 1 root root 17780736 Feb 27 19:48 TestApp_Mini1.1205.SC-1.core -rw------- 1 root root 17645568 Feb 27 19:51 TestApp_Mini1.22036.SC-1.core -rw------- 1 root root 17645568 Feb 27 21:19 TestApp_Mini1.18265.SC-1.core -rw------- 1 root root 17645568 Feb 27 21:57 TestApp_Mini1.26561.SC-1.core -rw------- 1 root root 21942272 Feb 28 18:41 trace_c.12571.SC-1.core -rw------- 1 root root 17645568 Feb 28 20:28 TestApp_Mini1.12224.SC-1.core -rw------- 1 root root 17645568 Feb 28 23:03 TestApp_Mini1.10678.SC-1.core -rw------- 1 root root 17780736 Mar 4 23:18 TestApp_Mini1.24828.SC-1.core -rw------- 1 root root 17645568 Mar 5 00:55 TestApp_Mini1.7980.SC-1.core -rw------- 1 root root 17645568 Mar 5 03:23 TestApp_Mini1.22503.SC-1.core -rw------- 1 root root 20017152 Mar 5 05:54 trace_c.8873.SC-1.core -rw------- 1 root root 17645568 Mar 5 07:07 TestApp_Mini1.5912.SC-1.core -rw------- 1 root root 17645568 Mar 5 07:44 TestApp_Mini1.27950.SC-1.core -rw------- 1 root root 19673088 Mar 5 08:11 trace_c.12280.SC-1.core -rw------- 1 root root 37220352 Mar 13 01:17 lttng-consumerd.10182.SC-1.core -rw------- 1 root root 19460096 Mar 13 01:58 trace_c.19116.SC-1.core [SC-1:Wed Mar 13 15:12:35/cluster/dumps] # rpm -wa |grep -i trace -wa: unknown option [SC-1:Wed Mar 13 15:12:51/cluster/dumps] # rpm -qa |grep -i trace strace-32bit-4.5.18-10.20.1 ERIC-Trace-common-CXP9019681_1-1-R2A05 ERIC-TRACE_EA_SC-CXP9040241_1-debuginfo-1-R2A04 ERIC-TRACE_EA_SC-CXP9040241_1-1-R2A04 ERIC-Trace-P-CXP9019681_1-1-R2A05 ERIC-Trace-debuginfo-1-R2A05 ERIC-TRACE_EA_Common-CXP9040232_1-debuginfo-1-R2A04 strace-4.5.18-10.20.1 ERIC-TRACE_EA_Common-CXP9040232_1-1-R2A04 ERIC-Trace-C-CXP9019680_1-1-R2A05 [SC-1:Wed Mar 13 15:12:56/cluster/dumps] # gdb /usr/lib64/lttng/libexec/lttng-consumerd lttng-consumerd.10182.SC-1.core GNU gdb (GDB) SUSE (7.3-0.6.1) Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-suse-linux". For bug reporting instructions, please see: ... Reading symbols from /usr/lib64/lttng/libexec/lttng-consumerd...Reading symbols from /usr/lib/debug/usr/lib64/lttng/libexec/lttng-consumerd.debug...done. done. [New LWP 10185] [New LWP 10182] [New LWP 10184] [New LWP 10631] [New LWP 10186] Missing separate debuginfo for /lib64/libpopt.so.0 Try: zypper install -C "debuginfo(build-id)=895e7fadb3a3371343391d34280543ea4df24a03" Missing separate debuginfo for /lib64/libpthread.so.0 Try: zypper install -C "debuginfo(build-id)=bb81b1117fc93fc0eafb3e96eabfca4d8976c879" Missing separate debuginfo for /lib64/libc.so.6 Try: zypper install -C "debuginfo(build-id)=2636ed5ff526582a49ed9b6e982c231335db1620" Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2 Try: zypper install -C "debuginfo(build-id)=3d68b93f8701971da6133437486f3909223534f8" Missing separate debuginfo for /lib64/librt.so.1 Try: zypper install -C "debuginfo(build-id)=d2c6a2f5b4c35e0fb41d45e2be66c3b73b686a44" Missing separate debuginfo for Try: zypper install -C "debuginfo(build-id)=74dac034c5964f9441a4606642c4f7c8d4f82afa" Missing separate debuginfo for /lib64/libpopt.so.0 Try: zypper install -C "debuginfo(build-id)=895e7fadb3a3371343391d34280543ea4df24a03" Missing separate debuginfo for /lib64/libpthread.so.0 Try: zypper install -C "debuginfo(build-id)=bb81b1117fc93fc0eafb3e96eabfca4d8976c879" [Thread debugging using libthread_db enabled] Missing separate debuginfo for /lib64/libc.so.6 Try: zypper install -C "debuginfo(build-id)=2636ed5ff526582a49ed9b6e982c231335db1620" Missing separate debuginfo for /lib64/ld-linux-x86-64.so.2 Try: zypper install -C "debuginfo(build-id)=3d68b93f8701971da6133437486f3909223534f8" Missing separate debuginfo for /lib64/librt.so.1 Try: zypper install -C "debuginfo(build-id)=d2c6a2f5b4c35e0fb41d45e2be66c3b73b686a44" Core was generated by `lttng-consumerd --quiet -u --consumerd-cmd-sock /var/run/lttng/ustconsumerd64/c'. Program terminated with signal 11, Segmentation fault. #0 0x00007f960a2533f4 in pthread_mutex_lock () from /lib64/libpthread.so.0 (gdb) bt #0 0x00007f960a2533f4 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x0000000000404d09 in consumer_add_stream (stream=0xa000000, ht=0x627960) at consumer.c:585 #2 0x000000000040903a in consumer_thread_data_poll (data=0x626fd0) at consumer.c:2378 #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? () (gdb) bt full #0 0x00007f960a2533f4 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000000000404d09 in consumer_add_stream (stream=0xa000000, ht=0x627960) at consumer.c:585 ret = 0 relayd = 0x626fd0 __PRETTY_FUNCTION__ = "consumer_add_stream" __func__ = "consumer_add_stream" #2 0x000000000040903a in consumer_thread_data_poll (data=0x626fd0) at consumer.c:2378 pipe_readlen = 4 num_rdy = 1 num_hup = 0 high_prio = 0 ret = 0 i = 0 pollfd = 0x62efc0 local_stream = 0x62b8b0 new_stream = 0xa000000 nb_fd = 0 ctx = 0x626fd0 len = -11 __func__ = "consumer_thread_data_poll" #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 No symbol table info available. ---Type to continue, or q to quit--- #5 0x0000000000000000 in ?? () No symbol table info available. (gdb) (gdb) (gdb) (gdb) (gdb) thread apply all bt full Thread 5 (Thread 0x7f96088bc700 (LWP 10186)): #0 0x00007f9609fa4796 in poll () from /lib64/libc.so.6 No symbol table info available. #1 0x00000000004059f1 in lttng_consumer_poll_socket (consumer_sockpoll=0x7f96088bbc80) at consumer.c:981 num_rdy = 18 __func__ = "lttng_consumer_poll_socket" #2 0x0000000000409b66 in consumer_thread_sessiond_poll (data=0x626fd0) at consumer.c:2596 sock = 18 client_socket = -1 ret = 1 consumer_sockpoll = {{fd = 4, events = 3, revents = 0}, {fd = 18, events = 3, revents = 0}} ctx = 0x626fd0 __func__ = "consumer_thread_sessiond_poll" #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 4 (Thread 0x7f96080bb700 (LWP 10631)): #0 0x00007f9609faa689 in syscall () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f960a89f730 in call_rcu_wait (crdp=) at urcu-call-rcu-impl.h:204 ---Type to continue, or q to quit--- No locals. #2 call_rcu_thread (arg=) at urcu-call-rcu-impl.h:291 cbcount = 18446744073709551611 cbs = 0x0 crdp = 0x62efe0 rt = 0 ret = __func__ = "call_rcu_thread" #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 0x7f96098be700 (LWP 10184)): #0 0x00007f9609fae2f3 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x000000000041550c in compat_epoll_wait (events=0x7f96098bdc50, timeout=-1) at compat-epoll.c:225 ret = new_size = 160160976 __func__ = "compat_epoll_wait" #2 0x0000000000408440 in consumer_thread_metadata_poll (data=0x626fd0) at consumer.c:2106 ret = 1 i = 1 ---Type to continue, or q to quit--- pollfd = 402 revents = 16 nb_fd = 1 stream = 0x7f96000875d0 iter = {iter = {node = 0x7f96000875d8, next = 0x634d20}} node = 0x7f96000875d0 events = {epfd = 16, nb_fd = 9, alloc_size = 16, init_size = 2, events = 0x62ee60} ctx = 0x626fd0 len = -1 __func__ = "consumer_thread_metadata_poll" __PRETTY_FUNCTION__ = "consumer_thread_metadata_poll" #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 0x7f960aeb4900 (LWP 10182)): #0 0x00007f960a251f95 in pthread_join () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00000000004039fd in main (argc=7, argv=0x7fff69f7cb78) at lttng-consumerd.c:390 ret = 0 status = 0x418550 __func__ = "main" ---Type to continue, or q to quit--- Thread 1 (Thread 0x7f96090bd700 (LWP 10185)): #0 0x00007f960a2533f4 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #1 0x0000000000404d09 in consumer_add_stream (stream=0xa000000, ht=0x627960) at consumer.c:585 ret = 0 relayd = 0x626fd0 __PRETTY_FUNCTION__ = "consumer_add_stream" __func__ = "consumer_add_stream" #2 0x000000000040903a in consumer_thread_data_poll (data=0x626fd0) at consumer.c:2378 pipe_readlen = 4 num_rdy = 1 num_hup = 0 high_prio = 0 ret = 0 i = 0 pollfd = 0x62efc0 local_stream = 0x62b8b0 new_stream = 0xa000000 nb_fd = 0 ctx = 0x626fd0 len = -11 __func__ = "consumer_thread_data_poll" #3 0x00007f960a2517b6 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. ---Type to continue, or q to quit--- #4 0x00007f9609fadc6d in clone () from /lib64/libc.so.6 No symbol table info available. #5 0x0000000000000000 in ?? () No symbol table info available. (gdb) (gdb) (gdb) (gdb) (gdb) quit [SC-1:Wed Mar 13 15:14:09/cluster/dumps] # [SC-1:Wed Mar 13 15:22:47/cluster/dumps] # [SC-1:Wed Mar 13 15:22:47/cluster/dumps] # cmw-repository-list ERIC-TRACE_EA_SC-CXP9040241_1-R2A04 Used ERIC-COREMW_COMMON-CXP9017566_1-R5E02 Used ERIC-COREMW_SC-CXP9017565_1-R5E02 Used ERIC-COREMW_OPENSAF-CXP9017656_1-R5E02 Used ERIC-TRACE_EA_Common-CXP9040232_1-R2A04 Used ERIC-TraceP-CXP9019681_1-R2A05 Used ERIC-TraceC-CXP9019680_1-R2A05 Used [SC-1:Wed Mar 13 15:22:50/cluster/dumps] # cluster info Product Id: ANA90139/3 Version: R4C06 Distribution Adaptation: SLES Installation Type: RAM Source info: 6f0991cfa15e9601c40f45daa080095aa129429d [SC-1:Wed Mar 13 15:22:53/cluster/dumps] # cd /cluster/temp/tdlt/SDP/