Bug #400


SessionD dies at session activation when domain and event collide

Added by Jesus Garcia over 11 years ago. Updated over 11 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


When a session is activated with a domain and an event within that domain, sessiond dies and coredumps.

The session activation has been done using the lttng API.
The provided log was produced by running sessiond/consumerd with verbose options.
The log also shows syslog and saflog output as well as print statements from the Trace common component.
NOTE: verbose mode is only shown before sessiond dies. After it gets relaunched, it goes back to normal.

It seems that when verbose mode is on, the core dump does not happen.
This test case activates a session in a cluster and the verbose mode was only used in the first node (SC-1).
So, I attached the core dumps from the other 3 nodes (SC-2, PL3, PL4):
rw------ 1 root root 67825664 Nov 15 15:47 lttng-sessiond.19509.SC-2.core
rw------ 1 root root 68014080 Nov 15 15:47 lttng-sessiond.12270.PL-3.core
rw------ 1 root root 67825664 Nov 15 15:47 lttng-sessiond.12155.PL-4.core

The session was created with the following expressions:

Note that this fault seems to happen exclusively when the two expressions are using a "higher" loglevel value.
That is, when using LOGLEVEL_ONLY and the respective log level that goes with the event, the fault did not occur.
For example:
com_ericsson_cba_trace_testapp_lowtraf:TenPerSecA,TRACE_DEBUG_MODULE(10) LOGLEVEL_ONLY

The following part of the log captures the last messages seen from sessiond/consumerd:

Error: Error ustctl create event com_ericsson_cba_trace_testapp_lowtraf:TenPerSecA for app pid: 12491 with ret -1026
lttng-sessiond: ust-app.c:1277: create_ust_app_event: Assertion `ret != -LTTNG_UST_ERR_EXIST' failed.
DEBUG1: Incoming command on sock [in consumer_thread_sessiond_poll() at consumer.c:2505]
DEBUG1: Consumer received unexpected message size 0 (expects 4416) [in lttng_ustconsumer_recv_cmd() at ust-consumer.c:109]
PERROR: sendmsg: Broken pipe [in lttcomm_send_unix_sock() at unix.c:204]
DEBUG1: Communication interrupted on command socket [in consumer_thread_sessiond_poll() at consumer.c:2516]
DEBUG1: consumer_thread_receive_fds exiting [in consumer_thread_sessiond_poll() at consumer.c:2526]
DEBUG1: poll num_rdy : 1 [in consumer_thread_data_poll() at consumer.c:2251]
DEBUG1: consumer_data_pipe wake up [in consumer_thread_data_poll() at consumer.c:2275]
DEBUG1: Consumer delete flagged data stream [in validate_endpoint_status_data_stream() at consumer.c:1938]
DEBUG1: polling thread exiting [in consumer_thread_data_poll() at consumer.c:2403]
DEBUG1: Metadata event catched in thread [in consumer_thread_metadata_poll() at consumer.c:2030]
DEBUG1: Metadata thread pipe hung up [in consumer_thread_metadata_poll() at consumer.c:2051]
DEBUG1: Metadata poll thread exiting [in consumer_thread_metadata_poll() at consumer.c:2160]
PERROR: sendmsg: Broken pipe [in lttcomm_send_unix_sock() at unix.c:204]
DEBUG1: Consumer destroying it. Closing everything. [in lttng_consumer_destroy() at consumer.c:1196]

Build Info:
babeltrace: * 08ac0e0 (HEAD, origin/master, origin/HEAD) Fix SEEK_BEGIN for streams that do not contain any event
lttng-tools: * 94e60b1 (HEAD, origin/master, origin/HEAD) Fix: Warn if session is running with lttng view
lttng-ust: * dca6e79 (HEAD, origin/master, origin/HEAD) Remove LIBFORMAT config declaration, unused
userspace-rcu: * f864c15 (HEAD, origin/master, origin/HEAD) urcu-qsbr: skip Q.S. reporting if already reported


sessiond_dies_colliding_domain_and_event.txt (55.1 KB) sessiond_dies_colliding_domain_and_event.txt Log Jesus Garcia, 11/15/2012 06:18 PM
lttng-sessiond.19509.SC-2.core.gz (128 KB) lttng-sessiond.19509.SC-2.core.gz sessiond core dump Jesus Garcia, 11/15/2012 06:18 PM
lttng-sessiond.12270.PL-3.core.gz (130 KB) lttng-sessiond.12270.PL-3.core.gz sessiond core dump Jesus Garcia, 11/15/2012 06:18 PM
lttng-sessiond.12155.PL-4.core.gz (128 KB) lttng-sessiond.12155.PL-4.core.gz sessiond core dump Jesus Garcia, 11/15/2012 06:18 PM
Actions #1

Updated by David Goulet over 11 years ago

  • Status changed from New to Feedback
  • Assignee set to David Goulet
  • Target version changed from 2.0.x stable to 2.1 stable

That would be interesting to confirm if this is reproducible with the new rc8 lttng-tools which supports overlapping events.


Actions #2

Updated by Jesus Garcia over 11 years ago

Hi David,

I was not able to reproduce this one with rc8.


Actions #3

Updated by David Goulet over 11 years ago

  • Status changed from Feedback to Invalid

Also available in: Atom PDF