LTTng » LTTng-tools

When a session is activated with a domain and an event within that domain, sessiond dies and coredumps.

The session activation has been done using the lttng API.
The provided log was produced by running sessiond/consumerd with verbose options.
The log also shows syslog and saflog output as well as print statements from the Trace common component.
NOTE: verbose mode is only shown before sessiond dies. After it gets relaunched, it goes back to normal.

It seems that when verbose mode is on, the core dump does not happen.
This test case activates a session in a cluster and the verbose mode was only used in the first node (SC-1).
So, I attached the core dumps from the other 3 nodes (SC-2, PL3, PL4):
rw------ 1 root root 67825664 Nov 15 15:47 lttng-sessiond.19509.SC-2.core
rw------ 1 root root 68014080 Nov 15 15:47 lttng-sessiond.12270.PL-3.core
rw------ 1 root root 67825664 Nov 15 15:47 lttng-sessiond.12155.PL-4.core

The session was created with the following expressions:
com_ericsson_cba_trace_testapp_lowtraf,TRACE_DEBUG(14)
com_ericsson_cba_trace_testapp_lowtraf:TenPerSecA,TRACE_DEBUG(14)

Note that this fault seems to happen exclusively when the two expressions are using a "higher" loglevel value.
That is, when using LOGLEVEL_ONLY and the respective log level that goes with the event, the fault did not occur.
For example:
com_ericsson_cba_trace_testapp_lowtraf
com_ericsson_cba_trace_testapp_lowtraf:TenPerSecA,TRACE_DEBUG_MODULE(10) LOGLEVEL_ONLY

The following part of the log captures the last messages seen from sessiond/consumerd:

Error: Error ustctl create event com_ericsson_cba_trace_testapp_lowtraf:TenPerSecA for app pid: 12491 with ret -1026
lttng-sessiond: ust-app.c:1277: create_ust_app_event: Assertion `ret != -LTTNG_UST_ERR_EXIST' failed.
DEBUG1: Incoming command on sock [in consumer_thread_sessiond_poll() at consumer.c:2505]
DEBUG1: Consumer received unexpected message size 0 (expects 4416) [in lttng_ustconsumer_recv_cmd() at ust-consumer.c:109]
PERROR: sendmsg: Broken pipe [in lttcomm_send_unix_sock() at unix.c:204]
DEBUG1: Communication interrupted on command socket [in consumer_thread_sessiond_poll() at consumer.c:2516]
DEBUG1: consumer_thread_receive_fds exiting [in consumer_thread_sessiond_poll() at consumer.c:2526]
DEBUG1: poll num_rdy : 1 [in consumer_thread_data_poll() at consumer.c:2251]
DEBUG1: consumer_data_pipe wake up [in consumer_thread_data_poll() at consumer.c:2275]
DEBUG1: Consumer delete flagged data stream [in validate_endpoint_status_data_stream() at consumer.c:1938]
DEBUG1: polling thread exiting [in consumer_thread_data_poll() at consumer.c:2403]
DEBUG1: Metadata event catched in thread [in consumer_thread_metadata_poll() at consumer.c:2030]
DEBUG1: Metadata thread pipe hung up [in consumer_thread_metadata_poll() at consumer.c:2051]
DEBUG1: Metadata poll thread exiting [in consumer_thread_metadata_poll() at consumer.c:2160]
PERROR: sendmsg: Broken pipe [in lttcomm_send_unix_sock() at unix.c:204]
DEBUG1: Consumer destroying it. Closing everything. [in lttng_consumer_destroy() at consumer.c:1196]

Build Info:
babeltrace: * 08ac0e0 (HEAD, origin/master, origin/HEAD) Fix SEEK_BEGIN for streams that do not contain any event
lttng-tools: * 94e60b1 (HEAD, origin/master, origin/HEAD) Fix: Warn if session is running with lttng view
lttng-ust: * dca6e79 (HEAD, origin/master, origin/HEAD) Remove LIBFORMAT config declaration, unused
userspace-rcu: * f864c15 (HEAD, origin/master, origin/HEAD) urcu-qsbr: skip Q.S. reporting if already reported