LTTng » LTTng-UST

I'm running a stress test launching 3000 applications emitting 100 events per second during 20 seconds each which causes the consumerd to segfault. While that is reproducible somewhat easily (that's the problem I was trying to debug), I just ran into a case that causes both one of the traced applications and the consumer daemon to segfault.

Traced application backtrace follows:

Core was generated by `./TestApp_100perSecOnly 20 np'.
Program terminated with signal 11, Segmentation fault.
#0  print_cmd (handle=1634037881, cmd=1935763820) at lttng-ust-comm.c:212
212        if (cmd_name_mapping[cmd]) {
(gdb) bt
#0  print_cmd (handle=1634037881, cmd=1935763820) at lttng-ust-comm.c:212
#1  ust_listener_thread (arg=0x7f3314c19b20 <local_apps>) at lttng-ust-comm.c:1066
#2  0x00007f3313d8cdd2 in start_thread () from /usr/lib/libpthread.so.0
#3  0x00007f33132a5ced in clone () from /usr/lib/libc.so.6

(gdb) up
#1  ust_listener_thread (arg=0x7f3314c19b20 <local_apps>) at lttng-ust-comm.c:1066
1066    
(gdb) print lum
$13 = {
  handle = 1634037881, 
  cmd = 1935763820, 
  padding = " integer { size = 27; align = 1;", 
  u = {
    channel = {
      len = 2334102031740531488, 
      type = (LTTNG_UST_CHAN_METADATA | unknown: 1634082876), 
      padding = "lse; } := uint27_t;\n\ntrace {\n\tmajor = 1;\n\tminor = 8;\n\tuuid = \"f3a29a7d-7c01-4dfd-b463-696c4884ec49\";\n\tbyte_order = le;\n\tpacket.header := struct {\n\t\tuint32_t magic;\n\t\tuint8_t  uuid[16];\n\t\tuint32_t stre"..., 
      data = 0x7f33129bda54 "\n\ttracer_major = 2;\n\ttracer_minor = 2;\n\ttracer_patchlevel = 0;\n\tvpid = 18492;\n\tprocname = \"TestApp_100perS\";\n};\n\nclock {\n\tname = monotonic;\n\tuuid = \"6a86dfb3-e819-4f9b-a6c2-b31292b16173\";\n\tdescription"...
    }, 
    stream = {
      len = 2334102031740531488, 
      stream_nr = 1634082877, 
      padding = "lse; } := uint27_t;\n\ntrace {\n\tmajor = 1;\n\tminor = 8;\n\tuuid = \"f3a29a7d-7c01-4dfd-b463-696c4884ec49\";\n\tbyte_order = le;\n\tpacket.header := struct {\n\t\tuint32_t magic;\n\t\tuint8_t  uuid[16];\n\t\tuint32_t stre"...
    }, 
    event = {
      instrumentation = (unknown: 1734964000), 
      name = "ned = false; } := uint27_t;\n\ntrace {\n\tmajor = 1;\n\tminor = 8;\n\tuuid = \"f3a29a7d-7c01-4dfd-b463-696c4884ec49\";\n\tbyte_order = le;\n\tpacket.header := struct {\n\t\tuint32_t magic;\n\t\tuint8_t  uuid[16];\n\t\tuint3"..., 
      loglevel_type = (LTTNG_UST_LOGLEVEL_RANGE | unknown: 544106848), 
      loglevel = 1965170749, 
      padding = "st\";\n\ttracer_nam", 
      u = {
        padding = "e = \"lttng-ust\";\n\ttracer_major = 2;\n\ttracer_minor = 2;\n\ttracer_patchlevel = 0;\n\tvpid = 18492;\n\tprocname = \"TestApp_100perS\";\n};\n\nclock {\n\tname = monotonic;\n\tuuid = \"6a86dfb3-e819-4f9b-a6c2-b31292b1617"...
      }
    }, 
    context = {
      ctx = 1734964000, 
      padding = "ned = false; } :", 
      u = {
        padding = "= uint27_t;\n\ntrace {\n\tmajor = 1;\n\tminor = 8;\n\tuuid = \"f3a29a7d-7c01-4dfd-b463-696c4884ec49\";\n\tbyte_order = le;\n\tpacket.header := struct {\n\t\tuint32_t magic;\n\t\tuint8_t  uuid[16];\n\t\tuint32_t stream_id;\n\t"...
      }
    }, 
    version = {
      major = 1734964000, 
      minor = 543450478, 
      patchlevel = 1634082877
    }, 
    tracepoint = {
      name = " signed = false; } := uint27_t;\n\ntrace {\n\tmajor = 1;\n\tminor = 8;\n\tuuid = \"f3a29a7d-7c01-4dfd-b463-696c4884ec49\";\n\tbyte_order = le;\n\tpacket.header := struct {\n\t\tuint32_t magic;\n\t\tuint8_t  uuid[16];\n\t\tu"..., 
      loglevel = 1836016649, 
      padding = "ain = \"ust\";\n\ttr" 
    }, 
    filter = {
      data_size = 1734964000, 
      reloc_offset = 543450478, 
      seqnum = 4279953930213269565
    }, 
    padding = " signed = false; } := uint27_t;\n" 
  }
}

I will submit a patch to check the command against the size of the command string array to make sure an invalid command does not trigger an out-of-bounds error. There is unfortunately no way to know if a command really is invalid or just "unknown"...

Maybe we should log the command's ID in such cases to make corrupted messages easier to spot?