LTTng » LTTng-tools

Description: 
============ 
When using non-default ports for streaming, the ports remain hanging in CLOSE_WAIT state forever even after "lttng destroy" and the remote side has kill the corresponding relayd process.

Commit used: 
============ 
userspace  : (oct12) 1fe734e wfcqueue: clarify locking usage
lttng-ust  : (oct09) 1c7b4a9 Fix: memcpy of string is larger than source
ttng-tools: (oct02)  4dbc372 ABI with support for compat 32/64 bits
babeltrace : (oct02) 052606b Document that list.h is LGPLv2.1, but entirely trivial

Scenario: 
========= 
  On Node-1:
    1a)_ netstat -etan
    1b)_ lttng-relayd -vvv -C tcp://0.0.0.0:51234 -D tcp://0.0.0.0:51235  &
    1c)_ Once the session has been started on Node-2:
         netstat -etan gives:
         Proto Recv-Q Send-Q Local Address     Foreign Address          State       User Inode
         tcp        0      0 0.0.0.0:51234      0.0.0.0:*               LISTEN      0    1625723
         tcp        0      0 0.0.0.0:51235      0.0.0.0:*               LISTEN      0    1625724
         tcp        0      0 192.168.0.1:51235  192.168.0.4:58137       ESTABLISHED 0    1625875
         tcp        0      0 192.168.0.1:51235  192.168.0.4:58139       ESTABLISHED 0    1625975
         tcp        0      0 192.168.0.1:51234  192.168.0.4:41174       ESTABLISHED 0    1625974
         tcp        0      0 192.168.0.1:51234  192.168.0.4:41172       ESTABLISHED 0    1625874

    1d)_ Once the session has been destroyed from Node-2,
         pkill lttng-relayd
         netstat -etan gives:
         Proto Recv-Q Send-Q Local Address     Foreign Address          State       User Inode
         tcp        0      0 192.168.0.1:51235  192.168.0.4:58137       FIN_WAIT2   0    1625875
         tcp        0      0 192.168.0.1:51235  192.168.0.4:58139       FIN_WAIT2   0    1625975
         tcp        0      0 192.168.0.1:51234  192.168.0.4:41174       FIN_WAIT2   0    1625974
         tcp        0      0 192.168.0.1:51234  192.168.0.4:41172       FIN_WAIT2   0    1625874

    1e)_ About 30 sec later:
         The above ports pair nolonger appear in netstat

  On Node-2:
    2a)_ <run an instrumented app>
    2b)_ netstat -etan
    2c)_ lttng create ses1
    2d)_ lttng enable-event an_event_from_the_running_instrumented_app -u
    2e)_ lttng enable-consumer -u -s ses1 -C tcp://192.168.0.1:51234 -D tcp://192.168.0.1:51235 -e
    2f)_ lttng start; sleep 20; lttng stop
    2g)_ netstat -etan
         Proto Recv-Q Send-Q Local Address     Foreign Address    State       User Inode
         tcp        0      0 192.168.0.4:1128  0.0.0.0:*          LISTEN      0    22611
         tcp        0      0 192.168.0.4:1130  0.0.0.0:*          LISTEN      0    20916
         tcp        0      0 192.168.0.4:1131  0.0.0.0:*          LISTEN      0    22141
         tcp        0      0 0.0.0.0:111       0.0.0.0:*          LISTEN      0    20739
         tcp        0      0 0.0.0.0:22        0.0.0.0:*          LISTEN      0    22237
         tcp        0      0 192.168.0.4:1022  0.0.0.0:*          LISTEN      0    20952
         tcp        0      0 192.168.0.4:41172 192.168.0.1:51234  ESTABLISHED 0    1474170
         tcp        0      0 192.168.0.4:58139 192.168.0.1:51235  ESTABLISHED 0    1477457
         tcp        0      0 192.168.0.4:41174 192.168.0.1:51234  ESTABLISHED 0    1477456
         tcp        0      0 192.168.0.4:58137 192.168.0.1:51235  ESTABLISHED 0    1477398

    2h)_ lttng destroy; lttng list 
    2i)_ From Node-1, kill the corresponding relayd .

    2j)_ netstat -etan
         Proto Recv-Q Send-Q Local Address     Foreign Address    State       User Inode
         tcp        0      0 192.168.0.4:1128  0.0.0.0:*          LISTEN      0    22611
         tcp        0      0 192.168.0.4:1130  0.0.0.0:*          LISTEN      0    20916
         tcp        0      0 192.168.0.4:1131  0.0.0.0:*          LISTEN      0    22141
         tcp        0      0 0.0.0.0:111       0.0.0.0:*          LISTEN      0    20739
         tcp        0      0 0.0.0.0:22        0.0.0.0:*          LISTEN      0    22237
         tcp        0      0 192.168.0.4:1022  0.0.0.0:*          LISTEN      0    20952
         tcp        0      0 192.168.0.4:41172 192.168.0.1:51234  CLOSE_WAIT  0    1474170
         tcp        0      0 192.168.0.4:58139 192.168.0.1:51235  CLOSE_WAIT  0    1477457
         tcp        0      0 192.168.0.4:41174 192.168.0.1:51234  CLOSE_WAIT  0    1477456
         tcp        0      0 192.168.0.4:58137 192.168.0.1:51235  CLOSE_WAIT  0    1477398

    2k)_ The above port-pair remain CLOSE_WAIT forever (still there after 24hrs).
         When a new session is created using the same -C and -D, 
         new set of port-pair being created and they eventually hang in CLOSE_WAIT as well.