LTTng » LTTng-tools

I've found why the segfault happens. I'll try to submit you a patch as soon as possible based on master. Here is a small analysis.

When the endpoint of the trace hangs up or dies (e.g lttng-relayd), the streams associated with it are flagged with an INACTIVE endpoint status. After that, every streams are validated and, in this case, the metadata stream gets deleted because the relayd is gone however the close metadata command from the session daemon has not yet been triggered.

So, the stream is deleted, the close metadata command arrives but the stream is no longer available at that point. Now, why this stream is found is because we keep a reference to the metadata stream directly in the channel thus accessing without any RCU protection.

This coredump can also be seen with the following scenario:
  - Create a session using streaming (and use perUID buffer)
  - launch the instrumented app 
  - start the session
  - sleep a few sec
  - kill the corresponding relayd
  - coredump with very similar gdb printout.

Commits used:
userspace-rcu: 56e676d (HEAD, origin/stable-0.7) Document: rculfhash ...
lttng-ust    : 352fce3 (HEAD, origin/master, origin/HEAD) Remove 0.x TODO
lttng-tools  : b31398b (HEAD, origin/master, origin/HEAD) Fix: increment ...

 

Hi David,

So far it looks very promising.
After 3hrs of our test run, no coredump yet :-)
We will let that traffic run overnight and see if something else happen.
But for now, we can safely assumed that this bug has been fixed with the patch above.

Thank You very much for your support,
Tan