Bug #1411
closedMemory leak when relay daemon exits before application starts
0%
Description
When the relay daemon is shutdown after creating a live session but before applications are started the shared memory allocated for tracing appears to remain and new memory is allocated for every application start.
How to reproduce:
host:~# lttng create micke --live
Spawning a session daemon
Spawning a relayd daemon
Live session micke created.
Traces will be output to tcp4://127.0.0.1:5342/ [data: 5343]
Live timer interval set to 1000000 us
host:~# lttng enable-event --userspace --all
All ust events are enabled in channel channel0
host:~# lttng start
Tracing started for session micke
host:~# killall -9 lttng-relayd
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 207Mi 15Gi 572Ki 320Mi 15Gi
Swap: 0B 0B 0B
host:~# ./micke-lttng
Mikael LTTNG 2015 - Starting
Mikael LTTNG 2015 - Signing out
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 248Mi 15Gi 40Mi 360Mi 15Gi
Swap: 0B 0B 0B
host:~# ./micke-lttng
Mikael LTTNG 2015 - Starting
Mikael LTTNG 2015 - Signing out
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 288Mi 15Gi 80Mi 400Mi 15Gi
Swap: 0B 0B 0B
host:~# lttng destroy micke
Destroying session micke..
Session micke destroyed
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 289Mi 15Gi 80Mi 400Mi 15Gi
Swap: 0B 0B 0B
host:~#
Version:
lttng-tools 2.13.11
Analyzis:
It seems that when the first application of a session starts, after the relay daemon has been shutdown, a failure to transfer streams to the relay deamon triggers a clean up through a call to ust_consumer_destroy_channel. But it appears that the cleanup isn't complete and the channel reference count remains incremented. Decrementing the reference count appears to be blocked in clean_channel_stream_list by stream->monitor = 0; preventing CONSUMER_CHANNEL_DEL from reaching consumer_del_channel(chan);
Information has it that this problem is NOT reproduced on 2.13 but I haven't tested that myself
Updated by Kienan Stewart 8 months ago
- Status changed from New to Feedback
- Assignee set to Mikael Beckius
Hi Mikael,
could you clarify what "./micke-lttng" is?
could you also clarify which versions of lttng-tools, lttng-ust, and liburcu are being used?
Information has it that this problem is NOT reproduced on 2.13 but I haven't tested that myself
Earlier you wrote
Version:
lttng-tools 2.13.11
thanks,
kienan
Updated by Mikael Beckius 8 months ago
I was supposed to add that the test application is just a simple app writing an lttng trace:
lttng_ust_tracepoint(micke_lttng, micke_tracepoint_count, timeSinceBoot, "Time since boot: ");
but I forgot to and I couldn't see that I could change the description.
could you also clarify which versions of lttng-tools, lttng-ust, and liburcu are being used?
Information has it that this problem is NOT reproduced on 2.13 but I haven't tested that myself
Ah, that's a typo. It should say 2.12.
UST: 2.13.6
URCU: 0.13.3
Micke
Updated by Jérémie Galarneau 8 months ago
- Assignee changed from Mikael Beckius to Jérémie Galarneau
- Target version set to 2.13
Updated by Mikael Beckius 8 months ago
Hello Jérémie!
Removing that line was the first thing I tried after understanding the logic of the cleanup but just to be sure, this is the result after updating lttng-tools on a clean build:
host:~# lttng create micke --live
Spawning a session daemon
Spawning a relayd daemon
Live session micke created.
Traces will be output to tcp4://127.0.0.1:5342/ [data: 5343]
Live timer interval set to 1000000 us
host:~# lttng enable-event --userspace --all
All ust events are enabled in channel channel0
host:~# lttng start
Tracing started for session micke
host:~# killall -9 lttng-relayd
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 133Mi 15Gi 0.0Ki 444Mi 15Gi
Swap: 0B 0B 0B
host:~# ./micke-lttng
Mikael LTTNG 2015 - Starting
Mikael LTTNG 2015 - Signing out
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 132Mi 15Gi 0.0Ki 444Mi 15Gi
Swap: 0B 0B 0B
host:~# ./micke-lttng
Mikael LTTNG 2015 - Starting
Mikael LTTNG 2015 - Signing out
host:~# free -h
total used free shared buff/cache available
Mem: 15Gi 132Mi 15Gi 0.0Ki 444Mi 15Gi
Swap: 0B 0B 0B
host:~#
Now "shared" is always at zero and "used" remains the same.
Micke
Updated by Jérémie Galarneau 8 months ago
That's good to hear, thanks for confirming!