Bug #615
closedconsumerd crashes during buffer snapshot
0%
Description
Commit IDs
tools: 81ea21bfd975bdf60dd30229a4caece7c98a2912 ust: 42f209e98d68936df04ec947a12e684155ce7cfe rcu: a5bae03d59c22e57263b8610b9fed99738303cf0
Problem description
consumerd crashes when 'lttng snapshot record' is executed. The crash only happens when an instrumented program is/has been run as non-privileged user.
System setup
sessiond is started as root and configured with following commands,
lttng create s1 --snapshot -o /trace/ lttng enable-channel ch -u -s s1 lttng enable-event -u -a -c ch lttng start
There is a user Bob, member in groups 'users' and 'tracing'.
Instrumented program is run as Bob, program can be killed or continue running, does not matter for fault reproduction.
Crash happens when 'lttng snapshot record -s s1' is executed. Crash is always reproducible.
A snapshot is dumped to output but is not readable. Seems like 'metadata' is missing.
root@du1:~# l /trace/snapshot-1-20130813-122306-2/ust/uid/1000/32-bit/ total 4 drwxrwx--- 2 root root 0 Aug 13 12:23 ./ drwxrwx--- 3 root root 0 Aug 13 12:23 ../ -rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_0 -rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_1 -rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_2 -rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_3
Attaching debug info!
Files
Updated by Mathieu Desnoyers over 11 years ago
I have not been able to reproduce it on my Intel machine.
We'll need more information from your end:
- The ownership and access rights of every directory under /trace (including /trace itself).
- Details on which user IDs are issuing the "lttng .." commands, including details about which groups they are in.
Can this be reproduced every time, or not ? It is reproduced right after lttng-sessiond is started, or is there extra interaction done by the user beforehand ?
Also, is the entire system 32-bit or 64-bit ? Moreover, I would guess this happens on PowerPC, right ?
We would also need to know the content of channel->key that triggers the assert at
lttng_ustconsumer_request_metadata (ctx=0x1004e4a8, channel=0x32f05d10, timer=0) at ust-consumer.c:1907
(within lttng-consumerd)
Thanks!
Mathieu
Updated by Mathieu Desnoyers over 11 years ago
- Status changed from New to Feedback
Updated by Mathieu Desnoyers over 11 years ago
Moreover, do you have a "tracing" group on your system ?
Can you check if the following commit fixes your issue ? It is in lttng-tools master branch.
commit 360e82e5c9b78f970c5318b5862f1646bc9cd882
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date: Fri Aug 16 16:07:54 2013 -0400
Fix: don't skip chmod if tracing group is not found
Fixes #620
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Thanks,
Mathieu
Updated by Stanislav Vovk over 11 years ago
- File sesd-dbg.txt sesd-dbg.txt added
I tried the above commit and still see same issue!
- '/trace' access rights, ownership
(tried with different output dirs, like /home/<user>, same issue still observed)
root@du1:# l /trace total 4 drwxrwxrwt 2 root root 0 Aug 19 12:16 ./ drwxr-xr-x 23 root root 4096 Aug 14 14:33 ../
- 'lttng' commands are issued by root,
root@du1:~# groups root
- Is reproducible every time. Sessiond is started and configured, nothing else is done with it before taking snapshots.
- System is entire 32-bit PowerPC.
- 'tracing' group exists and the non-privileged user is member of tracing.
sirpa@du1:~$ groups users tracing
- Attaching sessiond debug prints for 'lttng snapshot record' with channel->key value
DEBUG1 [12305/12310]: assert(key(5) == channel->key(7)) (in lttng_ustconsumer_request_metadata() at ust-consumer.c:1908) lttng-consumerd: ust-consumer.c:1909: lttng_ustconsumer_request_metadata: Assertion `key == channel->key' failed.
Updated by David Goulet about 11 years ago
We found the issue and were able to reproduce it on our side.
I just pushed (upstream master branch) a commit that fixes this bug on our side. I'll wait on your ACK before closing this issue since it's quite critical and we want to make sure it is fixed on your side.
commit 567eb353c7f88e2fdaa106eb7e0a38dbb8717792
Author: David Goulet <dgoulet@efficios.com>
Date: Mon Aug 19 12:08:06 2013 -0400
Fix: snapshot with multiple UIDsThis commit refs #615. Waiting on the confirmation of the author to
close it.Signed-off-by: David Goulet <dgoulet@efficios.com>Updated by Mathieu Desnoyers about 11 years ago
- Subject changed from consumerd carshes during buffer snapshot to consumerd crashes during buffer snapshot
fix typo in title
Updated by Stanislav Vovk about 11 years ago
Works great, thanks!
I don't see consumerd crash anymore. Snapshots can be recorded and read successfully. Tried with latest master on 'tools' and 'ust'.
One comment though regarding permissions on snapshot folder,
They are,
root@du1:~# l /trace/ total 4 drwxrwxrwt 3 root root 0 Aug 20 10:51 ./ drwxr-xr-x 23 root root 4096 Aug 14 14:33 ../ drwxrwx--- 3 root root 0 Aug 20 10:51 snapshot-1-20130820-105136-2/
non-privileged user can not read/remove snapshot! Is this a miss configuration by me or why is it like that? A user should be able to at least read the snapshot, and remove it as well otherwise user will run out of space eventually!
Cheers,
Stanislav
Updated by David Goulet about 11 years ago
The session was created by the user with UID 0 in this case (root) so the traces
are set with those rights.
Updated by David Goulet about 11 years ago
- Status changed from Feedback to Resolved