Project

General

Profile

Bug #615

consumerd crashes during buffer snapshot

Added by Stanislav Vovk over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
Start date:
08/13/2013
Due date:
% Done:

0%

Estimated time:

Description

Commit IDs

tools: 81ea21bfd975bdf60dd30229a4caece7c98a2912
ust:   42f209e98d68936df04ec947a12e684155ce7cfe
rcu:   a5bae03d59c22e57263b8610b9fed99738303cf0

Problem description
consumerd crashes when 'lttng snapshot record' is executed. The crash only happens when an instrumented program is/has been run as non-privileged user.

System setup
sessiond is started as root and configured with following commands,

lttng create s1 --snapshot -o /trace/
lttng enable-channel ch -u -s s1
lttng enable-event -u -a -c ch
lttng start

There is a user Bob, member in groups 'users' and 'tracing'.
Instrumented program is run as Bob, program can be killed or continue running, does not matter for fault reproduction.

Crash happens when 'lttng snapshot record -s s1' is executed. Crash is always reproducible.

A snapshot is dumped to output but is not readable. Seems like 'metadata' is missing.

root@du1:~# l /trace/snapshot-1-20130813-122306-2/ust/uid/1000/32-bit/      
total 4
drwxrwx--- 2 root root    0 Aug 13 12:23 ./
drwxrwx--- 3 root root    0 Aug 13 12:23 ../
-rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_0
-rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_1
-rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_2
-rw-rw---- 1 root root 4096 Aug 13 12:23 c_tri_3

Attaching debug info!


Files

lttng_dbg.txt (499 KB) lttng_dbg.txt sessiond debug output Stanislav Vovk, 08/13/2013 06:52 AM
consumerd-bt.txt (7.21 KB) consumerd-bt.txt gdb bt Stanislav Vovk, 08/13/2013 06:52 AM
ust-dbg.txt (47.2 KB) ust-dbg.txt Stanislav Vovk, 08/13/2013 06:52 AM
sesd-dbg.txt (6.44 KB) sesd-dbg.txt Stanislav Vovk, 08/19/2013 08:18 AM
#1

Updated by David Goulet over 6 years ago

  • Target version set to 2.3
#2

Updated by Mathieu Desnoyers over 6 years ago

I have not been able to reproduce it on my Intel machine.

We'll need more information from your end:

- The ownership and access rights of every directory under /trace (including /trace itself).
- Details on which user IDs are issuing the "lttng .." commands, including details about which groups they are in.

Can this be reproduced every time, or not ? It is reproduced right after lttng-sessiond is started, or is there extra interaction done by the user beforehand ?

Also, is the entire system 32-bit or 64-bit ? Moreover, I would guess this happens on PowerPC, right ?

We would also need to know the content of channel->key that triggers the assert at

lttng_ustconsumer_request_metadata (ctx=0x1004e4a8, channel=0x32f05d10, timer=0) at ust-consumer.c:1907

(within lttng-consumerd)

Thanks!

Mathieu

#3

Updated by Mathieu Desnoyers over 6 years ago

  • Status changed from New to Feedback
#4

Updated by Mathieu Desnoyers over 6 years ago

Moreover, do you have a "tracing" group on your system ?

Can you check if the following commit fixes your issue ? It is in lttng-tools master branch.

commit 360e82e5c9b78f970c5318b5862f1646bc9cd882
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Fri Aug 16 16:07:54 2013 -0400

    Fix: don't skip chmod if tracing group is not found

    Fixes #620

    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

Thanks,

Mathieu

#5

Updated by Stanislav Vovk over 6 years ago

I tried the above commit and still see same issue!

- '/trace' access rights, ownership
(tried with different output dirs, like /home/<user>, same issue still observed)

root@du1:# l /trace
total 4
drwxrwxrwt  2 root root    0 Aug 19 12:16 ./
drwxr-xr-x 23 root root 4096 Aug 14 14:33 ../

- 'lttng' commands are issued by root,

root@du1:~# groups
root

- Is reproducible every time. Sessiond is started and configured, nothing else is done with it before taking snapshots.

- System is entire 32-bit PowerPC.

- 'tracing' group exists and the non-privileged user is member of tracing.

sirpa@du1:~$ groups
users tracing

- Attaching sessiond debug prints for 'lttng snapshot record' with channel->key value

DEBUG1 [12305/12310]: assert(key(5) == channel->key(7)) (in lttng_ustconsumer_request_metadata() at ust-consumer.c:1908)
lttng-consumerd: ust-consumer.c:1909: lttng_ustconsumer_request_metadata: Assertion `key == channel->key' failed.

#6

Updated by David Goulet over 6 years ago

We found the issue and were able to reproduce it on our side.

I just pushed (upstream master branch) a commit that fixes this bug on our side. I'll wait on your ACK before closing this issue since it's quite critical and we want to make sure it is fixed on your side.

commit 567eb353c7f88e2fdaa106eb7e0a38dbb8717792
Author: David Goulet <>
Date: Mon Aug 19 12:08:06 2013 -0400

Fix: snapshot with multiple UIDs
This commit refs #615. Waiting on the confirmation of the author to
close it.
Signed-off-by: David Goulet &lt;&gt;
#7

Updated by Mathieu Desnoyers over 6 years ago

  • Subject changed from consumerd carshes during buffer snapshot to consumerd crashes during buffer snapshot

fix typo in title

#8

Updated by Stanislav Vovk over 6 years ago

Works great, thanks!
I don't see consumerd crash anymore. Snapshots can be recorded and read successfully. Tried with latest master on 'tools' and 'ust'.

One comment though regarding permissions on snapshot folder,
They are,

root@du1:~# l /trace/
total 4
drwxrwxrwt  3 root root    0 Aug 20 10:51 ./
drwxr-xr-x 23 root root 4096 Aug 14 14:33 ../
drwxrwx---  3 root root    0 Aug 20 10:51 snapshot-1-20130820-105136-2/

non-privileged user can not read/remove snapshot! Is this a miss configuration by me or why is it like that? A user should be able to at least read the snapshot, and remove it as well otherwise user will run out of space eventually!

Cheers,
Stanislav

#9

Updated by David Goulet over 6 years ago

The session was created by the user with UID 0 in this case (root) so the traces
are set with those rights.

#10

Updated by David Goulet over 6 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF