LTTng » LTTng-tools

I have 2 separate Docker containers. The user in each container is the same user as my non-root host user (same name, same UID, same GID). Users in the 2 Docker containers have the same $HOME, which is a directory mounted from the host.

To set up and configure each container:

1. Start container and pass all necessary info through:

host$ mkdir ~/fake-home
host$ docker run -it --env USER=$USER --env GROUP=$USER --env USER_ID=$(id -u) --env GROUP_ID=$(id -g) -v $HOME/fake-home:$HOME ubuntu:22.04 bash

2. Inside each container, create user, install lttng-tools, and start shell as $USER:

containerN$ groupdel "$GROUP" &>/dev/null || true
containerN$ groupadd -og "$GROUP_ID" "$GROUP" 
containerN$ useradd -M -u "$USER_ID" -g "$GROUP_ID" -d "/home/$USER" -s /bin/bash "$USER" 
containerN$ usermod -aG sudo "$USER" 
containerN$ apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y lttng-tools
containerN$ su - "$USER" 

Then, I can start a non-root session daemon in container 1 and trace just fine:

container1$ lttng list  # No session daemon, as expected
Error: No session daemon is available
container1$ lttng-sessiond --daemonize
container1$ pgrep lttng-sessiond  # As expected, we have a session daemon
459
container1$ cat $HOME/.lttng/lttng-sessiond.pid  # Matches above output
459
container1$ ps -o comm= $(cat $HOME/.lttng/lttng-sessiond.pid)  # That PID indeed corresponds to the session daemon
lttng-sessiond
container1$ lttng list  # Now this works, as expected
Currently no available recording session
container1$  # Trace, ..., works fine
container1$ babeltrace /home/$USER/lttng-traces/trace-container1  # Shows trace data
...

However, in container 2, LTTng thinks that there is already a session daemon, but tracing doesn't collect any data:

container2$ pgrep lttng-sessiond  # No output, since there is no session daemon, as expected
container2$ lttng-sessiond --daemonize  # Doesn't work, which is unexpected, since there isn't supposed to be a session daemon!
Error: A session daemon is already running.
container2$ cat $HOME/.lttng/lttng-sessiond.pid  # Matches output from container 1, since they have the same $HOME directory
459
container1$ ps -o comm= 459  # ...but that PID doesn't correspond to any process in this container! (no output)
container2$ lttng list  # Works, which is unexpected, since there isn't supposed to be a session daemon!
Currently no available recording session
container2$  # Trace, ..., works fine
container2$ babeltrace /home/$USER/lttng-traces/trace-container2  # Error, babeltrace says it's not a trace, which is unexpected
[error] Cannot open any trace for reading.  
[error] opening trace "/home/$USER/lttng-traces/trace-container2" for reading.
[error] none of the specified trace paths could be opened.
container2$ ls /home/$USER/lttng-traces/trace-container2  # The trace directory has a ust/ subdirectory, but it's empty
ust
container2$ export LTTNG_HOME=~/some-other-dir  # Setting/using a different LTTNG_HOME fixes everything

As mentioned in the last line above, using a different LTTNG_HOME allows container2 to successfully start its own non-root session daemon and collect trace data. However, the issue is that this fails silently: LTTng doesn't detect this weird configuration/state and doesn't give an error.

I can detect this myself by checking if the PID in the $HOME/.lttng/lttng-sessiond.pid file is a 'lttng-sessiond' process:

[ $(ps -o comm= $(cat $HOME/.lttng/lttng-sessiond.pid)) = 'lttng-sessiond' ]

If not, then that means that the session daemon isn't really "available," and I can report an error in my own tool and tell my user to set LTTNG_HOME to something other than $HOME and try again. I understand that this is a weird corner case, but it would be good if LTTng itself could handle this check, if possible.

Note: in my case, the $HOME directory inside the containers is not the actual host user's $HOME directory. However, if it were, maybe this could be reproducible with the host + 1 Docker container?