In your test procedure, a couple of operations are not detailed. How do you perform the preparation steps ?

"We have 4 CPUs. We isolate them with cpusets
We keep the system on CPU 0.
We start a sessiond daemon on CPU 2
We isolate a shell session on CPU 1 and then run the following commands as root :"

Also: what is your computer configuration ? How many processors, and how many thread per processor (hyperthreading ?) Have you made sure hyperthreading is disabled in your bios ?

I ran your test-case without any preparation steps (no CPU pinning performed), and there seems to be no event lost, either with or without --read-timer:

lttng-sessiond (as root)

# with read-timer:

lttng create lttngrt
lttng enable-channel u -u --num-subbuf 32 --subbuf-size 1048576 --read-timer 200
lttng enable-event -c u -u "npt:loop","npt:start","npt:stop" 
lttng start

as root:

./npt --eval-cpu-speed -l10000000
# CPU affinity set on CPU 1
# Application priority set to 99
# CPU frequency (evaluation): 2893.41 MHz
10000000 cycles done over 10000000.
Cycles duration:
        min:            1.093518 us
        max:            35.203394 us
        mean:           1.901982 us
        sum:            19019823.487977 us
        variance:       0.800043 us
        std dev:        0.894451 us
--------------------------
duration (us)   nb. cycles
--------------------------
1               7332645
2               2150347
3               400108
4               22521
5               47592
6               508
7               346
8               208
9               1064
10              2522
11              15075
12              23375
13              3240
14              246
15              73
16              42
17              29
18              24
19              17
20              4
21              6
22              2
23              4
32              1
35              1
--------------------------
Overruns (1000000+): 0

lttng destroy

babeltrace /home/compudj/lttng-traces/lttngrt-20130410-205542 > /dev/null
(no event discarded)
babeltrace /home/compudj/lttng-traces/lttngrt-20130410-205542 |wc -l
10000007

# without read-timer:

lttng create lttngrt
lttng enable-channel u -u --num-subbuf 32 --subbuf-size 1048576
lttng enable-event -c u -u "npt:loop","npt:start","npt:stop" 
lttng start

as root:

./npt --eval-cpu-speed -l10000000
# CPU affinity set on CPU 1
# Application priority set to 99
# CPU frequency (evaluation): 2893.41 MHz

10000000 cycles done over 10000000.
Cycles duration:
        min:            1.051353 us
        max:            354.635387 us
        mean:           3.081866 us
        sum:            30818657.399807 us
        variance:       0.408903 us
        std dev:        0.639455 us
--------------------------
duration (us)   nb. cycles
--------------------------
1               749444
2               973
3               9040419
4               158954
5               25511
6               2639
7               3573
8               8107
9               6541
10              1116
11              957
12              888
13              356
14              190
15              146
16              89
17              40
18              21
19              12
20              2
21              3
22              1
23              3
24              4
25              5
26              2
27              2
29              1
354             1
--------------------------
Overruns (1000000+): 0

lttng destroy

babeltrace /home/compudj/lttng-traces/lttngrt-20130410-210519 > /dev/null
(no event discarded)
babeltrace /home/compudj/lttng-traces/lttngrt-20130410-210519 |wc -l
10000007

Since the bug report, I worked with Yannick on the bug and discovered new elements that could help target the problem.
I'll try here to answer your questions and give the new elements.

OS and system¶

xaf@station11-64:~$ uname -a
Linux station11-64 3.2.0-4-rt-amd64 #1 SMP PREEMPT RT Debian 3.2.32-1 x86_64 GNU/Linux
xaf@station11-64:~$ cat /proc/cpuinfo | grep "^model name" 
model name    : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
model name    : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
model name    : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz
model name    : Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz

Hyperthreading off (working on real time...)

npt¶

Forgot to mention, the version used for these tests is the v0.02 (tag). The new version I'm working on is not currently mature enough and needs some configure options to work as the v0.02 works (it shouldn't have been pushed online yet... my bad). Sorry for that.

CPU shielding¶

# disable NMI watchdog
/sbin/sysctl -q -e -w "kernel.nmi_watchdog=0" 
# Restrict stopmachine to cpu 0
/sbin/sysctl -q -e -w "kernel.stopmachine_cpumask=0x1" 
# Restrict pagedrain to cpu 0
/sbin/sysctl -q -e -w "vm.pagedrain_cpumask=0x1" 
# Disable scheduler RT throttling
/sbin/sysctl -q -e -w "kernel.sched_rt_runtime_us=-1" 
# Disable softlockup detection.
/sbin/sysctl -q -e -w "kernel.softlockup_thresh=-1" 

# Set default IRQ affinity to cpu 0
echo 1 > /proc/irq/default_smp_affinity
# Set affinity for all active IRQs
for i in /proc/irq/[0-9]*; do
    echo 1 > $i/smp_affinity 2>/dev/null
done

# Mount cgroups
mkdir /cpusets
mount -t cgroup -ocpuset cpuset /cpusets

# Disable systemwide load balancing
cd /cpusets
echo 0 > cpuset.sched_load_balance

# Create system cpuset
mkdir system
echo 0 > system/cpuset.cpus
echo 0 > system/cpuset.mems
echo 0 > system/cpuset.sched_load_balance
echo 1 > system/cpuset.cpu_exclusive

# Move all tasks
for t in `cat tasks`; do
    echo $t > system/tasks
done

# Create NPT cpuset
mkdir npt
echo 1 > npt/cpuset.cpus
echo 0 > npt/cpuset.mems
echo 0 > npt/cpuset.sched_load_balance
echo 1 > npt/cpuset.cpu_exclusive

# Create LTTng cpuset
mkdir lttng
echo 2 > lttng/cpuset.cpus
echo 0 > lttng/cpuset.mems
echo 0 > lttng/cpuset.sched_load_balance
echo 1 > lttng/cpuset.cpu_exclusive

UST and kernel trace¶

We discovered that we can have drops of events in UST traces but also in kernel traces, depending on the configuration of the system. In all the cases the drops of events appeared, the kernel trace was also activated. I forgot to mention that in my bug report as I didn't see that it could be linked. By doing more tests and desactivating it, we found that it was linked.
In all cases, the test procedure was the following (in root/with sudo):

lttng create lttngrt
lttng enable-channel k -k --num-subbuf 32 --subbuf-size 1048576
lttng enable-event -c k -k -a
lttng enable-channel u -u --num-subbuf 32 --subbuf-size 1048576
lttng enable-event -c u -u "npt:loop","npt:start","npt:stop" 
./npt --eval-cpu-speed -l10000000
lttng destroy

This test procedure should be run in a shell put in the npt cpuset:

echo $$ > /cpuset/npt/tasks

echo $$ > /cpuset/lttng/tasks

Here are the information about the different tests made and the results obtained.

Without CPU shielding¶

LTTng v2.1.0¶

No drops of events were encountered with this configuration.
Another test we made was not using the cli() and sti() and we encountered event drops in the UST trace (not kernel) in this case. The following tests will only use npt with active cli() and sti() calls to analyze the regression of LTTng.

LTTng-tools (bd2d5dd) & UST (34a91bd)¶

Here, the number of dropped events is always 8602021 for UST. No event is dropped for the kernel.

With CPU shielding¶

LTTng v2.1.0¶

No drops of events were encountered with this configuration.

LTTng-tools (bd2d5dd) & UST (34a91bd)¶

1. When we added a new CPU to the LTTng cpuset (echo 2-3 > lttng/cpuset.cpus), but there where drops of kernel events.
2. When we put a very big subbuffer size for the kernel channel (4M instead of 1M here). In this case, the kernel was just never restarting working on the buffers after it used them a first time as it has as much space as needed for the whole kernel trace of the npt run. No drops of kernel events either in this case.

The idea that came from that is that the "consumerd"s of lttng were interfering. When the kernel consumerd needs to run, the UST consumerd has to wait, and thus if the kernel consumerd is not able to follow the rythm, he is just keeping all the CPU time. This situation seems weird to me as NPT is not generating any kernel event, and UST should not generate such events either.

Number of events expected for npt UST tracepoints¶

• 1 tracepoint npt:start
• 5 tracepoints npt:loop to prepare the npt work (before calculating loops time)
• 10000000 tracepoints npt:loop (-l option)
• 1 tracepoint npt:stop

With d1a996f5524540ebc8caa7c3c1bd3049018d212c (ust) and dec56f6cc894de41b312354d360b6a4c09fc199d (tools), and without the read-timer option, the comportement of LTTng seems ok.
But still have many drops with the read-timer option.