LTTng » LTTng-modules

irc discussion about this:

11:10 < Compudj> a kmalloc with GFP_ATOMIC | GFP_NOWAIT flags calling the page allocator on a 4.1.21-rt13 kernel: how can this happen ?
11:11 < Compudj> I wonder if there is a flag I should add in my code, or blame it on that specific kernel
11:11 < Compudj> it's called from preempt off context, so complaining about scheduling in atomic ctx
11:12 < Compudj> ref. https://bugs.lttng.org/issues/1155
11:14 < bigeasy> Compudj: where is this tree from?
11:14 < Compudj> bigeasy: I cannot find it anywhere :(
11:14 < bigeasy> Compudj: but to keep it brief, you can't kmalloc in preempt disable region
11:14 < Compudj> bigeasy: the only leads I get from a gurgle search is windriver
11:15 < Compudj> bigeasy: even with GFP_ATOMIC flag ?
11:15 < bigeasy> I have v4.1.12-rt13 
11:15 < bigeasy> yup, never
11:16 < bigeasy> we always try to breakout somehow from preempt disable section and local_irq_disable() sections to do kmalloc()
11:16 < Compudj> not really possible within a tracepoint :-(
11:17 < Compudj> because it's using preempt off to ensure consistency of probe callbacks
11:17 < Compudj> I just need to try to allocate memory for a temporary buffer when tracing a system call
11:17 < bigeasy> I think the i915 did this and I just disabled that tracepoint
11:17 < Compudj> and it's OK if it fails due to lack of memory
11:17 < Compudj> (rather than invoke page allocator)
11:19 < Compudj> bigeasy: GFP_ATOMIC is documented as "the caller cannot reclaim or sleep and is high priority. Users are typically interrupt 
                 handlers." 
11:19 < Compudj> so how comes it cannot be called from preempt-off ?
11:19 < Compudj> is there a documentation source describing this ?
11:22 < bigeasy> not sure. probably not. On RT the interrupt handler runs in threaded context so you can allocate memory even without GFP_ATOMIC 
                 :)
11:22 < Compudj> mergh...
11:22 < bigeasy> but the thing is, in page allocater you go after sleeping locks so you must be preemptible
11:22 < Compudj> ok, so it's preempt-rt specific
11:22 < bigeasy> yes
11:23 < Compudj> is there a GFP_MAGICWAND in preempt-rt that allows allocating memory with preempt disabled from an emergency pool ?
11:23 < bigeasy> we don't have this
11:23 < bigeasy> we have alsmost everything in preemptible context
11:23 < Compudj> except tracepoint handlers
11:24 < bigeasy> and keep the preempt disabled section small as possible
11:24 < Compudj> we should really introduce preemptible tracepoint handlers
11:24 < Compudj> there is srcu for this kind of stuff
11:24 < wildea01> bigeasy: can you use a kmem_cache?
11:24 < bigeasy> this reminds me of the amd-iommu patches I have somewhere solving a similar problem there
11:24 < rostedt> Compudj: what we've done in the histogram code is to have an allocated pool
11:24 < bigeasy> yes but in preempt-disable sectiions
11:25 < bigeasy> wildea01: ^
11:25 < rostedt> when the pool gets low, we kick off a irq work to do more allocation
11:25 < Compudj> rostedt: right, that would work
11:25 < rostedt> and in -rt that irq work runs in thread context
11:25 < bigeasy> wildea01: again. yes but _not_ in preempt-disabled sections
11:26 < wildea01> bigeasy: ok, thanks
11:26 < Compudj> rostedt: ok I'll try something along the same lines, thanks
11:26 < rostedt> np

We just merged two patches in the master branch that will help:
commit f771eda68a28a432a87a7a71b23aab9692828700
Create a memory pool for temporary tracepoint probes storage

commit ec85ce1db0ca20e17fedbc85dde39613d830cbab
Use the memory pool instead of kmalloc

Can you try with those and let us know if it fixes the problem ?

Thanks,

Julien

Fredrik Gustavsson wrote:

Julien Desfossez wrote:

We just merged two patches in the master branch that will help:
commit f771eda68a28a432a87a7a71b23aab9692828700
Create a memory pool for temporary tracepoint probes storage

commit ec85ce1db0ca20e17fedbc85dde39613d830cbab
Use the memory pool instead of kmalloc

Can you try with those and let us know if it fixes the problem ?

Thanks,

Julien

Hi,

Thanks for the quick support. Testing it out at the moment.

The arises quickly, within minutes (even seconds sometimes) when provoking the system.

The lttng_kernel_load will show you the lttng config.

I have try to reproduce it during the weekend but I have not been able to. So it seems like a these two commits did the trick.