https://bugs.lttng.org/https://bugs.lttng.org/themes/lttng/favicon/a.ico?14249722912018-02-21T16:06:17ZLTTng bugs repositoryLTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=32982018-02-21T16:06:17ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>Can you provide your kernel .config ?</p>
<p>Also, I cannot find the source code for the 4.1.21-rt13 kernel anywhere (neither linux-rt stable git tree nor linux-rt patches). The only mentions of it based on a quick Google search seem to indicate that it is a Wind River kernel. You may want to ask them for help if it's the case.</p>
<p>Moreover, does the issue reproduce rather quickly after tracing is activated, or it takes a while ? How is your tracing configured ?</p>
<p>It's really unexpected that a kmalloc() with GFP_ATOMIC | GFP_NOWAIT end up calling the page allocator. It sounds like there may be something fishy in your kernel. Can you reproduce with a maintained, updated, version of the preempt-rt kernel ?</p> LTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=32992018-02-21T16:28:11ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<ul></ul><p>irc discussion about this:</p>
<pre>
11:10 < Compudj> a kmalloc with GFP_ATOMIC | GFP_NOWAIT flags calling the page allocator on a 4.1.21-rt13 kernel: how can this happen ?
11:11 < Compudj> I wonder if there is a flag I should add in my code, or blame it on that specific kernel
11:11 < Compudj> it's called from preempt off context, so complaining about scheduling in atomic ctx
11:12 < Compudj> ref. https://bugs.lttng.org/issues/1155
11:14 < bigeasy> Compudj: where is this tree from?
11:14 < Compudj> bigeasy: I cannot find it anywhere :(
11:14 < bigeasy> Compudj: but to keep it brief, you can't kmalloc in preempt disable region
11:14 < Compudj> bigeasy: the only leads I get from a gurgle search is windriver
11:15 < Compudj> bigeasy: even with GFP_ATOMIC flag ?
11:15 < bigeasy> I have v4.1.12-rt13
11:15 < bigeasy> yup, never
11:16 < bigeasy> we always try to breakout somehow from preempt disable section and local_irq_disable() sections to do kmalloc()
11:16 < Compudj> not really possible within a tracepoint :-(
11:17 < Compudj> because it's using preempt off to ensure consistency of probe callbacks
11:17 < Compudj> I just need to try to allocate memory for a temporary buffer when tracing a system call
11:17 < bigeasy> I think the i915 did this and I just disabled that tracepoint
11:17 < Compudj> and it's OK if it fails due to lack of memory
11:17 < Compudj> (rather than invoke page allocator)
11:19 < Compudj> bigeasy: GFP_ATOMIC is documented as "the caller cannot reclaim or sleep and is high priority. Users are typically interrupt
handlers."
11:19 < Compudj> so how comes it cannot be called from preempt-off ?
11:19 < Compudj> is there a documentation source describing this ?
11:22 < bigeasy> not sure. probably not. On RT the interrupt handler runs in threaded context so you can allocate memory even without GFP_ATOMIC
:)
11:22 < Compudj> mergh...
11:22 < bigeasy> but the thing is, in page allocater you go after sleeping locks so you must be preemptible
11:22 < Compudj> ok, so it's preempt-rt specific
11:22 < bigeasy> yes
11:23 < Compudj> is there a GFP_MAGICWAND in preempt-rt that allows allocating memory with preempt disabled from an emergency pool ?
11:23 < bigeasy> we don't have this
11:23 < bigeasy> we have alsmost everything in preemptible context
11:23 < Compudj> except tracepoint handlers
11:24 < bigeasy> and keep the preempt disabled section small as possible
11:24 < Compudj> we should really introduce preemptible tracepoint handlers
11:24 < Compudj> there is srcu for this kind of stuff
11:24 < wildea01> bigeasy: can you use a kmem_cache?
11:24 < bigeasy> this reminds me of the amd-iommu patches I have somewhere solving a similar problem there
11:24 < rostedt> Compudj: what we've done in the histogram code is to have an allocated pool
11:24 < bigeasy> yes but in preempt-disable sectiions
11:25 < bigeasy> wildea01: ^
11:25 < rostedt> when the pool gets low, we kick off a irq work to do more allocation
11:25 < Compudj> rostedt: right, that would work
11:25 < rostedt> and in -rt that irq work runs in thread context
11:25 < bigeasy> wildea01: again. yes but _not_ in preempt-disabled sections
11:26 < wildea01> bigeasy: ok, thanks
11:26 < Compudj> rostedt: ok I'll try something along the same lines, thanks
11:26 < rostedt> np
</pre> LTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=33002018-02-24T02:14:24ZJulien Desfossezjdesfossez@efficios.com
<ul></ul><p>We just merged two patches in the master branch that will help:<br />commit f771eda68a28a432a87a7a71b23aab9692828700<br />Create a memory pool for temporary tracepoint probes storage</p>
<p>commit ec85ce1db0ca20e17fedbc85dde39613d830cbab<br />Use the memory pool instead of kmalloc</p>
<p>Can you try with those and let us know if it fixes the problem ?</p>
<p>Thanks,</p>
<p>Julien</p> LTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=33012018-02-24T14:53:07ZFredrik Gustavsson
<ul><li><strong>File</strong> <a href="/attachments/410">lttng_kernel_load</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/410/lttng_kernel_load">lttng_kernel_load</a> added</li><li><strong>File</strong> <a href="/attachments/411">stresstest</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/411/stresstest">stresstest</a> added</li></ul><p>Julien Desfossez wrote:</p>
<blockquote>
<p>We just merged two patches in the master branch that will help:<br />commit f771eda68a28a432a87a7a71b23aab9692828700<br />Create a memory pool for temporary tracepoint probes storage</p>
<p>commit ec85ce1db0ca20e17fedbc85dde39613d830cbab<br />Use the memory pool instead of kmalloc</p>
<p>Can you try with those and let us know if it fixes the problem ?</p>
<p>Thanks,</p>
<p>Julien</p>
</blockquote>
<p>Hi,</p>
<p>Thanks for the quick support. Testing it out at the moment.</p>
<p>The arises quickly, within minutes (even seconds sometimes) when provoking the system.</p>
<p>The lttng_kernel_load will show you the lttng config.</p> LTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=33022018-02-26T08:15:05ZFredrik Gustavsson
<ul></ul><p>Fredrik Gustavsson wrote:</p>
<blockquote>
<p>Julien Desfossez wrote:</p>
<blockquote>
<p>We just merged two patches in the master branch that will help:<br />commit f771eda68a28a432a87a7a71b23aab9692828700<br />Create a memory pool for temporary tracepoint probes storage</p>
<p>commit ec85ce1db0ca20e17fedbc85dde39613d830cbab<br />Use the memory pool instead of kmalloc</p>
<p>Can you try with those and let us know if it fixes the problem ?</p>
<p>Thanks,</p>
<p>Julien</p>
</blockquote>
<p>Hi,</p>
<p>Thanks for the quick support. Testing it out at the moment.</p>
<p>The arises quickly, within minutes (even seconds sometimes) when provoking the system.</p>
<p>The lttng_kernel_load will show you the lttng config.</p>
</blockquote>
<p>I have try to reproduce it during the weekend but I have not been able to. So it seems like a these two commits did the trick.</p> LTTng-modules - Bug #1155: BUG: scheduling while atomic and BUG: sleeping function called from invalid contexthttps://bugs.lttng.org/issues/1155?journal_id=33032018-02-26T14:31:58ZMathieu Desnoyersmathieu.desnoyers@efficios.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Thanks for the feedback!</p>