Project

General

Profile

Actions
Copy link

Bug #659

closed

Kernel crash when killing sessiond

Added by Christian Babeux over 11 years ago. Updated about 11 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
11/13/2013
Due date:
% Done:

0%

Estimated time:

Description

I was trying out the new live streaming feature and stumbled upon a kernel crash when issuing CTRL-C to the sessiond.

Here the kernel crash log:

Nov 13 15:08:11 Tau kernel: ring buffer relay-metadata: 2 records written, 0 records overrun
Nov 13 15:08:11 Tau kernel: [96B blob data]
Nov 13 15:08:11 Tau kernel: ring buffer: relay-metadata, cpu -1: 8192 bytes committed
Nov 13 15:08:12 Tau sudo[19434]: pam_unix(sudo:session): session closed for user root
Nov 13 15:08:12 Tau kernel: BUG: unable to handle kernel paging request at ffffffffa0988b38
Nov 13 15:08:12 Tau kernel: IP: [<ffffffffa08ebc10>] lttng_stream_ring_buffer_ioctl+0xd0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel: PGD 180f067 PUD 1813063 PMD 22ac2c067 PTE 0
Nov 13 15:08:12 Tau kernel: Oops: 0000 [#1] PREEMPT SMP 
Nov 13 15:08:12 Tau kernel: Modules linked in: lttng_tracer(O) lttng_statedump(O) lttng_ftrace(O) lttng_kprobes(O) lttng_lib_ring_buffer(O) lttng_kr
Nov 13 15:08:12 Tau kernel:  xts lrw gf128mul ablk_helper cryptd ehci_pci ext4 crc16 jbd2 mbcache ehci_hcd usbcore usb_common sd_mod ahci libahci libata scsi_m
Nov 13 15:08:12 Tau kernel: CPU 3 
Nov 13 15:08:12 Tau kernel: Pid: 19684, comm: lttng-consumerd Tainted: G           O 3.8.6-1-ARCH #1 LENOVO 0831CTO/0831CTO
Nov 13 15:08:12 Tau kernel: RIP: 0010:[<ffffffffa08ebc10>]  [<ffffffffa08ebc10>] lttng_stream_ring_buffer_ioctl+0xd0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel: RSP: 0018:ffff880217077ea8  EFLAGS: 00010286
Nov 13 15:08:12 Tau kernel: RAX: ffffffffa0988ac0 RBX: ffff8801e202ba00 RCX: 00007ff5ead3ab58
Nov 13 15:08:12 Tau kernel: RDX: ffff880217077ea8 RSI: ffffe8ffffc00cb0 RDI: ffff8801f5f96c68
Nov 13 15:08:12 Tau kernel: RBP: ffff880217077ec0 R08: ffffe8ffffc00cb0 R09: ffff8801e202ba00
Nov 13 15:08:12 Tau kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff5ead3ab58
Nov 13 15:08:12 Tau kernel: R13: 00007ff5ead3ab58 R14: 00007ff5ead3ab58 R15: 0000000000000001
Nov 13 15:08:12 Tau kernel: FS:  00007ff5ead3b700(0000) GS:ffff88023bd80000(0000) knlGS:0000000000000000
Nov 13 15:08:12 Tau kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 13 15:08:12 Tau kernel: CR2: ffffffffa0988b38 CR3: 00000001db7a8000 CR4: 00000000000007e0
Nov 13 15:08:12 Tau kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 13 15:08:12 Tau kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 13 15:08:12 Tau kernel: Process lttng-consumerd (pid: 19684, threadinfo ffff880217076000, task ffff8801d0bfdc20)
Nov 13 15:08:12 Tau kernel: Stack:
Nov 13 15:08:12 Tau kernel:  ffff8801e92a8010 ffff8801e202ba00 ffff880232f746d8 ffff880217077f30
Nov 13 15:08:12 Tau kernel:  ffffffff81199775 0000000000000000 0000000000000000 ffff8801e92a8010
Nov 13 15:08:12 Tau kernel:  0000000000000008 0000000000000001 ffff8802237ca238 ffff8801e202ba00
Nov 13 15:08:12 Tau kernel: Call Trace:
Nov 13 15:08:12 Tau kernel:  [<ffffffff81199775>] do_vfs_ioctl+0x2e5/0x4d0
Nov 13 15:08:12 Tau kernel:  [<ffffffff811999e1>] sys_ioctl+0x81/0xa0
Nov 13 15:08:12 Tau kernel:  [<ffffffff814c735d>] system_call_fastpath+0x1a/0x1f
Nov 13 15:08:12 Tau kernel: Code: 00 85 c0 79 c6 66 0f 1f 44 00 00 48 c7 c0 da ff ff ff eb c5 0f 1f 80 00 00 00 00 48 8b 42 48 48 85 c0 74 e7 48 8d 55 e8 4c 89
Nov 13 15:08:12 Tau kernel: RIP  [<ffffffffa08ebc10>] lttng_stream_ring_buffer_ioctl+0xd0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel:  RSP <ffff880217077ea8>
Nov 13 15:08:12 Tau kernel: CR2: ffffffffa0988b38
Nov 13 15:08:12 Tau kernel: ---[ end trace 7ce7dcdb7dc4e6f9 ]---
Nov 13 15:08:12 Tau kernel: BUG: unable to handle kernel paging request at ffffffffa0988b68
Nov 13 15:08:12 Tau kernel: IP: [<ffffffffa08ebbe0>] lttng_stream_ring_buffer_ioctl+0xa0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel: PGD 180f067 PUD 1813063 PMD 22ac2c067 PTE 0
Nov 13 15:08:12 Tau kernel: Oops: 0000 [#2] PREEMPT SMP 
Nov 13 15:08:12 Tau kernel: Modules linked in: lttng_tracer(O) lttng_statedump(O) lttng_ftrace(O) lttng_kprobes(O) lttng_lib_ring_buffer(O) lttng_kr
Nov 13 15:08:12 Tau kernel:  xts lrw gf128mul ablk_helper cryptd ehci_pci ext4 crc16 jbd2 mbcache ehci_hcd usbcore usb_common sd_mod ahci libahci libata scsi_m
Nov 13 15:08:12 Tau kernel: CPU 0 
Nov 13 15:08:12 Tau kernel: Pid: 19686, comm: lttng-consumerd Tainted: G      D    O 3.8.6-1-ARCH #1 LENOVO 0831CTO/0831CTO
Nov 13 15:08:12 Tau kernel: RIP: 0010:[<ffffffffa08ebbe0>]  [<ffffffffa08ebbe0>] lttng_stream_ring_buffer_ioctl+0xa0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel: RSP: 0018:ffff8801a74ebea8  EFLAGS: 00010286
Nov 13 15:08:12 Tau kernel: RAX: ffffffffa0988ac0 RBX: ffff88021ce37800 RCX: 00007ff5e9d38af8
Nov 13 15:08:12 Tau kernel: RDX: ffff8801a74ebea8 RSI: ffffe8ffffd80cb0 RDI: ffff8801f5f96c68
Nov 13 15:08:12 Tau kernel: RBP: ffff8801a74ebec0 R08: ffffe8ffffd80cb0 R09: ffff88021ce37800
Nov 13 15:08:12 Tau kernel: R10: 0000000000000008 R11: 0000000000000246 R12: 00007ff5e9d38af8
Nov 13 15:08:12 Tau kernel: R13: 00007ff5e9d38af8 R14: 00007ff5e9d38af8 R15: 0000000000000001
Nov 13 15:08:12 Tau kernel: FS:  00007ff5e9d39700(0000) GS:ffff88023bc00000(0000) knlGS:0000000000000000
Nov 13 15:08:12 Tau kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Nov 13 15:08:12 Tau kernel: CR2: ffffffffa0988b68 CR3: 00000001db7a8000 CR4: 00000000000007f0
Nov 13 15:08:12 Tau kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 13 15:08:12 Tau kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Nov 13 15:08:12 Tau kernel: Process lttng-consumerd (pid: 19686, threadinfo ffff8801a74ea000, task ffff88022ff72180)
Nov 13 15:08:12 Tau kernel: Stack:
Nov 13 15:08:12 Tau kernel:  00007ff5d8003b70 ffff88021ce37800 ffff880232f746d8 ffff8801a74ebf30
Nov 13 15:08:12 Tau kernel:  ffffffff81199775 0000380000000000 0000000000000000 0000000000000001
Nov 13 15:08:12 Tau kernel:  000000000000002e 000000000001fffe 0000000000000001 ffff88021ce37800
Nov 13 15:08:12 Tau kernel: Call Trace:
Nov 13 15:08:12 Tau kernel:  [<ffffffff81199775>] do_vfs_ioctl+0x2e5/0x4d0
Nov 13 15:08:12 Tau kernel:  [<ffffffff811999e1>] sys_ioctl+0x81/0xa0
Nov 13 15:08:12 Tau kernel:  [<ffffffff814c735d>] system_call_fastpath+0x1a/0x1f
Nov 13 15:08:12 Tau kernel: Code: 89 e1 e8 14 92 98 e0 48 98 48 83 c4 08 5b 41 5c 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 42 48 48 85 c0 74 17 48 8d 55 e8 4c 89
Nov 13 15:08:12 Tau kernel: RIP  [<ffffffffa08ebbe0>] lttng_stream_ring_buffer_ioctl+0xa0/0x1a0 [lttng_tracer]
Nov 13 15:08:12 Tau kernel:  RSP <ffff8801a74ebea8>
Nov 13 15:08:12 Tau kernel: CR2: ffffffffa0988b68
Nov 13 15:08:12 Tau kernel: ---[ end trace 7ce7dcdb7dc4e6fa ]---

Actions
Copy link
 #1

Updated by Mathieu Desnoyers over 11 years ago

Similar issue, with 2.4-rc, with some user-level changes to tools:

[  106.743943] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[  106.744071] IP: [<0000000000000090>] 0x8f
[  106.744143] PGD 20a458067 PUD 210417067 PMD 0 
[  106.744212] Oops: 0010 [#1] SMP 
[  106.744265] CPU 1 
[  106.744291] Modules linked in: lttng_probe_workqueue(O) lttng_probe_vmscan(O) lttng_probe_udp(O) lttng_probe_timer(O) lttng_probe_sunrpc(O) lttng_probe_statedump(O) lttng_probe_sock(O) lttng_probe_skb(O) lttng_probe_signal(O) lttng_probe_scsi(O) lttng_probe_sched(O) lttng_probe_rpm(O) lttng_probe_regulator(O) lttng_probe_regmap(O) lttng_probe_rcu(O) lttng_probe_random(O) lttng_probe_printk(O) lttng_probe_power(O) lttng_probe_net(O) lttng_probe_napi(O) lttng_probe_module(O) lttng_probe_kvm(O) lttng_probe_kmem(O) lttng_probe_jbd2(O) lttng_probe_jbd(O) lttng_probe_irq(O) lttng_probe_gpio(O) lttng_probe_compaction(O) lttng_probe_block(O) lttng_types(O) lttng_ring_buffer_metadata_mmap_client(O) lttng_ring_buffer_client_mmap_overwrite(O) lttng_ring_buffer_client_mmap_discard(O) lttng_ring_buffer_metadata_client(O) lttng_ring_buffer_client_overwrite(O) lttng_ring_buffer_client_discard(O) lttng_tracer(O) lttng_statedump(O) lttng_kprobes(O) lttng_lib_ring_buffer(O) lttng_kretprobes(O) ip6table_filter ip6_tables ebtable_nat ebtables ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc parport_pc ppdev lp parport cpufreq_conservative cpufreq_stats cpufreq_powersave cpufreq_userspace rfcomm bnep binfmt_misc uinput fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop snd_hda_codec_hdmi snd_hda_codec_realtek usb_storage uas hid_generic usbhid hid arc4 coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec iwlwifi acpi_cpufreq snd_hwdep i915 snd_pcm snd_page_alloc btusb thinkpad_acpi snd_seq bluetooth snd_seq_device aesni_intel ac aes_x86_64 mperf nvram evdev snd_timer i2c_i801 psmouse tpm_tis battery snd tpm tpm_bios aes_generic cryptd mac80211 wmi serio_raw video drm_kms_helper drm pcspkr i2c_algo_bit i2c_core cfg80211 soundcore rfkill button lpc_ich mfd_core processor ext4 crc16 jbd2 mbcache dm_mod sg sd_mod crc_t10dif ahci libahci libata scsi_mod ehci_hcd xhci_hcd microcode thermal thermal_sys sdhci_pci sdhci mmc_core e1000e usbcore usb_common [last unloaded: scsi_wait_scan]
[  106.747356] 
[  106.747366] Pid: 7357, comm: lttng-consumerd Tainted: G           O 3.5-trunk-amd64 #1 LENOVO 2306CTO/2306CTO
[  106.747524] RIP: 0010:[<0000000000000090>]  [<0000000000000090>] 0x8f
[  106.747631] RSP: 0018:ffff8802138bfe80  EFLAGS: 00010286
[  106.747733] RAX: ffff880212f32488 RBX: ffffffffffffffda RCX: 000000008008f620
[  106.747868] RDX: ffff8802138bfe90 RSI: ffff88021e25a490 RDI: ffff880213eed868
[  106.748003] RBP: 00007f170bd14a58 R08: ffff880212f32440 R09: 0000000000001cbd
[  106.748138] R10: 0000000000000004 R11: 0000000000000202 R12: 00007f170bd14a58
[  106.748273] R13: ffff88021d539910 R14: 00007f170ef45040 R15: 0000000000000003
[  106.748409] FS:  00007f170bd15700(0000) GS:ffff88021e240000(0000) knlGS:0000000000000000
[  106.748532] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  106.748610] CR2: 0000000000000090 CR3: 00000002142d9000 CR4: 00000000001407e0
[  106.748707] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  106.748802] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  106.748918] Process lttng-consumerd (pid: 7357, threadinfo ffff8802138be000, task ffff8802159d81c0)
[  106.749029] Stack:
[  106.749059]  ffffffffa06fbf0f ffffea0006be0570 0000000000000000 ffff880209ed7ec0
[  106.749175]  00000000ffffffe7 00007f170bd14a58 ffffffff8110d7a1 ffff8802094756c0
[  106.749288]  0000000000000000 0000000000000000 0000000000000000 ffff88020a4fd590
[  106.749392] Call Trace:
[  106.749455]  [<ffffffffa06fbf0f>] ? lttng_stream_ring_buffer_ioctl+0x66/0xff [lttng_tracer]
[  106.749574]  [<ffffffff8110d7a1>] ? do_vfs_ioctl+0x453/0x494
[  106.749664]  [<ffffffff8110d82d>] ? sys_ioctl+0x4b/0x72
[  106.749759]  [<ffffffff813657f9>] ? system_call_fastpath+0x16/0x1b
[  106.749851] Code:  Bad RIP value.
[  106.749901] RIP  [<0000000000000090>] 0x8f
[  106.749971]  RSP <ffff8802138bfe80>
[  106.750020] CR2: 0000000000000090
[  107.273515] ---[ end trace ec90b5340ec0e51e ]---
Actions
Copy link
 #2

Updated by Mathieu Desnoyers over 11 years ago

My previous commit is unrelated to the filed issue, as the cause of my oops has been found and fixed (Fix: tracepoint event name mapping at unregister).

Actions
Copy link
 #3

Updated by Mathieu Desnoyers about 11 years ago

  • Status changed from New to Resolved

Fixed by commit:

commit dd5a0db3ea07c46bee3c1814ef7326736f38a06e
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Feb 11 18:18:51 2014 -0500

    Fix: use after free in ring buffer clients

    Don't use ring buffer client's struct lttng_channel from ioctl which
    applies to ring buffer streams, because lttng_channel is freed while lib
    ring buffer stream and channel are still in use. Their lifetime persists
    until the consumer daemon releases its handles on the related stream
    file descriptors.

    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Actions
Copy link

Also available in: Atom PDF