Project

General

Profile

Actions

Bug #1166

closed

lttng-crash bus error (core dumped)

Added by Florea Irinel almost 6 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
05/30/2018
Due date:
% Done:

100%

Estimated time:

Description

Hi

If lttng-sessiond terminates unexpectedly after truncating the metadata file from --shm-path path (when doing metadata regenerate), afterwards when trying to recover the trace buffers using lttng-crash <- it Bus error (core dumped). This translates to:

$ lttng create s1 --shm-path=/tmp/test-lttng
$ lttng enable-event -u -a -s s1
$ lttng start
$ <path>/lttng-tools/tests/utils/testapp/gen-ust-events/gen-ust-events
$ pkill -9 "lttng-" # 1
$ cp /dev/null /tmp/test/s1-*/ust/uid/1000/64-bit/metadata # 2 - 1 and 2 simulate sessiond crash immediately after ftruncate() during metadata regenerate
$ lttng-crash --viewer=babeltrace /tmp/test-lttng/

A possible fix is :

---
 src/bin/lttng-crash/lttng-crash.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/bin/lttng-crash/lttng-crash.c b/src/bin/lttng-crash/lttng-crash.c
index ba28af9..0fb93fd 100644
--- a/src/bin/lttng-crash/lttng-crash.c
+++ b/src/bin/lttng-crash/lttng-crash.c
@@ -490,7 +490,8 @@ int check_magic(const uint8_t *magic)
 }

 static
-int get_crash_layout(struct lttng_crash_layout *layout, int fd)
+int get_crash_layout(struct lttng_crash_layout *layout, int fd,
+    const char *input_path, const char *input_file)
 {
     char *map;
     int ret = 0, unmapret;
@@ -501,7 +502,13 @@ int get_crash_layout(struct lttng_crash_layout *layout, int fd)
     const struct crash_abi_unknown *abi;
     uint16_t endian;
     enum lttng_crash_type layout_type;
+    struct stat sb;

+    fstat(fd, &sb);
+    if (sb.st_size <  RB_CRASH_DUMP_ABI_LEN) {
+        ERR("File truncated: %s/%s", input_path, input_file);
+        return -1;
+    }
     map = mmap(NULL, RB_CRASH_DUMP_ABI_LEN, PROT_READ, MAP_PRIVATE,
         fd, 0);
     if (map == MAP_FAILED) {
@@ -830,7 +837,7 @@ int extract_file(int output_dir_fd, const char *output_file,
     }

     /* Query the crash ABI layout */
-    ret = get_crash_layout(&layout, fd_src);
+    ret = get_crash_layout(&layout, fd_src, input_path, input_file);
     if (ret) {
         goto close_src;
     }
-- 
1.9.1

Files

Actions #1

Updated by Jonathan Rajotte Julien almost 6 years ago

  • Description updated (diff)
Actions #2

Updated by Mathieu Desnoyers over 4 years ago

The attached patch addresses the lttng-crash Bus error.

There is however a larger question we may want to open here, which might end up being a separate "Wishlist" item: if at all possible, it would be nice if sessiond could replace the metadata file atomically on metadata regenerate, so there is always a valid version to read on crash.

Actions #3

Updated by Mathieu Desnoyers over 4 years ago

  • Status changed from Feedback to Resolved
  • % Done changed from 0 to 100
Actions #4

Updated by Filip Pudak about 3 years ago

Mathieu Desnoyers wrote in #note-3:

Applied in changeset lttng-tools|e6c6e235ab87233cf273f536188d842768304c55.

Hi Mathiu Desnoyers,

We ran into this one recently as well. I noticed that your patch did not contain the testcase that was mentioned above.
Is there any plan to add this testcase or some other way of verifying that this issue was fixed?

/Filip

Actions #5

Updated by Mathieu Desnoyers about 3 years ago

I think the test-case should be made simpler for integration within lttng-tools: rather than regenerate the crash files and manually truncate the metadata to simulate the effect of a kill signal delivery racing with a metadata regenerate, we could just add small crash trace files along with the test which exhibit the truncated metadata condition.

If someone can contribute such a reproducible test, we would gladly take a patch adding it to lttng-tools under tests/regression/tools/crash/.

Actions

Also available in: Atom PDF