Bug #1005
closedNew trace can't generate in persistent memory file system
0%
Description
Using the new feature Recording trace data on persistent memory file systems in LTTng 2.7.
After the device restart due to system crash, we can read the crash log in the specific shm-path directory. When we want to trace the new log, the following debug log reported and the new data can't be traced to the specific shm-path.
DEBUG3 - 20:07:54.353680 [286/291]: mkdir() recursive /shm/lttng/mysession-20160302-182255/ust/uid/0/32-bit with mode 504 for uid 0 and gid 0 (in run_as_mkdir_recursive() at runas.c:468)
DEBUG1 - 20:07:54.353726 [286/291]: Using run_as worker (in run_as() at runas.c:449)
DEBUG3 - 20:07:54.354141 [286/291]: open() /shm/lttng/mysession-20160302-182255/ust/uid/0/32-bit/metadata with flags C1 mode 384 for uid 0 and gid 0 (in run_as_open() at runas.c:498)
DEBUG1 - 20:07:54.354202 [286/291]: Using run_as worker (in run_as() at runas.c:449)
PERROR - 20:07:54.354724 [286/291]: Opening metadata file: File exists (in ust_registry_session_init() at ust-registry.c:606)
DEBUG3 - 20:07:54.354801 [286/291]: rmdir_recursive() /shm/lttng/mysession-20160302-182255 with for uid 0 and gid 0 (in run_as_rmdir_recursive() at runas.c:524)
DEBUG1 - 20:07:54.354847 [286/291]: Using run_as worker (in run_as() at runas.c:449)
DEBUG3 - 20:07:54.355554 [287/287]: Attempting rmdir /shm/lttng/mysession-20160302-182255 (in utils_recursive_rmdir() at utils.c:1247)
DEBUG3 - 20:07:54.356905 [286/291]: Buffer registry per UID destroy with id: 0, ABI: 32, uid: 0 (in buffer_reg_uid_destroy() at buffer-registry.c:641)
For detail info, please see my attachment.
Files
Updated by jia fang over 8 years ago
- Copied from Bug #1004: New trace can't generate in persistent memory file system added
Updated by Jonathan Rajotte Julien over 8 years ago
- Status changed from New to Feedback
- Assignee set to Jonathan Rajotte Julien
- Priority changed from High to Normal
Hi Jia,
Does the file exists as the error suggests ?
Do you use lttng load and a xml session descriptor?
The main reason to use shm is minimize data loss so not squashing existing files make sense since we do not know if the user have collected them. Depending on your use case there is multiples solutions.
Cheers
Updated by jia fang over 8 years ago
Hi,
yes, we use lttng load and xml session descriptor.
My use case is:
The system crashed and then it reboot and set up, after that, I run my application with the same shm-path.
I want to the new log of my application can be record into my shm-path file.
I try to open shm-path file with O_APPEND or O_TRUNC, it seems like that my application can overwrite the system crash log.
Could you please give some suggestion ?
Or I have to save and delete my crash log manually, and then run my application ?
Thanks
Updated by Jonathan Rajotte Julien over 8 years ago
jia fang wrote:
Hi,
yes, we use lttng load and xml session descriptor.
So technically you control were the shm path is located and the trace output and these locations are static.
Could you attach the actual xml files for the load command (make sure to replace any confidential informations by fake ones)?
A list of the used commands would be helpful to.
My use case is:
The system crashed and then it reboot and set up, after that, I run my application with the same shm-path.
I want to the new log of my application can be record into my shm-path file.
If you have a set-up phase I would recommend checking for files presence in the shm path and tar it before starting lttng-sessiond/using lttng. All this automatically. Make sure to also delete the files.
I try to open shm-path file with O_APPEND or O_TRUNC, it seems like that my application can overwrite the system crash log.
What is your procedure on a system crash ? Reboot and ignore the shm files or save the shm files for further extraction/analysis?
Could you please give some suggestion ?
Or I have to save and delete my crash log manually, and then run my application ?
I'm sure you could do this easily automatically on boot.
Thanks
Updated by jia fang over 8 years ago
Jonathan Rajotte Julien wrote:
jia fang wrote:
Hi,
yes, we use lttng load and xml session descriptor.
So technically you control were the shm path is located and the trace output and these locations are static.
Could you attach the actual xml files for the load command (make sure to replace any confidential informations by fake ones)?
A list of the used commands would be helpful to.
Attachment is my xml files
My use case is:
The system crashed and then it reboot and set up, after that, I run my application with the same shm-path.
I want to the new log of my application can be record into my shm-path file.If you have a set-up phase I would recommend checking for files presence in the shm path and tar it before starting lttng-sessiond/using lttng. All this automatically. Make sure to also delete the files.
So the easy way is to save and delete the files during the set-up phase before running application :)
I try to open shm-path file with O_APPEND or O_TRUNC, it seems like that my application can overwrite the system crash log.
What is your procedure on a system crash ? Reboot and ignore the shm files or save the shm files for further extraction/analysis?
My crash is "echo c > /proc/sysrq-trigger", and then my system will reboot automatically.
I want to reboot and "cp" my shm files but not "mv" (that is to say, I did not delete the shm files during set-up). After that, I run my application and then new log can be written into my shm files.
So I choose to use O_APPEND or O_TRUNC to open my shm files.
BTW, I found that even O_APPEND will not append new log to shm files. Both O_APPEND and O_TRUNC overwrite the shm files.
Could you please give some suggestion ?
Or I have to save and delete my crash log manually, and then run my application ?I'm sure you could do this easily automatically on boot.
Thanks
Updated by Jonathan Rajotte Julien over 8 years ago
jia fang wrote:
Jonathan Rajotte Julien wrote:
jia fang wrote:
Hi,
yes, we use lttng load and xml session descriptor.
So technically you control were the shm path is located and the trace output and these locations are static.
Could you attach the actual xml files for the load command (make sure to replace any confidential informations by fake ones)?
A list of the used commands would be helpful to.
Attachment is my xml files
Looks good to me.
My use case is:
The system crashed and then it reboot and set up, after that, I run my application with the same shm-path.
I want to the new log of my application can be record into my shm-path file.If you have a set-up phase I would recommend checking for files presence in the shm path and tar it before starting lttng-sessiond/using lttng. All this automatically. Make sure to also delete the files.
So the easy way is to save and delete the files during the set-up phase before running application :)
Yes since lttng will not squash existing shm files by design since it cannot know if the user have extracted/saved them.
I try to open shm-path file with O_APPEND or O_TRUNC, it seems like that my application can overwrite the system crash log.
What is your procedure on a system crash ? Reboot and ignore the shm files or save the shm files for further extraction/analysis?
My crash is "echo c > /proc/sysrq-trigger", and then my system will reboot automatically.
I want to reboot and "cp" my shm files but not "mv" (that is to say, I did not delete the shm files during set-up). After that, I run my application and then new log can be written into my shm files.
So I choose to use O_APPEND or O_TRUNC to open my shm files.
Why not 'mv'?
Where did you use O_APPEND/O_TRUNC ?
The principal scenario for using shm path would be:
1) The target crash
2) On set-up you have a script that look for orphan lttng files (in the shm path) and move them to a safe place for further analysis
3) Start lttng and tracing
Cheers
Updated by jia fang over 8 years ago
Jonathan Rajotte Julien wrote:
jia fang wrote:
Jonathan Rajotte Julien wrote:
jia fang wrote:
Hi,
yes, we use lttng load and xml session descriptor.
So technically you control were the shm path is located and the trace output and these locations are static.
Could you attach the actual xml files for the load command (make sure to replace any confidential informations by fake ones)?
A list of the used commands would be helpful to.
Attachment is my xml files
Looks good to me.
My use case is:
The system crashed and then it reboot and set up, after that, I run my application with the same shm-path.
I want to the new log of my application can be record into my shm-path file.If you have a set-up phase I would recommend checking for files presence in the shm path and tar it before starting lttng-sessiond/using lttng. All this automatically. Make sure to also delete the files.
So the easy way is to save and delete the files during the set-up phase before running application :)
Yes since lttng will not squash existing shm files by design since it cannot know if the user have extracted/saved them.
I try to open shm-path file with O_APPEND or O_TRUNC, it seems like that my application can overwrite the system crash log.
What is your procedure on a system crash ? Reboot and ignore the shm files or save the shm files for further extraction/analysis?
My crash is "echo c > /proc/sysrq-trigger", and then my system will reboot automatically.
I want to reboot and "cp" my shm files but not "mv" (that is to say, I did not delete the shm files during set-up). After that, I run my application and then new log can be written into my shm files.
So I choose to use O_APPEND or O_TRUNC to open my shm files.Why not 'mv'?
Because at that time I did not know I need to 'mv' :)
Where did you use O_APPEND/O_TRUNC ?
run_as_open(session->metadata_path...) in lttng-tools/src/bin/lttng-sessiond/ust-registry.c and run_as_open(session->metadata_path...) in lttng-tools/src/common/ust-consumer/ust-consumer.c
Do I misunderstand something ?
BRs
The principal scenario for using shm path would be:
1) The target crash
2) On set-up you have a script that look for orphan lttng files (in the shm path) and move them to a safe place for further analysis
3) Start lttng and tracingCheers
Updated by Jonathan Rajotte Julien over 8 years ago
Good.
Regarding the O_APPEND/O_TRUNC I would recommend against doing this change for the sake of crash data persistence.
With the change made to the set-up phase are you satisfied with the lttng shm feature and the lttng-crash utility ?
Cheers
Updated by jia fang over 8 years ago
Jonathan Rajotte Julien wrote:
Good.
Regarding the O_APPEND/O_TRUNC I would recommend against doing this change for the sake of crash data persistence.
OK, Thank you so much.
With the change made to the set-up phase are you satisfied with the lttng shm feature and the lttng-crash utility ?
We will change the set-up phase and other features are fine for us. If any problem, we will update.
Thanks again for quick reply :)
Cheers
Updated by Jonathan Rajotte Julien over 8 years ago
- Status changed from Feedback to Resolved