Use crash to show the code line of a backtrace entry like "ocfs2_truncate_file+0x127/0x6c0"

摘要: kdump crash backtrace code line

###1. Problem
I am debugging a hung issue of ocfs2 (a cluster filesystem in linux kernel). This hung is

caused by a ocfs2 testcase named "mmap_truncate". Naturally, I show the status of this process

and cat the processes's stack as below:

# pgrep -a mmap_truncate
24250 mmap_truncate -c 20 -s 300 /mnt/ocfs2//mmaptruncate.txt

# cat /proc/24250/stack
[<ffffffff8131b373>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffffa0545717>] ocfs2_truncate_file+0x127/0x6c0 [ocfs2]
[<ffffffffa05481a8>] ocfs2_setattr+0x698/0xa90 [ocfs2]
[<ffffffffa05403f3>] ocfs2_inode_unlock+0x33/0x80 [ocfs2]
[<ffffffff8121574e>] notify_change+0x1ae/0x380
[<ffffffff811f8ade>] do_truncate+0x5e/0x90
[<ffffffff811f8e38>] do_sys_ftruncate.constprop.11+0x108/0x160
[<ffffffff815e126e>] entry_SYSCALL_64_fastpath+0x12/0x6d

What if I want to know the code line of a function where it hung up? For example:

ocfs2_truncate_file+0x127

###2. Howto

Install kernel debuginfo and debugsource packages,etc. I'm using SUSE linux:

zypper install kernel-default-debuginfo kernel-default-debugsource

Install ocfs2 kernel module debuginfo package if you're using SLES, while openSUSE is not needed to do so.

zypper in ocfs2-kmp-default-debuginfo

uncompress

vmlinux-${uname -r}.gz

img

#gunzip /boot/vmlinux-4.4.19-60-default.gz

invoke crash to analyze living system. You'd better get a overview of crash from

man crash

crash  /boot/vmlinux-4.4.19-60-default

show if ocfs2 module is loaded by

mod

, and

help mod

for more info

crash> mod
ffffffffa04d8440  ocfs2_stackglue         20480  (not loaded)  [CONFIG_KALLSYMS]
ffffffffa04ecf40  ocfs2_nodemanager      225280  (not loaded)  [CONFIG_KALLSYMS]
ffffffffa05e1c80  ocfs2                 1019904  (not loaded)  [CONFIG_KALLSYMS]

load ocfs2 module

crash> mod -s ocfs2
MODULE       NAME                     SIZE  OBJECT FILE
ffffffffa05e1c80  ocfs2                 1019904  /lib/modules/4.4.19-60-default/kernel/fs/ocfs2/ocfs2.ko

show the code line of ocfs2_truncate_file+0x127, and search the output for the offset near 0x127==295

crash> dis -l ocfs2_truncate_file
...
/usr/src/debug/kernel-default-4.4.19/linux-4.4/linux-obj/../fs/ocfs2/file.c: 483
0xffffffffa0545717 <ocfs2_truncate_file+295>:   mov    0x18(%rsp),%rax
0xffffffffa054571c <ocfs2_truncate_file+300>:   lea    0x278(%r12),%rsi
0xffffffffa0545724 <ocfs2_truncate_file+308>:   lea    0x230(%rax),%rdi
0xffffffffa054572b <ocfs2_truncate_file+315>:   callq  0xffffffffa0568480 <ocfs2_resv_discard>
...

Jump to the line of source code!

vi /usr/src/debug/kernel-default-4.4.19/linux-4.4/linux-obj/../fs/ocfs2/file.c +483

and we got it!

down_write(&OCFS2_I(inode)->ip_alloc_sem);

感谢来自亚马逊的于鹏给予的帮助！