Ubuntu 16.04 Kernel Crash Dump

1. apt install linux-crashdump

2. /etc/default/kdump-tools by including the following line:

USE_KDUMP=1

3.  kdump-config show
DUMP_MODE:        kdump
USE_KDUMP:        1
KDUMP_SYSCTL:     kernel.panic_on_oops=1
KDUMP_COREDIR:    /var/crash
crashkernel addr: 0x2b000000
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.4.0-89-generic
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.4.0-89-generic
current state:    ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-4.4.0-89-generic root=UUID=505002b4-e453-4e7c-9eb5-a9af5ccf1391 ro irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

4. cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-89-generic root=UUID=505002b4-e453-4e7c-9eb5-a9af5ccf1391 ro crashkernel=384M-:128M

crashkernel=384M-2G:64M,2G-:128M

The above value means:

1).if the RAM is smaller than 384M, then don't reserve anything (this is the "rescue" case)

2).if the RAM size is between 386M and 2G (exclusive), then reserve 64M

3).if the RAM size is larger than 2G, then reserve 128M

5. cat /proc/sys/kernel/sysrq

If a value of 0 is returned the feature is disabled. Enable it with the following command :

sudo sysctl -w kernel.sysrq=1

6. echo c > /proc/sysrq-trigger
[   31.659002] SysRq : Trigger a crash
[   31.659749] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   31.662668] IP: [<ffffffff8139f166>] sysrq_handle_crash+0x16/0x20
[   31.662668] PGD 3bfb9067 PUD 368a7067 PMD 0
[   31.662668] Oops: 0002 [#1] SMP
[   31.662668] CPU 1
....

7.  ls /var/crash/
201708070427  kexec_cmd  linux-image-4.4.0-89-generic-201708070427.crash

8. apport-unpack /var/crash/linux-image-4.4.0-89-generic-201708070427.crash ~/tmp/

9. crash needs d gsym to work. Check whether debug directory exist in /usr/lib or not.If not, download it from http://ddebs.ubuntu.com/pool/main/l/linux/
The dbgsym should be the same with the kernel version.

10. crash /usr/lib/debug/boot/vmlinux-4.4.0-89-generic  /var/crash/201708070427/dump.201708070427