Kernel Module Debugging

Prerequisites

• 2 VMs - one to run the kernel and one to debug
  • In the following I will call them DEBUG_VM and RUN_VM
  • I used 2 arch-based VMs
• GDB installed in DEBUG_VM
  • Optional with GEF extension for easier use
• Needed for automation scripts later

  • Running ssh server on RUN_VM

    • Generate ssh keys without passphrase to connect from DEBUG_VM to RUN_VM without passwords

  • Passwordless sudo on RUN_VM

  • Create a target RUN_VM in your ~/.ssh/config

      Host RUN_VM
      	HostName <RUN_VM_IP>
      	User <username>
      	IdentityFile ~/.ssh/<your_key>
    

Setup

• Install same Kernel on both VMs
  • Must be configured for debugging
  • see https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html
  • the command make scripts_gdb creates an invalid python script scripts/gdb/linux/constants.py
    • Open the script and remove all UL's
  • Compile the Kernel on DEBUG_VM and install the produced kernel on both VMs
    • I just copied (rsync) the whole kernel folder from DEBUG_VM to RUN_VM and compiled it again but it should be possible to only exec make modules_install again
• RUN_VM needs to expose the debugging interface to the host machine
  • Exit VMware
  • Open the VMware config file of RUN_VM
    • e.g. C:\User\<username>\Documents\Virtual Machines\RUN_VM\RUN_VM.vmx
  • add the following lines:
    • debugStub.listen.guest64 = "TRUE"
    • debugStub.listen.guest64.remote = "TRUE"
• Boot both VMs on the same Kernel
• Optional: Move your kernel module (including Makefile) into a subfolder of the kernel build director on DEBUG_VM
  • e.g.
    • linux-6.3.8/mymodule/
      • Makefile
      • network-listener-ipv6.c
  • With this GDB is able to automatically find the source code for your kernel module
  • There might be other ways to achieve this but I only used this one
  • I will refer to this folder as mymodule
• Now you can compile kernel modules on DEBUG_VM copy the .ko file to RUN_VM and load the module
  • Its a good idea to automate this step. I create 2 scripts for that in mymodule

    • reload.sh script on RUN_VM:

        #!/bin/sh
        make LLVM=1
        sudo rmmod network-listener-ipv6
        sudo insmod network-listener-ipv6.ko
      

    • reload_remote.sh script on DEBUG_VM:

        #!/bin/sh
        # compile modules
        make LLVM=1
        # Copy modules
        scp *.ko RUN_VM:~/kernel/linux-6.3.8/mymodule/
        # Reload modules
        ssh -t RUN_VM -- bash '~/kernel/linux-6.3.8/mymodule/reload.sh
      

    • You need to replace:

      • the module name
      • the server name / ip RUN_VM
      • the path to mymodule
      • the path to reload.sh

Debugging

• Find your hosts IP address
  • I will refer to it as HOST_IP
• On DEBUG_VM go to your kernel build folder
• run gdb vmlinux
• connect to RUN_VM:
  • When using GEF:
    • gef-remote <HOST_IP> 8864
  • Without:
    • target remote <HOST_IP>:8864
• Load symbols:
  • lx-symbols
  • A list of kernel modules should be printed
• Set a breakpoint to your function
  • e.g. b <mymodule-path>/network-listener-ipv6.c:<line>
• continue: c
• Now you can start normal debugging
• Do not forget to run detach before you quite GDB otherwise RUN_VM will be dead and you must boot it again