Deprecated

The method described in this gist has been deprecated/superceeded by kiler129/early-vfio-pci-isolate tool.
It is more robust, configurable, and doesn't hack around scripts that are sometimes overwritten by system updates. The tool uses similar methods to the ones described below, but extends the capabilities by e.g. easy NVMe passthrough by S/N.

The description below has been preserved for historical context. At the time of writing, the tool above has been tested for ~6 months across multiple systems.

---

---

---

Scenario

You're running a KVM-based virtualization. You want to do PCI/PCIe passthrough of some device. You don't want it to attach to the host OS at all.

Your device looks like that:

00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

Problem

Usually the solutions are simple:

1. If you have only one device listing some module in Kernel modules (e.g. nvidiafb) you can add it to /etc/modprobe.d/some-file.conf as blacklist nvidiafb
2. If you have multiple and they're normal devices you just add options vfio-pci ids=8086:1c02 to some file in/etc/modprobe.d/ (make sure to use the id in [...] and not pci location 00:1f.2)

However, these will not work if your device is handled by something loaded very very VERY early... like a driver for your second SATA controller.

1. You cannot blacklist ahci (like in example here) because you will prevent all controllers from working (=no boot volume)
2. You cannot use modprobe.d to set options because vfio-pci loads waaaaay too late.

Solution

There are two prerequisites:

1. vfio-pci must be availbale before rootfs is attached
2. vfio-pci must load before ahci loads

The first is simple:

• add vfio-pci to /etc/initramfs-tools/modules
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• it will place the module in initramfs disk (in /etc/conf/modules)

The second is more complicated:

• entry in /etc/initramfs-tools/modules will load vfio-pci before the rootfs is mounted
• however, /etc/conf/modules from ramdisk is loaded after some scripts (see /init in ramdisk)
• these scripts (scripts/init-top/) load some drivers... and udev... and udev loads ahci
• solution:
  • create /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci with

    #!/bin/sh
    modprobe vfio-pci ids=8086:1c02

  • chmod +x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci
  • edit /usr/share/initramfs-tools/scripts/init-top/udev and change PREREQS="" to PREREQS="load_vfio-pci"
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• note: this will not work if placed in "standard place" (/etc/initramfs-tools/scripts...) as dependencies are not cross-directory and /usr/share comes first

Verify

Without the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

With the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: vfio-pci
	Kernel modules: ahci