Install

• Follow the official ZFSBootMenu guide until the last section Prepare for first boot.

  • My zroot is a mirror, so I have also created a mirror for my EFI boot partitions. Important point here is to use metadata version 1.0 to store all md-related stuff at the end of partition. Each partition in the mirror can be used as regular FAT32 EFI boot partition.

    mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 --metadata=1.0 --homehost=<future_host_name> --name=efi /dev/nvme0n1p1 /dev/nvme1n1p1
    

  • Then create FAT32 filesystem on /dev/md1 and mount it to /boot/efi (via /etc/fstab).

• Follow the Remote Access to ZFSBootMenu guide. Some hints:

  • Checkout dracut-crypt-ssh repo and copy the contents of 60crypt-ssh directory to /usr/lib/dracut/modules.d/60crypt-ssh.

  • Comment out a few lines at the end of module-setup.sh to not install helpers:

      #install the required helpers
    #  inst "$moddir"/helper/console_auth /bin/console_auth
    #  inst "$moddir"/helper/console_peek.sh /bin/console_peek
    #  inst "$moddir"/helper/unlock /bin/unlock
    #  inst "$moddir"/helper/unlock-reap-success.sh /sbin/unlock-reap-success
    

  • Blacklist a few unnecessary modules in /etc/zfsbootmenu/dracut.conf.d/zfsbootmenu.conf (I have added everything after dash):

    ...
    omit_dracutmodules+=" btrfs zfs resume systemd systemd-initrd dracut-systemd plymouth dash systemd-battery-check systemd-cryptsetup systemd-pcrphase systemd-networkd iscsi nbd nfs "
    ...
    

  • Add ed25519 key to /etc/zfsbootmenu/dracut.conf.d/dropbear.conf:

    ...
    dropbear_ed25519_key=/etc/dropbear/ssh_host_ed25519_key
    ...
    

  • Install isc-dhcp-client (needed for network-legacy dracut module):

    apt install isc-dhcp-client
    

• Proceed with the remaining steps in ZFSBootMenu guide.

• To make sure ZFSBootMenu is rebuilt every time kernel is upated, create the following hook:

  cat > /etc/kernel/postinst.d/zbm <<EOF
  #!/bin/sh
  
  generate-zbm -k "$2"
  EOF
  chmod +x /etc/kernel/postinst.d/zbm
  

ZFS

Auto-import and mount encrypted pools

If you have more encrypted ZFS pools (additionally to the one from which you boot), ZFSBootMenu will not import them. You can import these pools after the first boot (so that the cache is updated and they will be imported automatically next time). To automatically load encryption key (and mount the volumes):

• Create the following service:

  cat > /etc/systemd/system/[email protected] <<EOF
  [Unit]
  Description=Load ZFS encryption key for %I
  DefaultDependencies=no
  After=zfs-import.target
  Before=zfs-mount.service
  
  [Service]
  Type=oneshot
  RemainAfterExit=yes
  ExecStart=/usr/sbin/zfs load-key %I
  
  [Install]
  WantedBy=zfs-mount.service
  EOF
  

• Enable it for your pool(s):

  systemctl enable --now zfs-load-key@<your_pool_name>
  

Notifications with ZED

• Install necessary packages:

  apt install mailutils ssmtp
  

• Configure ssmtp (/etc/ssmtp/ssmtp.conf):

  #
  # Config file for sSMTP sendmail.
  #
  # The person who gets all mail for userids < 1000
  # Make this empty to disable rewriting.
  [email protected]
  
  # The place where the mail goes. The actual machine name is required no
  # MX records are consulted. Commonly mailhosts are named mail.domain.com
  mailhub=my.smtp.server:587
  
  AuthUser=smtp_username
  AuthPass=smtp_password
  
  # Alternatives are UseTLS and UseSSL. Not all work with all SMTP servers.
  UseSTARTTLS=YES
  
  # Where will the mail seem to come from?
  rewriteDomain=my.domain.tld
  
  # The full hostname
  hostname=thishost.my.domain.tld
  

• Configure from email addresses in /etc/ssmtp/revaliases:

  root:[email protected]
  myuser:[email protected]
  

• Test that it works:

  echo "Testing 123" | mail -s "Test Subject" [email protected]
  

• Configure ZED (/etc/zfs/zed.d/zed.rc, only changed/important configuration is shown, the rest was left unchanged):

  # Email address that will receive alerts.
  # Multiple addresses can be specified if they are delimited by whitespace.
  ZED_EMAIL_ADDR="[email protected]"
  
  # Minimum number of seconds between notifications for a similar event.
  ZED_NOTIFY_INTERVAL_SECS=3600
  
  # If set to 1 - successful jobs will also be reported. If set to 0 - only errors will be reported.
  ZED_NOTIFY_VERBOSE=1
  

• Make sure that ZED service is enabled and running:

  systemctl status zfs-zed.service
  

  • Enable it if necessary:

    systemctl enable --now zfs-zed.service
    

• To test ZED configuration run scrub. After it is completed - you should receive an email.

  zpool scrub zroot
  

Podman

• Install Podman

  apt install podman
  

• Make sure that the user that will be running rootless containers has enough subuids and subgids (check in /etc/subuid and /etc/subgid). If not - add subuids and subgids for the container user:

  usermod --add-subuids 100000-165536 --add-subgids 100000-165536 <your_user>
  

• To auto-start containers on reboot user must always have an active session:

  loginctl enable-linger <your_user>
  

• Make Podman aware of the new /etc/subuid and /etc/subgid files:

  podman system migrate
  

• Enable auto-update for containers:

  systemctl --user enable podman-auto-update.timer
  

• Enable Podman socket:

  systemctl --user enable --now podman.socket
  

• Create directory for quadlets:

  mkdir -p ~/.config/containers/systemd
  

• Use fuse-overlayfs instead ov vfs:

  cat > ~/.config/containers/storage.conf <<EOF
  [storage]
  driver = "overlay"
  
  [storage.options.overlay]
  mount_program = "/usr/bin/fuse-overlayfs"
  EOF
  

Fixing network-online.target

Podman switched to pasta from slirp4netns since v5.0.0. With the default configuration containers do not start after reboot with errors like this:

Couldn't set IPv4 route(s) in guest: Invalid argument

Generated systemd services for Quadlets depend on podman-user-wait-network-online.service, which depends on network-online.target. In case of Debian, network-online.target will start after networking.service, which should guarantee that network interfaces are initialized before anything else starts. Apparently pasta wants more than that. To give it what it wants we need to enable ifupdown-wait-online.service:

systemctl enable --now ifupdown-wait-online.service

By default it waits for the exit code of ifquery --state <iface> to be 0 to consider given interface initialized. This is not enough for pasta. By modifying /etc/default/networking we can chanhge ifupdown-wait-online.service's configuration. Setting WAIT_ONLINE_METHOD=route was not enough. This is the changes I ended up with:

WAIT_ONLINE_METHOD=ping
WAIT_ONLINE_ADDRESS="1.2.3.4"

WAIT_ONLINE_ADDRESS is the address that will be ping'ed. I used the gateway's IP. I did not want to use something like google.com because I want this to work in case of the internet outage.

sysctl tweaks

• We might need to increase the maximum number of keys that the kernel should host for our container user, preventing the “Disk quota exceeded: OCI runtime error” issue:

  echo "kernel.keys.maxkeys=1000" > /etc/sysctl.d/maxkeys.conf
  

• Allow rootless containers to bind port 80 and higher:

  echo "net.ipv4.ip_unprivileged_port_start=80" > /etc/sysctl.d/allow-unprivileged-low-ports.conf
  

• Set memory overcommit mode:

  echo "vm.overcommit_memory=1" > /etc/sysctl.d/overcommit-memory.conf
  

• Increase max buffer size:

  cat > /etc/sysctl.d/max-buffer-size.conf <<EOF
  net.core.rmem_max=7500000
  net.core.wmem_max=7500000
  EOF
  

• Decrease swappiness (if swap is used):

  echo "vm.swappiness=10" > /etc/sysctl.d/swappiness.conf
  

• Apply changes:

  systemctl restart systemd-sysctl.service