Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save HanStolpo/5f5afc171bbae8788cef2d82d0315ce4 to your computer and use it in GitHub Desktop.

Select an option

Save HanStolpo/5f5afc171bbae8788cef2d82d0315ce4 to your computer and use it in GitHub Desktop.
Install nixos on an Equinix-metal server c3.medium.x86 via iPXE booting into an Ubuntu live disk

Install nixos on equinix-metal system over iPXE by first booting into Ubuntu live disk available over https://boot.netboot.xyz . I needed to go via the Ubuntu live disk since the NixOS network install does not work on equinix-metal at the moment. This is for the c3.medium.x86 configuration.


Derived from https://gist.github.com/chris-martin/4ead9b0acbd2e3ce084576ee06961000 with its introduction quoted below

These are instructions for booting from an Ubuntu liveCD and installing NixOS on a machine. I needed to do this because the > NixOS liveCD doesn't work on my machine (NixOS/nixpkgs#5829), so I'm just using the Ubuntu installation media as something to boot into.

Much of this is from discussion at NixOS/nixpkgs#14680.


Equinex-metal's documentation on iPXE boot can be found here https://metal.equinix.com/developers/docs/operating-systems/custom-ipxe/

iPXE boot using the url https://boot.netboot.xyz and select one of the Ubuntu live disk images

Login using the user name ubuntu, no password required.

Make your self super user, enable DNS, add ssh remote access

sudo -s
cd /root
resolvectl dns enp65s0f0 8.8.8.8
apt-get install openssh-server
mkdir .ssh
curl https://github.com/{your-git-hub-user-name}.keys > .ssh/authorized_keys

Login via SSH for a nicer terminal experience

create the file partitions, a boot and swap partition on the one drive and the rest all part of one logical volume.

parted /dev/sdd -- mklabel gpt
parted /dev/sdd -- mkpart ESP fat32 1MiB 512MiB
parted /dev/sdd -- set 1 esp on
parted /dev/sdd -- mkpart primary 512MiB -64Gib
parted /dev/sdd -- mkpart primary linux-swap  171Gib 100%
parted /dev/sda -- mklabel gpt
parted /dev/sda -- mkpart primary 0% 100%
parted /dev/sdb -- mklabel gpt
parted /dev/sdb -- mkpart primary 0% 100%
parted /dev/sdc -- mklabel gpt
parted /dev/sdc -- mkpart primary 0% 100%
vgcreate dm0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd2
lvcreate -n dm0 -l 100%VG dm0
mkfs.ext4 -L nixos /dev/dm-0
mkfs.ext4 -L boot /dev/sdd1
mkswap -L swap /dev/sdd3

Obtain the ISO.

wget https://nixos.org/nixos/download.html
ISO_URL=$(cat download.html | grep minimal | grep x86 | sed -r 's#.*"(https://.*\.iso)".*#\1#' | grep -v sha256)
wget $ISO_URL -O nixos.iso

Mount the ISO.

mkdir nixos
mount -o loop nixos.iso nixos

Extract the Nix store. We won't fully install Nix in Ubuntu, but we'll use a few of the executables from the store.

mkdir /nix
unsquashfs -d /nix/store nixos/nix-store.squashfs

Find the relevant programs in the nix store and add them to PATH.

NIXOS_INSTALL=$(find /nix/store -path '*-nixos-install/bin/nixos-install')
NIX_INSTANTIATE=$(find /nix/store -path '*-nix-*/bin/nix-instantiate')
NIXOS_GENERATE_CONFIG=$(find /nix/store -path '*-nixos-generate-config/bin/nixos-generate-config')
export PATH="$(dirname $NIXOS_INSTALL):$(dirname $NIX_INSTANTIATE):$(dirname $NIXOS_GENERATE_CONFIG):$PATH"

Create the build group and a build user.

groupadd --system nixbld
useradd --system --home-dir /var/empty --shell $(which nologin) -g nixbld -G nixbld nixbld0

Obtain a copy of the Nix package repo and construct a NIX_PATH.

NIX_REV=$(echo $ISO_URL | sed -r 's~.*(nixos-.*)/latest.*~\1~')
wget https://github.com/nixos/nixpkgs/archive/$NIX_REV.zip -O nixpkgs.zip
apt-get install zip
unzip nixpkgs.zip
mv nixpkgs-* nixpkgs
export NIX_PATH=nixpkgs=$HOME/nixpkgs

Mount the partitions you created earlier. Note that your device names may differ from these.

mount /dev/disk/by-label/nixos /mnt
mkdir /mnt/boot
mount /dev/disk/by-label/boot /mnt/boot

Copy the nix store from the ISO into where the nix store will be on your new install. (This step isn't necessary, and I'm not sure if it actually helps, but it may spare you from needing to download some things.)

mkdir /mnt/nix
unsquashfs -d /mnt/nix/store nixos/nix-store.squashfs

Get the networking meta data for your equinix-metal server.

apt-get install jq

echo "sshPublicKeys=$(curl -s https://metadata.platformequinix.com/metadata | jq '.ssh_keys')"

echo "hostName=$(curl -s https://metadata.platformequinix.com/metadata | jq '.hostname')"

echo "ipv4Address=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 4) | .address')"
echo "ipv4InternalAddress=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == false) | select(.address_family == 4) | .address')"
echo "ipv6Address=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 6) | .address')"

echo "ipv4Gateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 4) | .gateway')"
echo "ipv4Internalipv4InternalGateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == false) | select(.address_famil== 4) | .gateway')"
echo "ipv6ipv6Gateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 6) | .gateway')"

echo "eth0MacAddress=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.interfaces[] | select(.name == "eth0") | .mac')"

copy the following nix configuration to /mnt/etc/nixos/configuration.nix replacing the variables in the let block with the values obtained above.

{ config, pkgs, modulesPath, ... }:

let
  sshPublicKeys = [ ];

  hostName = "";

  ipv4PublicAddress = "";
  ipv4InternalAddress = "";
  ipv6Address = "";

  ipv4Gateway = "";
  ipv4InternalGateway = "";
  ipv6Gateway = "";

  eth0MacAddress = "";
in

{
  ########################
  # initial hardware configuration
  ########################
  imports =
    [
      (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [
    "mpt3sas"
    "sd_mod"
    "xhci_pci"
    "ahci"
  ];
  boot.initrd.kernelModules = [ "dm-snapshot" ];
  boot.kernelModules = [
    "kvm-amd"
    "dm_multipath"
    "dm_round_robin"
  ];
  boot.extraModulePackages = [];

  # emergency terminal access
  boot.kernelParams = [ "console=ttyS1,115200n8" ];

  ########################
  # filesystems and boot
  ########################
  fileSystems = {
    "/" = {
      label = "nixos";
      fsType = "ext4";
    };
    "./boot" = {
      label = "boot";
      fsType = "vfat";
    };
  };

  boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.systemd-boot = {
    enable = true;
    editor = false;
  };

  swapDevices = [
    {
      label = "swap";
    }
  ];

  ########################
  # allow ssh login
  ########################
  services.openssh.enable = true;
  services.openssh.permitRootLogin = "prohibit-password";
  services.openssh.passwordAuthentication = false;
  security.pam.enableSSHAgentAuth = true;
  users.users.root.openssh.authorizedKeys.keys = sshPublicKeys;

  # Set your time zone.
  time.timeZone = "Etc/UTC";

  networking.hostName = hostName;

  ########################
  # network setup
  ########################
  networking.useNetworkd = true;
  networking.useDHCP = false;
  networking.nameservers = [
    "147.75.207.207"
    "147.75.207.208"
  ];

  networking.bonds.bond0 = {
    driverOptions = {
      downdelay = "200";
      lacp_rate = "fast";
      miimon = "100";
      mode = "802.3ad";
      updelay = "200";
      xmit_hash_policy = "layer3+4";
    };

    interfaces = [
      "eth0"
      "eth1"
    ];
  };

  networking.interfaces.bond0 = {
    ipv4 = {
      addresses = [
        {
          address = ipv4PublicAddress;
          prefixLength = 31;
        }
        {
          address = ipv4InternalAddress;
          prefixLength = 31;
        }
      ];
      routes = [
        {
          address = "10.0.0.0";
          prefixLength = 8;
          via = ipv4InternalGateway;
        }
      ];
    };

    ipv6 = {
      addresses = [
        {
          address = ipv6Address;
          prefixLength = 127;
        }
      ];
    };

    # https://github.com/NixOS/nixpkgs/issues/69360
    macAddress = eth0MacAddress;
  };
  # so we don't have to guess the names of the interfaces when creating
  # the bond
  networking.usePredictableInterfaceNames = false;
  # This won't be necessary if the following patches are ever merged:
  #
  # https://github.com/NixOS/nixpkgs/pull/93589
  # https://github.com/NixOS/nixpkgs/pull/93598
  # https://github.com/NixOS/nixpkgs/pull/93635
  systemd.network.networks."40-bond0" = {
    gateway = [
      ipv4Gateway
      ipv6Gateway
    ];
    linkConfig = {
      MACAddress = eth0MacAddress;
    };
    # The bond0 interface keept getting stuck in a configuring state causing
    # systemd-networkd-wait-online.service to fail. The network would still
    # be up though, since the link was routable but configuring, and doing
    # `networkctl reconfigure bond0` would magically set the link to configured.
    # Binding the carrier to the interfaces seems to have fixed this issue.
    networkConfig = {
      BindCarrier = [ "eth0" "eth1" ];
    };
    routes = [
      {
        routeConfig = {
          Destination = "10.0.0.0/8";
          Gateway = ipv4InternalGateway;
        };
      }
    ];
  };

  ########################
  # initial packages to install
  ########################
  environment.systemPackages = with pkgs; [
    vim
    wget
    curl
    zip
    unzip
    jq
  ];

  # List services that you want to enable:

  system.stateVersion = "21.05"; # Did you read the comment?
}

Initiate the nixos installation

nixos-install --root /mnt

Reboot into the emergency console and enable UEFI boot in the bios.

Disable always boot from PXE on your equinix-metal dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment