Install nixos on equinix-metal system over iPXE by first booting into Ubuntu live disk available over https://boot.netboot.xyz . I needed to go via the Ubuntu live disk since the NixOS network install does not work on equinix-metal at the moment. This is for the c3.medium.x86 configuration.
Derived from https://gist.github.com/chris-martin/4ead9b0acbd2e3ce084576ee06961000 with its introduction quoted below
These are instructions for booting from an Ubuntu liveCD and installing NixOS on a machine. I needed to do this because the > NixOS liveCD doesn't work on my machine (NixOS/nixpkgs#5829), so I'm just using the Ubuntu installation media as something to boot into.
Much of this is from discussion at NixOS/nixpkgs#14680.
Equinex-metal's documentation on iPXE boot can be found here https://metal.equinix.com/developers/docs/operating-systems/custom-ipxe/
iPXE boot using the url https://boot.netboot.xyz and select one of the Ubuntu live disk images
Login using the user name ubuntu, no password required.
Make your self super user, enable DNS, add ssh remote access
sudo -s
cd /root
resolvectl dns enp65s0f0 8.8.8.8
apt-get install openssh-server
mkdir .ssh
curl https://github.com/{your-git-hub-user-name}.keys > .ssh/authorized_keysLogin via SSH for a nicer terminal experience
create the file partitions, a boot and swap partition on the one drive and the rest all part of one logical volume.
parted /dev/sdd -- mklabel gpt
parted /dev/sdd -- mkpart ESP fat32 1MiB 512MiB
parted /dev/sdd -- set 1 esp on
parted /dev/sdd -- mkpart primary 512MiB -64Gib
parted /dev/sdd -- mkpart primary linux-swap 171Gib 100%
parted /dev/sda -- mklabel gpt
parted /dev/sda -- mkpart primary 0% 100%
parted /dev/sdb -- mklabel gpt
parted /dev/sdb -- mkpart primary 0% 100%
parted /dev/sdc -- mklabel gpt
parted /dev/sdc -- mkpart primary 0% 100%
vgcreate dm0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd2
lvcreate -n dm0 -l 100%VG dm0
mkfs.ext4 -L nixos /dev/dm-0
mkfs.ext4 -L boot /dev/sdd1
mkswap -L swap /dev/sdd3Obtain the ISO.
wget https://nixos.org/nixos/download.html
ISO_URL=$(cat download.html | grep minimal | grep x86 | sed -r 's#.*"(https://.*\.iso)".*#\1#' | grep -v sha256)
wget $ISO_URL -O nixos.isoMount the ISO.
mkdir nixos
mount -o loop nixos.iso nixosExtract the Nix store. We won't fully install Nix in Ubuntu, but we'll use a few of the executables from the store.
mkdir /nix
unsquashfs -d /nix/store nixos/nix-store.squashfsFind the relevant programs in the nix store and add them to PATH.
NIXOS_INSTALL=$(find /nix/store -path '*-nixos-install/bin/nixos-install')
NIX_INSTANTIATE=$(find /nix/store -path '*-nix-*/bin/nix-instantiate')
NIXOS_GENERATE_CONFIG=$(find /nix/store -path '*-nixos-generate-config/bin/nixos-generate-config')
export PATH="$(dirname $NIXOS_INSTALL):$(dirname $NIX_INSTANTIATE):$(dirname $NIXOS_GENERATE_CONFIG):$PATH"Create the build group and a build user.
groupadd --system nixbld
useradd --system --home-dir /var/empty --shell $(which nologin) -g nixbld -G nixbld nixbld0Obtain a copy of the Nix package repo and construct a NIX_PATH.
NIX_REV=$(echo $ISO_URL | sed -r 's~.*(nixos-.*)/latest.*~\1~')
wget https://github.com/nixos/nixpkgs/archive/$NIX_REV.zip -O nixpkgs.zip
apt-get install zip
unzip nixpkgs.zip
mv nixpkgs-* nixpkgs
export NIX_PATH=nixpkgs=$HOME/nixpkgsMount the partitions you created earlier. Note that your device names may differ from these.
mount /dev/disk/by-label/nixos /mnt
mkdir /mnt/boot
mount /dev/disk/by-label/boot /mnt/bootCopy the nix store from the ISO into where the nix store will be on your new install. (This step isn't necessary, and I'm not sure if it actually helps, but it may spare you from needing to download some things.)
mkdir /mnt/nix
unsquashfs -d /mnt/nix/store nixos/nix-store.squashfsGet the networking meta data for your equinix-metal server.
apt-get install jq
echo "sshPublicKeys=$(curl -s https://metadata.platformequinix.com/metadata | jq '.ssh_keys')"
echo "hostName=$(curl -s https://metadata.platformequinix.com/metadata | jq '.hostname')"
echo "ipv4Address=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 4) | .address')"
echo "ipv4InternalAddress=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == false) | select(.address_family == 4) | .address')"
echo "ipv6Address=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 6) | .address')"
echo "ipv4Gateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 4) | .gateway')"
echo "ipv4Internalipv4InternalGateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == false) | select(.address_famil== 4) | .gateway')"
echo "ipv6ipv6Gateway=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.addresses[] | select(.public == true) | select(.address_family == 6) | .gateway')"
echo "eth0MacAddress=$(curl -s https://metadata.platformequinix.com/metadata | jq '.network.interfaces[] | select(.name == "eth0") | .mac')"copy the following nix configuration to /mnt/etc/nixos/configuration.nix replacing the
variables in the let block with the values obtained above.
{ config, pkgs, modulesPath, ... }:
let
sshPublicKeys = [ ];
hostName = "";
ipv4PublicAddress = "";
ipv4InternalAddress = "";
ipv6Address = "";
ipv4Gateway = "";
ipv4InternalGateway = "";
ipv6Gateway = "";
eth0MacAddress = "";
in
{
########################
# initial hardware configuration
########################
imports =
[
(modulesPath + "/installer/scan/not-detected.nix")
];
boot.initrd.availableKernelModules = [
"mpt3sas"
"sd_mod"
"xhci_pci"
"ahci"
];
boot.initrd.kernelModules = [ "dm-snapshot" ];
boot.kernelModules = [
"kvm-amd"
"dm_multipath"
"dm_round_robin"
];
boot.extraModulePackages = [];
# emergency terminal access
boot.kernelParams = [ "console=ttyS1,115200n8" ];
########################
# filesystems and boot
########################
fileSystems = {
"/" = {
label = "nixos";
fsType = "ext4";
};
"./boot" = {
label = "boot";
fsType = "vfat";
};
};
boot.loader.efi.canTouchEfiVariables = true;
boot.loader.systemd-boot = {
enable = true;
editor = false;
};
swapDevices = [
{
label = "swap";
}
];
########################
# allow ssh login
########################
services.openssh.enable = true;
services.openssh.permitRootLogin = "prohibit-password";
services.openssh.passwordAuthentication = false;
security.pam.enableSSHAgentAuth = true;
users.users.root.openssh.authorizedKeys.keys = sshPublicKeys;
# Set your time zone.
time.timeZone = "Etc/UTC";
networking.hostName = hostName;
########################
# network setup
########################
networking.useNetworkd = true;
networking.useDHCP = false;
networking.nameservers = [
"147.75.207.207"
"147.75.207.208"
];
networking.bonds.bond0 = {
driverOptions = {
downdelay = "200";
lacp_rate = "fast";
miimon = "100";
mode = "802.3ad";
updelay = "200";
xmit_hash_policy = "layer3+4";
};
interfaces = [
"eth0"
"eth1"
];
};
networking.interfaces.bond0 = {
ipv4 = {
addresses = [
{
address = ipv4PublicAddress;
prefixLength = 31;
}
{
address = ipv4InternalAddress;
prefixLength = 31;
}
];
routes = [
{
address = "10.0.0.0";
prefixLength = 8;
via = ipv4InternalGateway;
}
];
};
ipv6 = {
addresses = [
{
address = ipv6Address;
prefixLength = 127;
}
];
};
# https://github.com/NixOS/nixpkgs/issues/69360
macAddress = eth0MacAddress;
};
# so we don't have to guess the names of the interfaces when creating
# the bond
networking.usePredictableInterfaceNames = false;
# This won't be necessary if the following patches are ever merged:
#
# https://github.com/NixOS/nixpkgs/pull/93589
# https://github.com/NixOS/nixpkgs/pull/93598
# https://github.com/NixOS/nixpkgs/pull/93635
systemd.network.networks."40-bond0" = {
gateway = [
ipv4Gateway
ipv6Gateway
];
linkConfig = {
MACAddress = eth0MacAddress;
};
# The bond0 interface keept getting stuck in a configuring state causing
# systemd-networkd-wait-online.service to fail. The network would still
# be up though, since the link was routable but configuring, and doing
# `networkctl reconfigure bond0` would magically set the link to configured.
# Binding the carrier to the interfaces seems to have fixed this issue.
networkConfig = {
BindCarrier = [ "eth0" "eth1" ];
};
routes = [
{
routeConfig = {
Destination = "10.0.0.0/8";
Gateway = ipv4InternalGateway;
};
}
];
};
########################
# initial packages to install
########################
environment.systemPackages = with pkgs; [
vim
wget
curl
zip
unzip
jq
];
# List services that you want to enable:
system.stateVersion = "21.05"; # Did you read the comment?
}Initiate the nixos installation
nixos-install --root /mntReboot into the emergency console and enable UEFI boot in the bios.
Disable always boot from PXE on your equinix-metal dashboard.