Skip to content

Instantly share code, notes, and snippets.

@KiaraGrouwstra

KiaraGrouwstra/bisect-nix.md

Last active February 7, 2026 12:32
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save KiaraGrouwstra/70bf11002032b3c265512f4e17607631 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save KiaraGrouwstra/70bf11002032b3c265512f4e17607631 to your computer and use it in GitHub Desktop.
Download ZIP
debugging nix builds using git bisect
Raw
bisect-nix.md

⚠️ obsolete, instead see https://wiki.nixos.org/wiki/Bisecting

Bisecting is a feature of version control systems such as Git and Mercurial to easily pinpoint regressions. Owing to their reproducibility, Nix and NixOS are well-suited to this. As a result, there are a few common use-cases for Nix.

(If you don't have git available, you can make it available in a shell using:)

nix --extra-experimental-features 'nix-command flakes' shell nixpkgs#git

System flake

git bisect start $BAD $GOOD && git bisect run $CMD

nixpkgs package

either:

  • nix-bisect: smartly pick bisect bad/skip in automated bisects and give nicer outputs
  • hydrasect: prioritize cached commits in nixpkgs bisects - unfortunately still lacks nix package/flake
    • after each checkout run git checkout $(hydrasect-search | head -1)
  • nixpkgs-staging-bisecter: like hydrasect but minimize how many derivations you will build even in staging
  • run in nixpkgs repo, e.g. for signal-desktop:
cat >> .test<< EOF
#! /usr/bin/env bash
$(nix-build -A signal-desktop)/bin/signal-desktop --use-tray-icon --no-sandbox
EOF
chmod +x ./test
git bisect start -- pkgs/applications/networking/instant-messengers/signal-desktop/
git bisect bad
git bisect run sh -c './test; [ $? -eq 0 ]'

Flake input

Place bisect.sh in the system repo, then from there run:

chmod +x ./bisect.sh
git stash push flake.lock

From the dependency's repo, run steps like:

git bisect start $BAD $GOOD

export SYSTEM_REPO=/etc/nixos
export DEPENDENCY_INPUT=$(basename $PWD)  # e.g. nixpkgs
export DEPENDENCY_URL=https://github.com/NixOS/nixpkgs  # example

Then, if judgement needs manual intervention, run cycles of:

$SYSTEM_REPO/bisect.sh $DEPENDENCY_INPUT $DEPENDENCY_URL $SYSTEM_REPO
# git checkout $(hydrasect-search | head -1)  # only if the dep is nixpkgs
# <TEST>
git bisect good
git bisect bad

Or, if judgement can be automated (COMMAND optional, will rebuild):

git bisect run $SYSTEM_REPO/bisect.sh $DEPENDENCY_INPUT $DEPENDENCY_URL $SYSTEM_REPO && <COMMAND>
Raw
bisect.sh
#!/usr/bin/env -S nix --extra-experimental-features 'nix-command flakes' shell nixpkgs#git nixpkgs#jaq --command sh
set -xe
# script for `git bisect` for pinpointing regressions in flake inputs, see:
# https://gist.github.com/KiaraGrouwstra/70bf11002032b3c265512f4e17607631
# to be run from dep's perspective after: `git bisect $BAD $GOOD`
export DEPENDENCY_INPUT="${1:-nixpkgs}"
export DEPENDENCY_URL="${2:-https://github.com/nixos/nixpkgs}"
export SYSTEM_REPO="${3:-/etc/nixos}"
export FLAKE_PROFILE="${4:-default}"
LOCK_FILE=$SYSTEM_REPO/flake.lock
CURRENT_COMMIT=$(jaq -r ".nodes[\"${DEPENDENCY_INPUT}\"].locked.rev" $LOCK_FILE)
TARGET_COMMIT=$(git rev-parse HEAD)
sed -E -i "s/$CURRENT_COMMIT/$TARGET_COMMIT/g" $LOCK_FILE
CURRENT_HASH=$(jaq -r ".nodes[\"${DEPENDENCY_INPUT}\"].locked.narHash" $LOCK_FILE)
TARGET_HASH=$(nix flake prefetch "${DEPENDENCY_URL}/archive/${TARGET_COMMIT}.zip" --json | jaq -r .hash)
sed -E -i "s/$CURRENT_HASH/$TARGET_HASH/g" $LOCK_FILE
export DEPENDENCY_PATH=$PWD
cd $SYSTEM_REPO
sudo nixos-rebuild boot --flake ".#default"
cd $DEPENDENCY_PATH
@KiaraGrouwstra
Copy link
Author

KiaraGrouwstra commented Jul 1, 2024

looks like there's git bisect run sudo nixos-rebuild dry-activate --override-input nixpkgs . --flake /etc/nixos#default 😳

@KiaraGrouwstra
Copy link
Author

KiaraGrouwstra commented Jan 22, 2026 •
edited
Loading

bisecting these is non-trivial given how long this can take - esp given for mastodon for one it doesn't seem to be the sole commit updating the mastodon version that broke itupdate it did cause the Vite MissingExecutableError.
reducing rebuilds may need a combination of bisecting tools:

  • hydrasect: for the initial part of the bisect where hydra-cached commits are available, helps identify those commits (git checkout $(hydrasect | head -1)) to prevent local builds.
  • nixpkgs-staging-bisecter: for the later part of bisects, when hydra-cached commits are no longer available, --dry-runs hard to check out a cheap commit to test (bisecter.py --checkout nix-build ...) while minimizing rebuilds.
  • nix-bisect: helps better judge outcome (git bisect's good vs bad / skip) and gives nicer outputs than git bisect run.
    • bisect-env --try-pick REV CMD ARGS: runs command with cherry-picks.
    • extra-bisect: run skips on command failure (125) / user exit (129), env passes thru to bisect-env with existing patchsets (but no cherry-picks).
    • nix-build-status: lazily determine the status of a nix-instantiate in a bisect-friendly format. args: drvish / --file=. / --failure-line=None.

this could be achieved like e.g.:

#!/bin/sh
# bisect.sh
# sample usage: cd $HOME/code/nixpkgs && git bisect start BAD GOOD && extra-bisect run sh PATH/TO/bisect.sh

export DEP_DIR="${HOME}/code/nixpkgs"
export FLAKE_DIR="${HOME}/code/fediversity"
export NPINS_OVERRIDE_nixpkgs="${DEP_DIR}"
export ATTR="checks.x86_64-linux.test-mastodon-service"
export FAILURE_LINE='MissingExecutableError'

cached_commits="$(hydrasect 2>/dev/null)"
if [[ "$?" -eq 0 ]]; then
    cached_commit="$(echo $cached_commits | head -1)"
    echo "next cached commit: ${cached_commit}"
    git checkout -f "${cached_commit}"
else
    echo 'uncached'
    nixpkgs-staging-bisecter --checkout nix build --impure -f "${FLAKE_DIR}" ".#${ATTR}" -L
fi

bisect-env \
    --try-pick c155a84d2293b902521be32d6deae83186e127f6 \
    nix-build-status --file="${FLAKE_DIR}" --failure-line="${FAILURE_LINE}" --flake --argstr impure true "${ATTR}"

such a script depends on various questions:

  • is the project to be bisected distinct from the project being built? if so:
    • DEP_NAME: what is the dependency's alias in the project to be built?
    • DEP_DIR: what is the root directory (for the purpose of version control) of the dependency?
  • is the failing build built thru flakes? (yes: nix-bisect/bin/extra-bisect gets --flake. further implies different ways of overriding the dependency, if applicable, as per above: export NPINS_OVERRIDE_${DEP_NAME | snake_case}=${DEP_DIR} + --argstr impure true for npins, vs. --override-input "${DEP_NAME}" "${DEP_DIR}" for flakes)
  • is the project to be bisected nixpkgs? (if yes, use nixpkgs-staging-bisecter, as well as git bisect start's --first-parent - if not also hydrasect as implied by hydra builds for caching, see below.)
  • does the project to be bisected use hydra builds for caching? (implied by e.g. nixpkgs from above - if yes, use hydrasect (including preceding hydrascrape).) - just supports nixpkgs so far.
  • is the failing build a NixOS build operation? (yes: ATTR_PATH += .system.build.toplevel)
  • does the build failure stem from a dependency's build failure? (yes: nix-bisect/bin/extra-bisect gets --on-dependency-failure bad)
  • what patches should be applied to the dependency, if any? (e.g. in case some commit is no longer available to fetch - if any: use bisect-env with a --try-pick <commit> per commit to be cherry-picked)
  • BUILD_DIR: what is the path to the directory containing the entrypoint of the failing build?
  • ATTR_PATH: what is the attribute path (i.e. may contains .s) of the failing dependency?
  • what version control system is used by the project to bisect? (e.g. git, ... - git is needed for nix-bisect, hydrasect, nixpkgs-staging-bisecter, if not also by --first-parent. git is implied by bisecting nixpkgs or anything with fetch-tree scheme git / github / gitlab / sourcehut.)
  • BAD: what is the (oldest) known revision failing due to the regression?
  • GOOD: what is the (newest) known revision that succeeded, so before the regression?
  • FAILURE_LINE: what error line would identify the build failure?
  • NIXPKGS_STAGING_BISECTER_PY: where is your nixpkgs-staging-bisecter python script located? (e.g. $(nix-instantiate --eval --raw '(import <nixpkgs> { }).callPackage (builtins.fetchTree "github:KiaraGrouwstra/nixpkgs-staging-bisecter?branch=default-nix").outPath { }')/bin/nixpkgs-staging-bisecter)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment