Skip to content

Instantly share code, notes, and snippets.

@mbbx6spp

mbbx6spp/.00readme.org

Last active May 26, 2024 22:53
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save mbbx6spp/731076cb8fc620b064b8e5b28fb1c796 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save mbbx6spp/731076cb8fc620b064b8e5b28fb1c796 to your computer and use it in GitHub Desktop.
Download ZIP
A caching and persistent Nix shell with direnv
Raw
.00readme.org

Cached and Persistent Nix shell with direnv integration

Problem being solved

  • the default direnv Nix integration from projects loads a Nix shell every new terminal or every time shell.nix and/or default.nix changes. On larger projects with a lot of Nix shell dependencies, that can cause the terminal to take more than 6 seconds to load, which significantly degrades the developer experience (DX).
  • when a developer garbage collects in their Nix store, often the Nix shell dependencies are deleted also which causes a slow start the next time the project’s Nix shell is requested, also degrading developer experience (DX).

Solution outline

  • on first Nix shell load, the Nix shell is evaluated fully such that direnv dumps the environment from the spawn Nix shell process
  • on subsequent loads, a cache directory is inspected to see if the shell was already loaded, and applies the environment settings from the original loading of the Nix shell evaluation
  • to determine freshness of shell evaluation we update the nixexprs array in the Bash script above to ensure (see TODO comment in the .direnvrc file), which will watch for changes in that array; these are all used to determine the hash of the cached Nix shell evaluation to ensure consistency
  • to ensure all important GC roots are added to prevent premature GC-ing of the shell’s dependencies, we add roots of all indirects for the shell’s derivation

Caveats

It has two main caveats right now (for the infrequent case where you manually GC Nix packages from your store, ~every few months):

  1. to free up old packages no longer referenced in the Nix shell, you need to rm -rf .direnv.d in your dailykos project root then direnv reload and then nix-collect-garbage -d
  2. after collecting the nix “garbage”, you will then rm -rf .direnv.d and then it should take a few seconds only to pull in some bash doc and dev packages that aren’t creating a GC root under .direnv.d (I am out of ideas why it’s just those two packages).

How to use

  • put the .direnvrc file in your $HOME directory.
  • put the .envrc (and customize as necessary) to your project’s root, adjacent to shell.nix and default.nix.

Related works

  • the direnv project wiki contains similar use_nix overrides but numerous bugs existed based on my requirements, so I put this together to aid my developer experience. Check them out for yourself to compare: https://github.com/direnv/direnv/wiki/Nix
Raw
.direnvrc
# Put this file at ~/.direnvrc
hash_env() {
if has shasum; then
for e in "${@}"; do shasum -a 256 "${PWD}/${e}"; done | shasum -a 256 | cut -c -64
else
fail "do not have shasum to cache environment"
fi
}
load_nix() {
local -r orig_IN_NIX_SHELL="${IN_NIX_SHELL:-}"
local -ra envhash="${1}"
local -r envdrv="${envdir}/drv"
nix-shell --show-trace --run 'direnv dump' > "${envdir}/dump"
direnv apply_dump "${envdir}/dump" > "${envdir}/vars"
if [ -z "${orig_IN_NIX_SHELL:-}" ]; then
unset IN_NIX_SHELL
else
export IN_NIX_SHELL="$orig_IN_NIX_SHELL"
fi
# persist if hash isn't empty
if [ ! -z "${envhash}" ]; then
env IN_NIX_SHELL=1 \
nix-instantiate \
--add-root "${envdrv}" \
--indirect "shell.nix" \
> /dev/null
nix-store \
-r $(nix-store --query --references "${envdrv}") \
--add-root "$(cache_file "${envhash}")/result" \
--indirect \
> /dev/null
fi
}
cache_file() {
echo "${PWD}/.direnv.d/env-${1}"
}
use_nix() {
# TODO change list of files that should be watched by direnv to ensure shell consistency at run-time.
local -ra nixexprs=(shell.nix default.nix version.nix nixpkgs.nix etc/shellHook)
local -r hash="$(hash_env ${nixexprs[@]})"
local -r envdir="$(cache_file "${hash}")"
mkdir -p "${envdir}"
if [ -z "${hash:-}" ]; then
load_nix "${hash}"
else
if [ -e "${PWD}/.direnv.d/env-${hash}/drv" ]; then
log_status "using cached environment (${hash})"
else
load_nix "${hash}"
fi
fi
for e in "${nixexprs[@]}"; do
watch_file "${e}"
done
watch_file "${direnv}/drv"
watch_file "${direnv}"
source_env "${envdir}/vars"
}
Raw
.envrc
# sample .envrc file for your project that contains shell.nix, etc
# if you have a .env that needs to be loaded by dotenv you should include this, otherwise remove these two lines
dotenv
# this is the critical peice
use nix
@vyorkin
Copy link

vyorkin commented Dec 17, 2019

Is there any advantages against https://github.com/target/lorri ?

@amarrella
Copy link

amarrella commented May 5, 2020

I had to edit https://gist.github.com/mbbx6spp/731076cb8fc620b064b8e5b28fb1c796#file-direnvrc-L12 to remove the read-only parameter to make it work (I use darwin). Not sure if it's expected

@SuzanneSoy
Copy link

SuzanneSoy commented May 22, 2020 •
edited
Loading 

  if [ ! -z "${hash}" ]; then

on line 25, I guess that was meant to be ${envhash}, not ${hash} ?

@mbbx6spp
Copy link
Author

mbbx6spp commented May 26, 2024 •
edited
Loading

Is there any advantages against https://github.com/target/lorri ?

It eliminated the need to install a heavyweight dependency that does something very simple.

I had to edit https://gist.github.com/mbbx6spp/731076cb8fc620b064b8e5b28fb1c796#file-direnvrc-L12 to remove the read-only parameter to make it work (I use darwin). Not sure if it's expected

This is due to Darwin shipping a 12+ year old version of bash.

on line 25, I guess that was meant to be ${envhash}, not ${hash} ?

Yes, thank you. Fixing for posterity even though this is now already folded in to nix-direnv and offered implicitly by Nix Flakes.

:)

PSA: Nobody should be using the above shared in Sept 2018!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment