Here we will explain the steps needed to install fenics development version on ex3 (the HPC cluster at Simula).
Note that the recipe here is rather general and could probalby be adopted to most HPC systems. Note that at the time of writing there is an open PR to add FEniCS devlopment version, and this has to merged (or you need to use my fork) to make it work.
First we need a working C and Fortran compiler before installing anything. To get this I used
module purge
module use /cm/shared/ex3-modules/latest/modulefiles
module load gcc-10.1.0
module load libgfortran-5.0.0
Next you need to get spack. The easiest way to do this is
git clone https://github.com/spack/spack.git
. ./spack/share/spack/setup-env.sh
Now before you continue, you need to figure out where (or which node) you want to install FEnICs.
For example say you want to install it on the defq partition. Then you should start an interactive session on that node, e.g first load slurm
module load slurm
and then run an interative session usign srun
srun --partition=defq --nodes=1 --ntasks-per-node=1 --time=05:00:00 --pty bash -i
Now we will create a new envionrment
spack env create fenics-dev-defq
spack env activate fenics-dev-defq
and we will add fenics developement version
spack add fenics@=master%gcc@=10.1.0 + python ^[email protected] py-pip
Here we also specify that we want python3.10 (I had some issues with python3.11) and we also add pip so that we can add addional pacakges (i.e installing our problem specific software). I aslo set the environment variable PETSC_ARCH to linux-gnu-real-32 (not sure if this is strictly necessary, but better safe than sorry 😊), i.e
export PETSC_ARCH=linux-gnu-real-32
Finally you want do
spack concretize
spack install
which will take a few hours to complete.
Now, in your submit script you can e.g do
#!/bin/bash
#SBATCH --job-name="my-fenics-job"
#SBATCH --partition=defq
#SBATCH --time=3-00:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --output=%j-%x-stdout.txt
#SBATCH --error=%j-%x-stderr.txt
module use /cm/shared/ex3-modules/latest/modulefiles
module load gcc-10.1.0
module load libgfortran-5.0.0
. /home/henriknf/local/src/spack/share/spack/setup-env.sh
spack env activate fenics-dev-defq
python3 -c "import dolfin; print(dolfin.__version__)"