Same issue here -- using afternotok seems to fix the problem (thanks for the heads up, @bpow!). I'm using slurm 22.05.0, and can't find any online documentation for slurm using afternotokay. For others looking to use this, these were the errors I was getting before switching to afternotok:

In the task stderr:

sbatch: error: Batch job submission failed: Job dependency problem

So I ran into an interesting problem with line 30, where I was running a workflow that ran the following (BASH-dependent) command in its scripts:

set -euo pipefail

But this is basically telling the system to use sh instead of bash:

system: {
  "job-shell": "/bin/sh"
}

On many systems /bin/sh is just a symlink to /bin/bash, but not Ubuntu, which is what most containers are based on. So this workflow tried to run, within an Ubuntu-based container, and threw the error:

set: Illegal option -o pipefail

Which is to say, it's much safer to instead set:

system: {
  "job-shell": "/bin/bash"
}

Thanks @oneillkza, good shout - I've changed this in the config above :)

Thank you for sharing this @illusional! This saved me quite a lot of time getting cromwell running on AWS parallel cluster slurm+ubuntu22.04.

I also had to change from afternotokay to afternotok.

I made one addition to the local provider to set submit-docker to also use singularity by adding the following.

runtime-attributes: """
  String cacheLocation = "/fsx/resources/environments/containers/"
  String? docker
""",

submit-docker: """docker_subbed=$(sed -e 's/[^A-Za-z0-9._-]/_/g' <<< ${docker})
image=${cacheLocation}/$docker_subbed.sif
lock_path=${cacheLocation}/$docker_subbed.lock

if [ ! -f "$image" ]; then
  singularity pull $image docker://${docker} || echo Failed to pull $image
fi

chmod +x ${script}
singularity exec --bind ${cwd}:${docker_cwd} $image ${script}""",