A scheduler exists because the cluster is shared. Everything confusing about Slurm follows from that: why your job waits, why it was killed, why the node looks idle but will not take it.
Batch scripts and #SBATCH directives. What
--nodes, --ntasks and --cpus-per-task each mean,
and why asking for the wrong one wastes an allocation.
Reading the REASON column. Resources, Priority, ReqNodeNotAvail, Dependency, QOSMaxJobsPerUser — each means something different about what to do next.
squeue, sinfo and
scontrol show are three views of one truth. Learning to cross-check
them is most of troubleshooting.
idle, mix, alloc, drain, down. What drains a node, what resumes it, and what happens to the jobs that were on it.
sacct after the fact: exit codes, MaxRSS,
elapsed against the limit. How to tell OOM from a timeout from a bug.
srun --pty bash and salloc:
getting a shell on a compute node, and why that is not the same as ssh.