A login node and twenty-one compute nodes, with a scheduler that really allocates them. Submit a job and it lands on named hardware; drain a node and the queue notices. Slurm, Spack and Apptainer on the same machines, because that is where you meet them.
On a real cluster these are not separate subjects. You build software with Spack, wrap it with Apptainer, and run it through Slurm — and the interesting failures happen where they meet. So they share a cluster here too.
Submit, inspect and troubleshoot work on a cluster you do not own. Partitions, limits, priorities, node states — and why your job is still pending.
Build a scientific stack from source without root. Specs, variants, concretization, environments — and what a module actually is.
Containers that run as you, not as root. Definition files, SIF images, bind mounts, GPU passthrough — and running one inside a batch job.
The cluster keeps one record of which job holds which cores. Every view — squeue, sinfo, scontrol show node — is computed from it, so they cannot disagree. Nothing prints a success it did not perform.
--nodes=4 takes four nodes, or waits. Ask for more cores than the
partition has and it stays pending with the reason a real controller
would give. Cancel it and the cores come back.
Submitting writes to slurmctld.log. Starting writes to the slurmd log on each allocated node. You can grep for your own job ID and watch the allocation happen.
Pipes, redirects, command substitution, separate stdout and stderr, vim and nano, ssh between nodes, a filesystem with permissions. The scheduler is the new part; the machine underneath is not.