see also:

• Job farming with SLURM
• List of SLURM Constraints and its Usage

General

The batch system on SuperMUC-NG is the open-source workload manager SLURM (Simple Linux Utility for Resource management). For details about the SLURM batch system, see Slurm Workload Manager.

Submit hosts are usually login nodes that permit to submit and manage batch jobs.

Intel processors on SuperMUC-NG support the hyperthreading mode which might increase the performance of your application. With hyperthreading, you have to increase the number of MPI tasks per node from 48 to 96 in your job script. Please be aware that with 96 MPI tasks per node each process gets only half of the memory by default. If you need more memory, you have to specify it in your job script and use the fat nodes (see example batch scripts).

List of relevant commands

sacct is currently not working.

Queues (SLURM partitions) and their limits

• Batch queues are called partitions in SLURM. 
• The allocation granularity is multiples of one node (only complete nodes are allocated and accouonted for).
• Scheduling and priorization is based on a multifactor scheme including wait time, job size, partition, and required quality of service.

The following partitions are available. Check with sinfo for more details and special partitions:

srun and mpiexec

With SLURM srun command users can spawn any kind of application, process or task inside a job allocation or directly start executing a parallel job (and indirectly ask SLURM to create the appropriate allocation). It can be a shell command, any single-/multi-threaded executable in binary or script format, MPI application or hybrid application with MPI and OpenMP. When no allocation options are defined with srun command the options from sbatch or salloc are inherited.

Note: srun at LRZ is defined as the alias
srun='I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so /usr/bin/srun'.
Since aliases are not inherited the alias is only available in the login shell or in the initial batch script, everywhere else it falls back to /usr/bin/srun. Use the full syntac in these cases.

Note: mpiexec is the preferred and supported way to start applications. srun might fail for hyperthreaded applications.

salloc / srun  for interactive processing

salloc is used the allocated nodes for interactive processing. The options for resource specification in salloc/srun/sbatch are the same.
Currently, at least --account, --time and --partition must be specified!
srun can be used instead of mpiexec. srun or  mpiexec  execute on the nodes previously allocated by the salloc.
There is no advantage by using salloc  over  sbatch --partition=test in terms of wait time.

login node> salloc -t 00:30:00 -p test -A <my-desired-project-id> -N 4
salloc: Pending job allocation 48417
salloc: job 48417 queued and waiting for resources
salloc: job 48417 has been allocated resources
salloc: Granted job allocation 48417
salloc: Waiting for resource configuration
salloc: Nodes i01r01c01s[03-06] are ready for job

i01r01c01s03> srun  --nodes=2 --ntasks=2 --partition=test hostname
i01r01c01s03.sng.lrz.de
i01r01c01s04.sng.lrz.de

i01r01c01s03> mpiexec -n 5 hostname 
i01r01c01s03.sng.lrz.de
i01r01c01s03.sng.lrz.de
i01r01c01s03.sng.lrz.de
i01r01c01s03.sng.lrz.de
i01r01c01s03.sng.lrz.de

i01r01c01s03> srun -n 5 hostname 
i01r01c01s05.sng.lrz.de
i01r01c01s04.sng.lrz.de
i01r01c01s06.sng.lrz.de
i01r01c01s03.sng.lrz.de
i01r01c01s03.sng.lrz.de

i01r01c01s03> exit
exit
salloc: Relinquishing job allocation 48417
login node> 

sbatch Command / #SBATCH option

Batch job options and resources can be given as command line switches to sbatch (in which case they override script-provided values), or they can be embedded into a SLURM job script as a comment line of the form.

Batch Job Examples

#!/bin/bash 
# Job Name and Files (also --job-name) 
#SBATCH -J jobname
#Output and error (also --output, --error): 
#SBATCH -o ./%x.%j.out 
#SBATCH -e ./%x.%j.err 
#Initial working directory (also --chdir): 
#SBATCH -D ./
#Notification and type
#SBATCH --mail-type=END
#SBATCH --mail-user=<youremail>
# Wall clock limit: 
#SBATCH --time=24:00:00
#SBATCH --no-requeue
#Setup of execution environment
#SBATCH --export=NONE 
#SBATCH --get-user-env
#SBATCH --account=<projectID>
#constraints are optional
#--constraint="scratch&work"

#Important
module load slurm_setup

#... (general part)
#SBATCH --partition=general 
#Number of nodes and MPI tasks per node:
#SBATCH --nodes=128 
#SBATCH --ntasks-per-node=48 
#SBATCH --ntasks-per-core=1
#SBATCH --cpus-per-task=1 
#Run the program: 
mpiexec -n $SLURM_NTASKS ./myprog 

#... (general part) 
#SBATCH --partition=general 
#Number of nodes and MPI tasks per node:
#SBATCH --nodes=128
#SBATCH --ntasks-per-node=96 
#SBATCH --ntasks-per-core=2 
#SBATCH --cpus-per-task=1 
#Note: Needs specific MPI version > 2019
#Run the program: 
#Pinning
#Task0->CPU0, Task1->CPU48 .. 
mpiexec -n $SLURM_NTASKS ./myprog 

#Optional: use more explicit pinning (spreading)
#I_MPI_PIN_PROCESSOR_LIST=0-95
#Task0->CPU0, Task1->CPU1

#... (general part)
#SBATCH --partition=general
#Number of nodes and MPI tasks per node:
#SBATCH --nodes=128
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=6
#Run the program: 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 

#Default Pinning:
#Thread0/Task0 ->CPU0 or CPU48
#Thread1/Task0 ->CPU1 or CPU49
#Thread2/Task0 ->CPU2 or CPU50
mpiexec -n $SLURM_NTASKS ./myprog

#export I_MPI_PIN_CELL=core
#export I_MPI_PIN_DOMAIN=omp:compact
#Thread0/Task0 ->CPU0, Thread1/Task0 ->CPU1
#Thread2/Task0 ->CPU2, Thread3/Task0 ->CPU3#....
#Thread0/Task1 ->CPU6, Thread1/Task1 ->CPU7

#export I_MPI_PIN_CELL=unit
#export I_MPI_PIN_DOMAIN=omp:compact
#Thread0/Task0 ->CPU0, Thread1/Task0 ->CPU1
#Thread2/Task0 ->CPU2  !!! 
#Thread3/Task0 ->CPU48,  Thread4/Task0 ->CPU49, #Thread5/Task0 ->CPU50  !!!
#Thread0/Task1 ->CPU3, Thread1/Task1 ->CPU4 

#explicit pinning with masks
#export I_MPI_PIN_CELL=unit
#export I_MPI_PIN_DOMAIN=[3F,FC0,3F000,FC0000,etc.]

#... (general part) 
#SBATCH --partition=general 
#Number of nodes and MPI tasks per node: 
#SBATCH --nodes=128  
#SBATCH --ntasks-per-node=8  
#SBATCH --cpus-per-task=12 

#Run the program:  
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK 
mpiexec -n $SLURM_NTASKS ./myprog

#Optional: Resulting in same pinning as default 
#export I_MPI_PIN_CELL=unit
#export I_MPI_PIN_DOMAIN=omp
#Thread0/Task0 ->CPU0, Thread1/Task0 ->CPU1
#Thread5/Task0 ->CPU5 !!!
#Thread6/Task0 ->CPU48, Thread7/Task0 ->CPU49

#... (general part)
#SBATCH --partition=fat
#Number of nodes and MPI tasks per node: 
#SBATCH --nodes=64 
#SBATCH --ntasks-per-node=48
#SBATCH --ntasks-per-core=1 
#SBATCH --cpus-per-task=1
#Run the program: 
mpiexec -n $SLURM_NTASKS ./myprog

#... (general part)
#SBATCH --partition=micro
#Number of nodes and MPI tasks per node:
#SBATCH --nodes=1 
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=48
#Run the program:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./myprog
#Default:KMP_AFFINITY=granularity=thread,compact,1,0
#Thread0 ->CPU0 
#Thread1 ->CPU1

#… (general part)  
#SBATCH --partition=large
#Max number of islands and max waittime
#SBATCH --switches=2@24:00:00 
#SBATCH --nodes=1024
#SBATCH --ntasks-per-node=48 
#SBATCH --cpus-per-task=1 
mpiexec -n $SLURM_NTASKS ./myprog

#... (general part
#SBATCH -o Array_job.%A_%a.out 
#SBATCH -e Array_job.%A_%a.err
#SBATCH --partition=general 
#Number of nodes and MPI tasks per node:
#SBATCH --nodes=10
#SBATCH --ntasks=480 
#SBATCH --cpus-per-task=1  
#SBATCH --array=1-10
mpiexec -n  $SLURM_NTASKS ./myprog  \
         <in.$SLURM_ARRAY_TASK_ID 

#... (general part)
#SBATCH --ear=off
... 

#!/bin/bash 
# Chain of batch jobs with dependencies 
NR_OF_JOBS=6 
JOB_SCRIPT=./my_batch_script 
echo "Submitting chain of $NR_OF_JOBS jobs for batch script $JOB_SCRIPT" 
#submit and get JOBID
JOBID=$(  sbatch ${JOB_SCRIPT} 2>&1 \
       | awk '{print $(NF)}'  ) 
I=1 
while [ $I -lt $NR_OF_JOBS ] 
do  
   JOBID=$( sbatch --dependency=afterok:$JOBID \
         $JOB_SCRIPT 2>&1 \
       | awk '{print $(NF)}'  )  
   I=$(( $I+1 ))
done 

Input Environment Variables

Upon startup, sbatch will read and handle the options set in the following environment variables.  Note that environment variables will override any options set in a batch script, and command line options will override any environment variables. Some which may be used by you in $HOME/.profile:

Output Environment Variables

The Slurm controller will sets the variables in the environment of the batch script

if [ $SLURM_PROCID ]
then
    ./master
else
    ./slave
fi

File Patterns

sbatch allows for a filename pattern to contain one or more replacement symbols, which are a percent sign "%" followed by a letter 

Example:   #SBATCH -o ./%x.%j.out

Useful commands

Show the estimated start time of a job: sqeueue --start [-u <userID>]

Guidelines for resource selection

Processing Mode

• Jobs that  only use one or at most a few hardware cores perform serial processing and are not supported on SuperMUC-NG. Use the SuperMUC-Cloud for such purposes.
• Bunches of multiple independent tasks can be bundled into one job, using one or more nodes.

Run time limits 

• Please note that all job classes impose a maximum run time limit. It can be adjusted downward for any individual job. Since the scheduler uses a backfill algorithm, the better you specify a realistic runtime limit, the better throughput of your job may be achieved.

Islands/Switches

• When a tree topology is used, this defines the maximum count of switches desired for the job allocation and optionally the maximum time to wait for that number of switches. If Slurm finds an allocation containing more switches than the count specified, the job remains pending until it either finds an allocation with desired (lower) switch count or the time limit expires. It there is no switch count limit, there is no delay in starting the job. This trades off better performance vs. shorter wait time in the queue

Memory Requirements

• The total memory available in user space for the set of nodes requested by the job must not be exceeded.
• The memory used on each individual node must not be exceeded by all tasks run on that node.
• Applications exist for which the memory usage is unsymmetric. In this case it may become necessary to work with a variable number of tasks per node. One relevant scenario is a master-worker scheme where the master may need an order of magnitude more memory and therefore requires a node of its own, while worker nodes can share a node. LRZ provides the "mixed" partition for using thin and fat nodes concurrently

Disk and I/O Requirements

• The disk and I/O requirements are not controlled by the batch scheduling system, but rely on parallel shared file systems, which provide system-global services with respect to bandwidth - this means that the total I/O bandwidth is shared between all users. The consequence is that all I/O may be significantly slowed down if heavily used by multiple users at the same time, or even - for large scale parallel jobs - by a single user. At present, LRZ can not make any Quality of Service assurance for I/O bandwidth.
• The appropriate usage of the parallel file systems is essential.
• Please consult File Systems of SuperMUC-NG for more detailed technical information.

Licences

• Some jobs may make use of licensed software, either from the LRZ software application stack, or of software installed in the user's HOME directory. In many cases, the software needs to access a license server because there exist limits on how many instances of the software may run and who may access it at all.
• There is no connection from SuperMUC-NG to the outside. Check with LRZ if you are in need of such licenses.
• LRZ is currently not able to manage license contingents. The reason is that a significant additional effort is required, not only with suitable configuration of SLURM, but also with how the license servers are managed. The situation implies that a job will fail if the usage limit of a licensed software is exceeded when the job starts.

Conversion of scripts from LoadLeveler and other Workload Managers table

• see: List of the most common command, environment variables, and job specification options used by the major workload management systems

Specific Topics (jobfarming, constraints)

• Job farming with SLURM
• List of SLURM Constraints and its Usage

SLURM Documentation

• SLURM Workload Manager at LRZ
• Command/option Summary (two pages) 
• Documentation for SLURM at SchedMD
• The manual pages slurm(1), sinfo(1), squeue(1), scontrol(1), scancel(1), sview(1)