Job Processing on the Linux-Cluster

Page Content

What you should know at the beginning

All programs in the parallel or serial segments of the cluster must be started up using either

  • a SLURM batch script or
  • an interactive SLURM shell.

In order to access the SLURM infrastructure described here, please first log in to a login node of the cluster as described in Access and Login to the Linux-Cluster.

This document and its subdocuments provide information on:

Choose the compute resources of the Linux Cluster with great care! In particular, please be aware that misuse (see below) of the resources described here can result in the invalidation of the violating account!

Please respect our Linux Cluster policies and rules!

Do you need help or do you fear misusing the HPC resources?

Just get in touch with us for further consulting via Linux Cluster request at LRZ Servicedesk or via HPC Lounge.

Step 1: Get the resource which fits my needs aka "jobs that the Slurm scheduler likes"

In order to allocate the optimal resource for your job, you may consider different job types, e.g.:

  • standard distributed memory parallel job, e.g. using MPI
  • shared-memory parallel job on a single node, e.g. using OpenMP
  • single-core (serial) job
  • single-node parallel job requiring a fraction of both the available CPU cores and memory
  • single-node parallel job requiring a fraction of the available CPU cores but needing almost all memory of a regular compute node on CoolMUC-4
  • large-memory job

The following decision matrix provides assistance in making the right choice.

Decision matrix to select the appropriate partition for
production jobs (interactive tests)
How many CPU resources
does my job need?		How much memory does my job need?
up to 488 GB
per node		489 - 1000 GB
per node		up to 6000 GB
per node
1 - 10 CPU cores

serial_std
serial_long
(cm4_inter)

serial_std
serial_long
(cm4_inter, teramem_inter)

teramem_inter
(teramem_inter)

more than 8 CPU cores
and max. 1 compute node

cm4_tiny
(cm4_inter)
teramem_interteramem_inter

2 - 4 compute nodes

cm4_std
(cm4_inter)

 

 
more than 4 compute nodes

CoolMUC-4 cannot satisfy this requirement!
Please check whether an application for a (test) project on SuperMUC-NG is a suitable alternative.

Step 2: Based on my choice, what job specifications do I have to set?

In the following, we list the appropriate cluster- and partition-specific Slurm settings which need to be specified in your job. For full job script examples, please consult Running parallel jobs on the Linux Cluster or Running serial jobs on the Linux Cluster.

Cluster "cm4"

Partition name

Slurm job settings

cm4_std
--clusters=cm4
--partition=cm4_std
--qos=cm4_std
cm4_tiny
--clusters=cm4
--partition=cm4_tiny
--qos=cm4_tiny

Cluster "serial"

Partition name

Slurm job settings

serial_std
--clusters=serial
--partition=serial_std
--mem=<memory_per_node_GB>G
# e.g.: --mem=100G
serial_long
--clusters=serial
--partition=serial_long
--mem=<memory_per_node_GB>G
# e.g.: --mem=100G

Cluster "inter"

Partition name

Slurm job settings

cm4_inter
--clusters=inter
--partition=cm4_inter
teramem_inter
--clusters=inter
--partition=teramem_inter
--mem=<memory_per_node_GB>G
# e.g.: --mem=100G

Step 3: Check further specifications and limits of clusters and partitions

Cluster specificationsLimitsNode usage
Slurm
cluster
segment
Slurm
partition

Compute
nodes
in partition

CPU cores
per node

GPUs
per node

Node range
per job
min - max

Minimum
CPU limit
(physical cores)

Maximum
CPU limit
(physical cores)

Maximum
job runtime
(hours)

Maximum
running (submitted)
jobs per user

Memory limit

Cluster system CoolMUC-4: Sapphire Rapids (Intel(R) Xeon(R) Platinum 8480+) nodes
cm4cm4_std

100

(overlapping
partitions)

112 (physical)

224 (logical)

--

2 - 4

112 per job

448 per job

242 (25)

488 GiB per node

exclusive
cm4_tiny1 - 1

9 per job

112 per job

244 (25)

default: 2.1 GiB per logical CPU core

overall limit: 488 GiB per node

shared
inter
cm4_inter6--1 - 4

1 per job

112 per job

8

1 (2)

Cluster system CoolMUC-4: Ice Lake (Intel(R) Xeon(R) Platinum 8380) nodes
serialserial_std10

80 (physical)

160 (logical)

--1 - 1
1

20 per job

96 per user
(in sum over all jobs, see remarks)

24

96 (200)

default: 6.2 GiB per logical CPU core

overall limit: 1000 GiB per node

shared
serial_long

2

--

168

96 (200)

Cluster system Teramem: single-node shared-memory system (Intel Xeon Platinum 8360HL), 6 TB memory

interteramem_inter1

96 (physical)

192 (logical)

--

1 - 1

1 per job96 per job240

1 (1)

approx. 60 GiB
per physical core
available

shared

Remarks on CPU limit

CPU limit refers to the number of available hardware (physical) cores. When setting up your Slurm jobs, please consider the following characteristics and restrictions:

  • Jobs may incorporate all logical cores, i.e., the number of physical cores times number of threads that can simultaneously run on one core via hyperthreading. In case of CoolMUC-4, this is twice the number of physical cores. Shared-memory jobs (e.g. using OpenMP) or hybrid jobs (e.g. using MPI + OpenMP) may benefit from that. Please also refer to our Slurm job examples.
  • The total number of requested cores, i.e. the product of "tasks per node" (e.g. MPI processes) and "CPUs per tasks" (e.g. OpenMP threads) ...
    • Must not exceed the maximum number of cores which is available or allowed to be used per compute node on a particular partition!
    • Must not be smaller than the minimum number of cores allowed to be used on a compute node!
  • Important: Due to limited hardware resources in the serial cluster, the following restrictions apply:
    • Only a subset of CPU cores per node can be used per job (see table)! 
    • The total number of requested CPU cores by a user on the entire serial cluster, which are in use at the same time in running jobs, is also limited (see table)! You may distribute the CPU cores to different jobs.

The terms "tasks per node" and "CPUs per tasks" refer to the according Slurm specifications "--tasks-per-node" and "--cpus-per-task". Please refer to our Slurm documentation and CoolMUC-4 job script examples.

Remarks on Memory limit

Keep in mind that the default memory per job on shared partitions (serial cluster, cm4_tiny, cm4_inter) scales with the number of allocated CPU cores. In other words, by default, your job will only get a portion of the memory, according to the number of requested CPU cores!
If more memory is required, this has to be defined in the job via the "–mem" option.

General remarks

Please note:

  • Nodes on partitions cm4_tiny, cm4_inter, serial_std, serial_long and teramem_inter are used as shared resources, i.e., multiple jobs/users share those nodes. cm4_std only provides exclusive nodes for jobs.
  • Partitions cm4_std, cm4_tiny, serial_std and serial_long are only intended for batch jobs (via sbatch command, see below)!
  • The partition cm4_inter is intended for interactive jobs (via salloc command, see below). Due to the short job runtime, this partition is suitable for small test jobs.
  • Both batch jobs and interactive jobs can be run on the partition teramem_inter.

Common Slurm commands on the Linux Cluster for job submission and job management

Once submitted, a Slurm job will be queued for some time, depending on how many jobs are presently submitted and how many resources are available. Eventually, more or less after previously submitted jobs have completed, the job will be started on one or more of the systems determined by its resource requirements. Slurm provides several commands to check the status of waiting or running jobs, to inspect or even modify (modification only to limited extent) waiting/running jobs, to obtain information of finished jobs and to delete waiting/running jobs. Click here to get an overview of commonly used Slurm commands on the Linux Cluster.