Overview

The interactive cluster segment of the Linux Cluster provides two partitions: cm4_inter and teramem_inter. Please consider different limits on both partitions. cm4_inter only allows very short jobs, whereas on teramem_inter job runtimes of multiple days are possible. As the resources in the interactive cluster segment are limited, you may consider the following typical use cases.

• Short application runs which do not require a huge amount of memory (less than 0.5 TB per node) may use cm4_inter.
• cm4_inter is also suitable for testing of parallel programs, for example OpenMP jobs, MPI jobs or jobs using hybrid parallelization. Jobs may request multiple compute nodes.
• The Teramem – a single node with approx. 6 TB of memory – is the only system at LRZ which can satisfy memory requirements beyond 1 TB on a single node. Before running jobs on the partition teramem_inter, please consider:
  
  • Typical Teramem jobs may use a relatively small amount of CPU resources in shared-memory applications.
  • Typical Teramem jobs may run applications without any distributed-memory parallelization.
  • It is discouraged to run MPI applications on the Teramem! If possible and if required, run the job on the partition cm4_std. If this is insufficient please consult the decision matrix.

This page briefly describes how to use the different interactive partitions and shows examples of typical use cases.

Commands

This page shows interactive jobs with exemplary parametrization. Check Job Processing on the Linux-Cluster for details on the interactive cluster. In order to start an interactive job, you need to log in to the Linux Cluster. Then, invoke Slurm's salloc command to reserve the needed resource.

salloc settings used on this page

Start interactive jobs on CoolMUC-4 partition cm4_inter

After invoking the salloc command, you may have to wait some time, if all compute resources are occupied by other jobs. Slurm will grant access to the requested resources when available for your job. After resource allocation via salloc the user will be automatically logged in to the compute node! For example, you want to submit a job running for 30 minutes using 20 CPU cores on the partition cm4_inter in the cluster inter:

userid@cm4login2:~> salloc -M inter -p cm4_inter -n 20 -t 00:30:00
salloc: Pending job allocation 303762
salloc: job 303762 queued and waiting for resources
salloc: job 303762 has been allocated resources
salloc: Granted job allocation 303762
userid@cm4r15c06s05:~>

Then, a suitably parameterized call to a parallel tool (usually srun or mpiexec) is used to start up your parallel application MYAPP, using the resources assigned by SLURM. It is usually necessary to load a MPI module first.

Start interactive jobs on Teramem partition teramem_inter

In order to start an interactive Slurm shell on the Teramem, you need to modify the salloc command. The Teramem is a shared system, i.e., multiple jobs/users may share this system. That is, we recommend to specifiy the number of required CPU resources and required memory (e.g. using units "G" for GB or "T" for TB) per job:

userid@cm4login2:~> salloc -M inter -p teramem_inter -n 1 -c 2 --mem=2T -t 00:30:00
salloc: Pending job allocation 303880
salloc: job 303880 queued and waiting for resources
salloc: job 303880 has been allocated resources
salloc: Granted job allocation 303880
userid@teramem2:~>

General notes

• If the run time of an interactive job is not defined, it will run for 15 minutes by default!
• Once the allocation expires, the program will be signalled and killed. The interactive job is no longer available for starting further programs!
• If neither number of tasks per job nor number of tasks per node is specified, 2 tasks (threads) will be assigned to the job by default.
• Modules are not auto-loaded by default (e.g. for libraries or software packages)! Please load all required modules explicitly. Modules might be exported from the login shell on the login node. Please check!

Examples

Run a pure shared-memory job via OpenMP with 20 threads on the partition cm4_inter

userid@loginnode:~> salloc -M inter -p cm4_inter -n 1 -c 20 -t 00:30:00
userid@computenode:~> export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
userid@computenode:~> srun ./my_program.exe

Run a parallel job via MPI on a single compute node using 20 parallel tasks on the partition cm4_inter

userid@loginnode:~> salloc -M inter -p cm4_inter -n 20 -c 1 -t 00:30:00
userid@computenode:~> module load intel-mpi
userid@computenode:~> mpiexec ./my_mpi_program.exe

Run a hybrid job on two compute nodes using 2 MPI tasks in total and 112 OpenMP threads per task on partition cm4_inter

userid@loginnode:~> salloc -M inter -p cm4_inter -N 2 --ntasks-per-node=1 -c 112 -t 00:30:00
userid@computenode:~> module load intel-mpi
userid@computenode:~> export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
userid@computenode:~> mpiexec -np ./my_hybrid_program.exe

Run a shared-memory job using 4 OpenMP threads and requesting 2000 GB of main memory per job on partition teramem_inter

userid@loginnode:~> salloc -M inter -p teramem_inter -n 1 -c 4 --mem=2000G -t 00:30:00
userid@computenode:~> export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
userid@computenode:~> srun ./my_program.exe

SLURM_CPUS_PER_TASK