General

The MATLAB module is not visible and cannot be loaded

Check your module environment by listing currently loaded modules. As part of LRZ software stack MATLAB is provided via Spack. It is mandatory that a Spack module is loaded. If not, check for available Spack modules and load the newest module.
In order to get access to the list of supported MATLAB releases, you have to load the Spack module, for example spack/release/21.1.1.

> module list                       # list loaded modules
> module avail spack                # show all available Spack modules
> module avail spack/release        # show available Spack modules (releases only)
> module switch spack/release/20.1  # load Spack (example)
> module avail matlab               # check for MATLAB

Fall back to old MATLAB installation

You may also load old installations of MATLAB via an old Spack module. This might be helpful for particular use cases. But, this is a fallback solution! Old software stacks are not supported anymore! Some MATLAB features might not work because setup of MATLAB installation does not match updated system configurations, among others.

Starting MATLAB may fail with various errors. Common issues are licensing errors, such as:

License checkout failed.
License Manager Error -83
This version is newer than the version of the license file and/or network license manager on the
server machine. 
Make sure the license manager has been updated to the latest release.

Troubleshoot this issue by visiting: 
https://www.mathworks.com/support/lme/R2022a/83

Diagnostic Information:
Feature: MATLAB 
  1. Please check MATLAB - Getting Started for relevant information on the releases.
  2. If no information on the problem available: Open an incident ticket at the LRZ Servicedesk and follow the path via category "Software problem".

MATLAB PCT jobs usually have trouble to involve all CPU cores, e. g. 28 cores on CoolMUC-2. A typical reason is the lack of available memory. Assuming 56 GB of memory on CoolMUC-2 and 28 tasks defined in the job script, each worker will have a maximum of 2 GB of memory (minus memory overhead used for the worker).

Workaround

As a workaround you may try to reduce the number of workers by doing following adjustments (20 tasks is an example, please try meaningful values for your usecase):

Slurm job script
# This is not a full working example. Only relevant modifications are shown.
...
#SBATCH --nodes=1
#SBATCH --tasks-per-node=20
#SBATCH --cpus-per-task=1
...
matlab -nodisplay -singleCompThread -r "myMatlabFunction"
...
MATLAB script
% This is not a full working example. Only relevant modifications are shown.

% Your code

...

% get number of workers from Slurm environment
nw = str2num(getenv('SLURM_NTASKS_PER_NODE'))

% disable multithreading
% (Probably, next line is deprecated. Please also use MATLAB's commandline argument "-singleCompThread".)
maxNumCompThreads(1);

% parallel pool initialization

% shut down existing parallel pool
if ~isempty(gcp('nocreate'))
    poolobj = gcp('nocreate');
    delete(poolobj);
end

% create a local cluster object
pc = parcluster('local');
% set number of workers
pc.NumWorkers = nw;
% set the JobStorageLocation to SCRATCH (default: HOME -> not recommended)
pc.JobStorageLocation = strcat(getenv('SCRATCH'));
% start the parallel pool
poolobj = parpool(pc, nw);

...

% Your code

It might be necessary to reduce the number of workers significantly, for example down to 50 percent of all available cores, e. g. 14 workers on CoolMUC-2. However, due to a significant amount of unused resources, this results in an inefficient computation. A hybrid solution might improve the job performance. That is, you might try to use both workers and threads in your job, e. g. 14 workers and 2 threads per worker. Many MATLAB-intrinsic functions and libraries support multithreading. Your job might benefit from that. A few modifications of the basic workaround are necessary:

Slurm job script
# This is not a full working example. Only relevant modifications are shown.
...
#SBATCH --nodes=1
#SBATCH --tasks-per-node=14
#SBATCH --cpus-per-task=2
...
matlab -nodisplay -r "myMatlabFunction"
...
MATLAB script
% This is not a full working example. Only relevant modifications are shown.

% Your code

...

% get number of workers and threads from Slurm environment
nw = str2num(getenv('SLURM_NTASKS_PER_NODE'));
nthreads = str2num(getenv('SLURM_CPUS_PER_TASK'));

% set multithreading
% (Probably, next line is deprecated.)
maxNumCompThreads(nthreads);

% parallel pool initialization

% shut down existing parallel pool
if ~isempty(gcp('nocreate'))
    poolobj = gcp('nocreate');
    delete(poolobj);
end

% create a local cluster object
pc = parcluster('local');
% set number of workers
pc.NumWorkers = nw;
% set number of threads per worker
pc.NumThreads = nthreads;
% set the JobStorageLocation to SCRATCH (default: HOME -> not recommended)
pc.JobStorageLocation = strcat(getenv('SCRATCH'));
% start the parallel pool
poolobj = parpool(pc, nw);

...

% Your code

Problem applicability

Your job uses Intel-MPI modules and runs both MATLAB and OpenMP program(s).

Problem description

The performance of OpenMP programs may be affected by the environment setting KMP_AFFINITY. In order to obtain reasonable performance on the Linux Cluster, the default value is set by the Intel-MPI module:

setenv		KMP_AFFINITY granularity=thread,compact,1,0

However, due to unknown issues between MATLAB and Intel MPI, this setting may decrease the performance of MATLAB PCT. The MATLAB module overwrites KMP_AFFINITY:

setenv      KMP_AFFINITY granularity=thread,none

Workaround

In complex workflows, we recommend to reset the environment variable in your Slurm job script as needed:

# This is not a full working example. Only relevant modifications are shown.
...
export KMP_AFFINITY=granularity=thread,compact,1,0
# now run my OpenMP program
...
export KMP_AFFINITY=granularity=thread,none
# now run MATLAB
...

As SOSTOOLS is not part of the MATLAB release and users do not have permissions to modify software installations provided by LRZ, you may install it into the user space, e.g. your HOME directory. This manual is based on following sources:

Recommended instructions for installation on Linux Cluster:

StepInstructionComment
1nothing to do

MATLAB requirements

  • (tick) MATLAB releases on LRZ Linux Cluster
  • (tick) Current version of MATLAB Symbolic Math Toolbox
2
Terminal (bash)
cm2login1:~> cd $HOME
cm2login1:~> mkdir software

Login to Linux Cluster and prepare installation of third-party software

For third-party software, we will create a directory called "software" in our HOME directory.

3
Terminal (bash)
cm2login1:~> cd software
cm2login1:~/software> git clone https://github.com/sqlp/sedumi.git
cm2login1:~/software> cd sedumi

# Load and run MATLAB: exemplary choice, you may choose another MATLAB release
cm2login1:~/software/sedumi> module load matlab/R2022a-generic
cm2login1:~/software/sedumi> matlab -singleCompThread -nodisplay -r "install_sedumi; quit"

Download and install SDP solver into software directory

As described in SOSTOOLS documentation, you need to install a SDP solver. The MATLAB installation does not provide such an solver. The SOSTOOLS documentation provides a list of SDP packages. Exemplary, we use SeDuMi. However, you also try the other packages. We recommend to test different packages. The choice might have significant impact on the job performance.

You need to load a MATLAB module and run MATLAB in order to install SeDuMi.

4
Terminal (bash)
cm2login1:~/software> git clone https://github.com/oxfordcontrol/SOSTOOLS.git

Download SOSTOOLS into software directory


5
MATLAB script
addpath(genpath(fullfile(getenv('HOME'),'software/sedumi/')));
addpath(genpath(fullfile(getenv('HOME'),'software/SOSTOOLS/')));

Add SDP solver and SOSTOOLS to MATLAB's search path

This should be done before your application scripts run any function/script from the third-party software directory. addpath(genpath(...)) will add all subdirectories.

Alternative:

You may also add the commands to a dedicated MATLAB script (e.g. "startup.m"), which is part of MATLAB's search path. That applies to the current working directory.

Example: Put "startup.m" into your HOME directory, change directory to your HOME and run MATLAB there. MATLAB will automatically load "startup.m".

MATLAB Parallel Server

The following bash command performs a MPS license check and can be used in two cases:

  • You may check, whether you are allowed to use the MPS .
  • The output might be helpful in troubleshooting of licensing issues. Please attach it to incident tickets submitted to the support.

If MATLAB prints the list of installed toolboxes then everything is fine. Otherwise, you will receive an error message.

> matlab -dmlworker -nodisplay -r "ver,quit"

For troubleshooting purposes you may also run a license check using the license command (as shown below) within a MATLAB session. If it returns "1" then the MPS license exists.

>> license('checkout','distrib_computing_toolbox')

You tried to submit a MPS job. Do you get an error which looks like the following example?

Error using parallel.Cluster/batch (line 158)
Too many workers requested. The job requires 200, but the cluster with profile 'coolmuc local R2021a' supports a maximum of 448.

The job size is too big. Please reduce the number of workers.

The number of MPS licenses is limited. Slurm will automatically queue MPS jobs unless licenses are available (please consider: our Slurm regulations are applicable and there is no guaranty that jobs start immediatly). You may check the availability of MPS licenses via following bash command on one of the login nodes:

> scontrol show licenses --clusters=cm2 | grep -A1 "MATLAB_Distrib_Comp_Engine"

Do you get an error which looks like the following example?

============================= JOB SUBMISSION INFO =============================
Job will be submitted to Slurm via following command:

sbatch ...
===============================================================================

/dss/dsshome1/lrz/sys/spack/release/21.1.1/opt/x86_64/matlab/R2021a-gcc-cbij4ux/bin/glnxa64/matlab_helper: symbol lookup error: /lrz/sys/tools/intel-mpi-wrappers/lib/sles15wa.so: undefined symbol: dlsym
Error using parallel.Cluster/batch (line 158)
Job submission failed because the plugin function 'communicatingSubmitFcn.m' errored.

Error in job_config (line 63)
job = ch.batch(fhandle, 4, {}, 'Pool', num_worker);

Caused by:
    Error using communicatingSubmitFcn (line 113)
    Submit failed with the following message:

The library libdl.so cannot be found. This error occurs in combination with particular Intel modules. Workaround: Please quit your Matlab session and restart with a modified command:

> LD_PRELOAD=/usr/lib64/libdl.so matlab <matlab-arguments>

Using the default Intel-MPI module (Version 2019), unintended crashes of parallel MATLAB jobs might occur. Please switch to Intel MPI 2018. Example:

> module remove intel-mpi
> module load intel-mpi/2018-intel
> module matlab-mcr/R2021a-generic

Matlab Compiler Runtime

Does your (SuperMUC-NG) job show the following error?

libXt.so.6: cannot open shared object file: No such file or directory

As a workaround you have to modify the LD_LIBRARY_PATH environment variable in your job script in order to link to the missing library.

Example on SuperMUC-NG
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lrz/sys/spack/release/21.1.1/opt/skylake_avx512/libxt/1.1.5-gcc-twemnx6/lib

Do you get the following error?

Error: Could not find version X of the MATLAB Runtime.
Attempting to load libmwmclmcrrt.so.X.
Please install the correct version of the MATLAB Runtime.
Contact your vendor if you do not have an installer for the MATLAB Runtime.

The MATLAB release version used for compilation has to match the MCR version at runtime! The update level may be different. Example:

  • MATLAB version on development system: R2020b (Update 5)
  • MCR version on target system: R2020b (Update 3)

That is, by switching to a new MCR version on the target system, you have to recompile your application on the development system using the appropriate MATLAB version.

Please also refer to the example given at MATLAB Compiler Runtime (MCR) and Job Farming.

  • No labels