General

The MATLAB module is not visible and cannot be loaded

Check your module environment by listing currently loaded modules. As part of LRZ software stack MATLAB is provided via Spack. It is mandatory that a Spack module is loaded. If not, check for available Spack modules and load the newest module.
In order to get access to the list of supported MATLAB releases, you have to load the Spack module, for example spack/release/21.1.1.

> module list                       # list loaded modules
> module avail spack                # show all available Spack modules
> module avail spack/release        # show available Spack modules (releases only)
> module switch spack/release/20.1  # load Spack (example)
> module avail matlab               # check for MATLAB

Fall back to old MATLAB installation

You may also load old installations of MATLAB via an old Spack module. This might be helpful for particular use cases. But, this is a fallback solution! Old software stacks are not supported anymore! Some MATLAB features might not work because setup of MATLAB installation does not match updated system configurations, among others.

Starting MATLAB may fail with various errors. Common issues are licensing errors, such as:

License checkout failed.
License Manager Error -83
This version is newer than the version of the license file and/or network license manager on the
server machine. 
Make sure the license manager has been updated to the latest release.

Troubleshoot this issue by visiting: 
https://www.mathworks.com/support/lme/R2022a/83

Diagnostic Information:
Feature: MATLAB 
  1. Please check MATLAB - Getting Started for relevant information on the releases.
  2. If no information on the problem available: Open an incident ticket at the LRZ Servicedesk and follow the path via category "Software problem".

MATLAB PCT jobs usually have trouble to involve all CPU cores, e. g. 28 cores on CoolMUC-2. A typical reason is the lack of available memory. Assuming 56 GB of memory on CoolMUC-2 and 28 tasks defined in the job script, each worker will have a maximum of 2 GB of memory (minus memory overhead used for the worker).

Workaround

As a workaround you may try to reduce the number of workers by doing following adjustments (20 tasks is an example, please try meaningful values for your usecase):

Slurm job script
# This is not a full working example. Only relevant modifications are shown.
...
#SBATCH --nodes=1
#SBATCH --tasks-per-node=20
#SBATCH --cpus-per-task=1
...
matlab -nodisplay -singleCompThread -r "myMatlabFunction"
...
MATLAB script
% This is not a full working example. Only relevant modifications are shown.

% Your code

...

% get number of workers from Slurm environment
nw = str2num(getenv('SLURM_NTASKS_PER_NODE'))

% disable multithreading
% (Probably, next line is deprecated. Please also use MATLAB's commandline argument "-singleCompThread".)
maxNumCompThreads(1);

% parallel pool initialization

% shut down existing parallel pool
if ~isempty(gcp('nocreate'))
    poolobj = gcp('nocreate');
    delete(poolobj);
end

% create a local cluster object
pc = parcluster('local');
% set number of workers
pc.NumWorkers = nw;
% set the JobStorageLocation to SCRATCH (default: HOME -> not recommended)
pc.JobStorageLocation = strcat(getenv('SCRATCH'));
% start the parallel pool
poolobj = parpool(pc, nw);

...

% Your code

It might be necessary to reduce the number of workers significantly, for example down to 50 percent of all available cores, e. g. 14 workers on CoolMUC-2. However, due to a significant amount of unused resources, this results in an inefficient computation. A hybrid solution might improve the job performance. That is, you might try to use both workers and threads in your job, e. g. 14 workers and 2 threads per worker. Many MATLAB-intrinsic functions and libraries support multithreading. Your job might benefit from that. A few modifications of the basic workaround are necessary:

Slurm job script
# This is not a full working example. Only relevant modifications are shown.
...
#SBATCH --nodes=1
#SBATCH --tasks-per-node=14
#SBATCH --cpus-per-task=2
...
matlab -nodisplay -r "myMatlabFunction"
...
MATLAB script
% This is not a full working example. Only relevant modifications are shown.

% Your code

...

% get number of workers and threads from Slurm environment
nw = str2num(getenv('SLURM_NTASKS_PER_NODE'));
nthreads = str2num(getenv('SLURM_CPUS_PER_TASK'));

% set multithreading
% (Probably, next line is deprecated.)
maxNumCompThreads(nthreads);

% parallel pool initialization

% shut down existing parallel pool
if ~isempty(gcp('nocreate'))
    poolobj = gcp('nocreate');
    delete(poolobj);
end

% create a local cluster object
pc = parcluster('local');
% set number of workers
pc.NumWorkers = nw;
% set number of threads per worker
pc.NumThreads = nthreads;
% set the JobStorageLocation to SCRATCH (default: HOME -> not recommended)
pc.JobStorageLocation = strcat(getenv('SCRATCH'));
% start the parallel pool
poolobj = parpool(pc, nw);

...

% Your code

MATLAB Parallel Server

The following bash command performs a MPS license check and can be used in two cases:

  • You may check, whether you are allowed to use the MPS .
  • The output might be helpful in troubleshooting of licensing issues. Please attach it to incident tickets submitted to the support.

If MATLAB prints the list of installed toolboxes then everything is fine. Otherwise, you will receive an error message.

> matlab -dmlworker -nodisplay -r "ver,quit"

For troubleshooting purposes you may also run a license check using the license command (as shown below) within a MATLAB session. If it returns "1" then the MPS license exists.

>> license('checkout','distrib_computing_toolbox')

You tried to submit a MPS job. Do you get an error which looks like the following example?

Error using parallel.Cluster/batch (line 158)
Too many workers requested. The job requires 200, but the cluster with profile 'coolmuc local R2021a' supports a maximum of 448.

The job size is too big. Please reduce the number of workers.

The number of MPS licenses is limited. Slurm will automatically queue MPS jobs unless licenses are available (please consider: our Slurm regulations are applicable and there is no guaranty that jobs start immediatly). You may check the availability of MPS licenses via following bash command on one of the login nodes:

> scontrol show licenses --clusters=cm2 | grep -A1 "MATLAB_Distrib_Comp_Engine"

Do you get an error which looks like the following example?

============================= JOB SUBMISSION INFO =============================
Job will be submitted to Slurm via following command:

sbatch ...
===============================================================================

/dss/dsshome1/lrz/sys/spack/release/21.1.1/opt/x86_64/matlab/R2021a-gcc-cbij4ux/bin/glnxa64/matlab_helper: symbol lookup error: /lrz/sys/tools/intel-mpi-wrappers/lib/sles15wa.so: undefined symbol: dlsym
Error using parallel.Cluster/batch (line 158)
Job submission failed because the plugin function 'communicatingSubmitFcn.m' errored.

Error in job_config (line 63)
job = ch.batch(fhandle, 4, {}, 'Pool', num_worker);

Caused by:
    Error using communicatingSubmitFcn (line 113)
    Submit failed with the following message:

The library libdl.so cannot be found. This error occurs in combination with particular Intel modules. Workaround: Please quit your Matlab session and restart with a modified command:

> LD_PRELOAD=/usr/lib64/libdl.so matlab <matlab-arguments>

Using the default Intel-MPI module (Version 2019), unintended crashes of parallel MATLAB jobs might occur. Please switch to Intel MPI 2018. Example:

> module remove intel-mpi
> module load intel-mpi/2018-intel
> module matlab-mcr/R2021a-generic

Matlab Compiler Runtime

Does your (SuperMUC-NG) job show the following error?

libXt.so.6: cannot open shared object file: No such file or directory

As a workaround you have to modify the LD_LIBRARY_PATH environment variable in your job script in order to link to the missing library.

Example on SuperMUC-NG
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lrz/sys/spack/release/21.1.1/opt/skylake_avx512/libxt/1.1.5-gcc-twemnx6/lib

Do you get the following error?

Error: Could not find version X of the MATLAB Runtime.
Attempting to load libmwmclmcrrt.so.X.
Please install the correct version of the MATLAB Runtime.
Contact your vendor if you do not have an installer for the MATLAB Runtime.

The MATLAB release version used for compilation has to match the MCR version at runtime! The update level may be different. Example:

  • MATLAB version: R2020b (Update 5)
  • MCR version: R2020b (Update 3)
  • No labels