Page Content


Introduction

The purpose of the MATLAB Compiler Runtime is to run MATLAB standalone applications on systems without a MATLAB installation. The standalone application is compiled by the MATLAB Compiler which requires a licensed version of MATLAB. In contrast, the standalone application should be executable on any system providing a MCR installation, and without the need of a MATLAB license. On LRZ systems we provide all MCR versions which correspond to the installed MATLAB versions.

MCR-based standalones may run embarrassingly parallel. So-called MPMD (Multiple-Program Multiple-Data) or job-farming techniques are used to run many processes of a program simultaneously and independently, that is, without the need of communication. Typical applications are:

  • parameter studies,
  • independent processing of large datasets which are split into subsets.

Example: Basic Parallel MPMD Application with MATLAB MCR

Prerequisites

  1. The job must be parallelizable. That is, the work can be divided into independent subtasks (e. g. embarrassing parallelization).
  2. The compilation of the application's MATLAB script(s) must be feasible. That is, some MATLAB components may not be compiled or my not be available in the MCR.

Now what? A brief summary

  1. Your MATLAB script needs to be serial with respect to PCT parallelization. If necessary, convert your MATLAB script, e. g. by removing parfor loops. The work of the parfor loops will be done by the MPMD concept. MATLAB's intrinsic multithreading is still usable.
  2. Create a standalone application by compiling your script(s) within MATLAB.
  3. The execution of the standalone application will require the MATLAB Compiler Runtime (MCR).
  4. In your Slurm job script, run the standalone application via MPI. MPI will distribute the application to all resources you have allocated in the job. That is, multiple instances of your application will run simultaneously and independently.
  5. How does each instance of my application know, what task to do?
    MPI assigns an ID to each instance (environment variable PMI_RANK), which can be evaluated within the job script or even the MATLAB script.

General Workflow

The following table depicts the procedure from creating the standalone application to the submission of a Slurm job.

StepComment

1. Create folder structure of example

mcr_mpmd_example
├── bin
├── mcr_build.m
├── mcrjob.cmuc2.slurm
└── src
    ├── dep1
    │   └── matmul_serial.m
    ├── dep2
    │   └── plot_save_matrix.m
    └── mcr_run.m
-> base path
-> path for executable file
-> build script running MATLAB Compiler
-> Slurm script to run MCR job
-> application sources: m-files
-> dependencies
-> serial matrix-matrix multiplication
-> dependencies
-> plot result
-> run script of application ("main" function)
2. Prepare application sources
mcr_run.m 
function mcr_run(size_A_str, size_B_str)
 
%===============================================================================
% MATLAB MPMD EXAMPLE: RUNTIME CODE
%
% This function will be compiled by Matlab Compiler. This function
% - summarizes some configuration steps, e. g. defining dependencies
% - calls the Matlab function implemented by the user
%
% This function calls matmul_serial.m (matrix-matrix-multiplication with
% threading support) as dependency 1 and a plotting function as dependency 2.
%
% Each MPI process will run this code by evaluating its own rank.
%===============================================================================


%% =============================================================================
% Each process will use the MPI rank as task identifier.
% Get my MPI rank from the MPI environment.
%% =============================================================================
myrank = str2num(getenv('PMI_RANK'));


%% =============================================================================
% Create a log file for this task to avoid cluttered output of all MPI tasks in
% the Slurm job output file.
%% =============================================================================
fp = fopen(sprintf('JOB_%s_TaskID_%d.log', getenv('SLURM_JOB_ID'), myrank), 'w');


%===============================================================================
% All processes say hello.
% Show some information, also useful for troubleshooting
%===============================================================================
fprintf(fp, 'MPI process rank=%d: I run on compute node %s and will handle task %d.\n', ...
            myrank, getenv('HOSTNAME'), myrank);
fprintf(fp, 'MATLAB temporary directory = %s\n', tempdir);
fprintf(fp, 'MCR_CACHE_ROOT             = %s\n', getenv('MCR_CACHE_ROOT'));
fprintf(fp, 'MCR root dir               = %s\n', ctfroot);


%===============================================================================
% Run user code.
% Scale result by taskID to show MPMD effect.
%===============================================================================
% get sizes of input matrices for computation of C = A*B from commandline
% arguments
size_A = str2num(size_A_str);
size_B = str2num(size_B_str);
 
[C, comptime] = matmul_serial(size_A, size_B);
C = C * myrank;
plot_save_matrix(C, ['matmul_result_taskID_' num2str(myrank)]);
fprintf(fp, '(rank=%d) computation time = %.4f s.\n', myrank, comptime);


%% =============================================================================
% close log
%% =============================================================================
fclose(fp);
  • This is the "main" function. It will be compiled by MATLAB Compiler. In this example, mcr_run.m will act as purely serial standalone application.
  • It can be adjusted to any usecase.
  • Main purposes are:
    • organizing the parallel environment,
    • calling application code,
    • displaying debug information.

Within the scope of the job-farming approach, MPI will be used to spawn multiple instances of compiled mcr_run.exe. Therefore, each instance needs to identify its individual sub-task. Consequently, we need one important modification.

The easiest way: Each instance obtains its MPI rank from the MPI environment. The rank will be used to identify sub-task(s) assigned to each instance. The MPI rank is unique. In general, the range is 0, 1, ..., ("number of Slurm tasks" - 1).

matmul_serial.m 
function [C, comptime] = matmul_serial(size_A, size_B)

%===============================================================================
% MATLAB EXAMPLE: SERIAL HELLO WORLD
%                 -> matrix-matrix multiplication C = A*B
%
% INPUT
%   size_A, size_B ... 2-element row vectors defining sizes of A and B
% OUTPUT
%   C ................ result
%   comptime ......... computation time (matrix product only)
%===============================================================================

%===============================================================================
% Check input
%===============================================================================
if nargin~=2
    error('Invalid number of input arguments!');
end
if size_A(2)~=size_B(1)
    error(sprintf('Dimension mismatch of A (%d columns) and B (%d rows)!',...
                  size_A(2), size_B(1)));
end

%===============================================================================
% Work
%===============================================================================
% Hello message from compute node
fprintf('Hello from MATLAB process PID=%d running on node %s!\n',...
        feature('getpid'),...
        strtrim(evalc('system(''hostname'');')));

% generate well-defined matrices
NA = prod(size_A);
NB = prod(size_B);
A = reshape( linspace( 1,NA, NA), size_A );
B = reshape( linspace(NB, 1, NB), size_B );

% compute
tic;
C = A*B;
comptime = toc;
fprintf('serial computation of matrix-matrix product:\n');
fprintf('\ttime = %.2f s\n', comptime);
plot_save_matrix.m 
function plot_save_matrix(A, fname)

%===============================================================================
% Plot matrix A and save figure to file fname.
%===============================================================================

fig = figure('visible','off');
imagesc(A);
colorbar;
saveas(gcf, [fname '.png']);
  • dependencies of application mcr_run.m:
    • matmul_serial.m: similar to example shown in section Common Batch Jobs
    • plot_save_matrix.m: auxiliary plotting function

3. Check build script

mcr_build.m 
function mcr_build(srcpath, mainfunc, threadingopt)

%===============================================================================
% MATLAB MCR EXAMPLE: BUILD CODE
%
% This function calls Matlab Compiler to build an executable from user code.
%
% INPUT
%   srcpath ........ base path to sources (m-files)
%   mainfunc ....... name of main M-function to be compiled
%                    -> file name of function only (no path)
%                    -> without extension ".m"
%                    -> according m-file must be located in srcpath
%   threadingopt ... set multithreading mode
%                    -> disable: '-singleCompThread'
%                    -> enable: '' (leave string empty)
%===============================================================================

%===============================================================================
% Construct basic compiler command.
%===============================================================================
cmd = ['mcc' ...
        ' -m' ...
        ' -R -nodisplay' ...
        ' -R ' threadingopt ...
        ' -o ' mainfunc ...
        ' ' srcpath '/' mainfunc '.m'];

%===============================================================================
% Find dependencies and add them to the command.
% -> all subdirectories in srcpath are assumed to be dependencies
%===============================================================================
fcont = dir(srcpath);
for d = fcont'
    if d.isdir && ~strcmp(d.name,'.') && ~strcmp(d.name,'..')
        cmd = [cmd ' -a ' fullfile(srcpath, d.name)];
    end
end
cmd = strrep(cmd, '//', '/');

%===============================================================================
% Compile!
%===============================================================================
fprintf('Compiling %s via following mcc-command:\n\t%s\n\n', mainfunc, cmd);
eval(cmd);
movefile(mainfunc, ['./bin/' mainfunc '.exe']);
This is a generic build script which can be used for any use case. It should not be necessary to modify it.
4. Compile application
> module load <MATLAB MODULE> # EDIT HERE (see supported releases)
> matlab -nodisplay -r "mcr_build('./src/', 'mcr_run', '-singleCompThread');quit"

The "standalone" application will be created. In this example we will get "mcr_run.exe".

This step can be done on any computer providing a Matlab installation and all necessary toolboxes.

The MATLAB release version used for compilation has to match the MCR version at runtime! The update level may be different.

If you work on the Linux Cluster, please load the desired MATLAB module and then call the command to compile. It is allowed to run this compilation procedure on the login nodes.

5. (A) Prepare Slurm batch script
> module avail matlab-mcr
mcrjob.cmuc2.slurm (Linux Cluster CoolMUC-2) 
#!/bin/bash
#SBATCH -o ./%x.%j.%N.out
#SBATCH -e ./%x.%j.%N.err
#SBATCH -D ./
#SBATCH -J mcr_mpmd
#SBATCH --get-user-env
#SBATCH --export=NONE
#SBATCH --clusters=cm2_tiny
#SBATCH --partition=cm2_tiny
#SBATCH --nodes=1
#SBATCH --tasks-per-node=10
#SBATCH --cpus-per-task=1
#SBATCH --time=0:10:00
  
#===============================================================================
# USER-DEFINED INPUT
#===============================================================================
WORKDIR=$HOME/MATLAB/EXAMPLES/doku.lrz.de/MCR/mcr_mpmd_example  # Edit here
APPNAME=./bin/mcr_run.exe                                       # Edit here
MCRMODULE=<MATLAB-MCR MODULE>                                   # Edit here
  
SIZE_MATRIX_A="[2000 1000]"
SIZE_MATRIX_B="[1000 8000]"
  
#===============================================================================
# SET MODULES AND ENVIRONMENT VARIABLES
#===============================================================================
module load slurm_setup
  
# Load Matlab Compiler Runtime.
module rm matlab
module load $MCRMODULE
  
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/runtime/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/bin/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/os/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/opengl/lib/glnxa64
  
# Set MCR cache path to SCRATCH. Don't use HOME directory!
export MCR_CACHE_ROOT=${SCRATCH}/MCR_MPMD_JOBID${SLURM_JOB_ID}
# Set general temp. path (some MATLAB releases want to have TMP).
export TMP=$SCRATCH
  
#===============================================================================
# RUN APPLICATION
# -> MPI call needs to be configured, depending on the APP (Matlab script or
#    function? Does it need arguments?)
#===============================================================================
cd $WORKDIR
mpiexec $APPNAME "${SIZE_MATRIX_A}" "${SIZE_MATRIX_B}"        # Edit here

First panel:

  • show available MCR modules on LRZ systems

Second panel:

  • Job script to run application on CoolMUC-2

The MATLAB release version used for compilation has to match the MCR version at runtime! The update level may be different.

In the script, you need to load a Matlab MCR module, not Matlab!

5. (B) Adjust Slurm batch script on SuperMUC-NG
mcrjob.supermuc-ng.slurm (SuperMUC-NG) 
#!/bin/bash
#SBATCH -o ./%x.%j.%N.out
#SBATCH -e ./%x.%j.%N.err
#SBATCH -D ./
#SBATCH -J mcr_mpmd
#SBATCH --get-user-env
#SBATCH --export=NONE
#SBATCH --partition=test
#SBATCH --account=MY_PROJECT_ID    # Edit here
#SBATCH --ear=off
#SBATCH --nodes=1
#SBATCH --tasks-per-node=10
#SBATCH --cpus-per-task=1
#SBATCH --time=0:10:00

#===============================================================================
# USER-DEFINED INPUT
#===============================================================================
WORKDIR=$HOME/MATLAB/EXAMPLES/doku.lrz.de/MCR/mcr_mpmd_example  # Edit here
APPNAME=./bin/mcr_run.exe                                       # Edit here
MCRMODULE=<MATLAB-MCR MODULE>                                   # Edit here
  
SIZE_MATRIX_A="[2000 1000]"
SIZE_MATRIX_B="[1000 8000]"
  
#===============================================================================
# SET MODULES AND ENVIRONMENT VARIABLES
#===============================================================================
module load slurm_setup
  
# Load Matlab Compiler Runtime.
module rm matlab
module load $MCRMODULE
  
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/runtime/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/bin/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/os/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/opengl/lib/glnxa64

# Workaround on SuperMUC-NG to link to missing libraries
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lrz/sys/spack/release/21.1.1/opt/skylake_avx512/libxt/1.1.5-gcc-twemnx6/lib

# Set MCR cache path to SCRATCH. Don't use HOME directory!
export MCR_CACHE_ROOT=${SCRATCH}/MCR_MPMD_JOBID${SLURM_JOB_ID}
# Set general temp. path (some MATLAB releases want to have TMP).
export TMP=$SCRATCH
  
#===============================================================================
# RUN APPLICATION
# -> MPI call needs to be configured, depending on the APP (Matlab script or
#    function? Does it need arguments?)
#===============================================================================
cd $WORKDIR
mpiexec $APPNAME "${SIZE_MATRIX_A}" "${SIZE_MATRIX_B}"        # Edit here

Workaround for "missing library" error

Jobs (mainly on SuperMUC-NG) may show the following error:

libXt.so.6: cannot open shared object file: No such file or directory

As a workaround you may modify the LD_LIBRARY_PATH environment variable in your job script.

6. Run job

For example:

> sbatch mcrjob.supermuc-ng.slurm

Example

Construction site