Step | Comment |
---|
1. Create folder structure of example |
---|
mcr_mpmd_example
├── bin
├── data
├── mcr_build.m
├── mcrjob.cmuc2.slurm
├── output
└── src
├── dep1
│ └── matmul_serial.m
├── dep2
│ └── plot_save_matrix.m
└── mcr_run.m
|
-> base path
-> path for executable file
-> path to data files created by application (log files, plots)
-> build script running MATLAB Compiler
-> Slurm script to run MCR job
-> directory for Slurm job output files
-> application sources: m-files
-> dependencies
-> serial matrix-matrix multiplication
-> dependencies
-> plot result
-> run script of application ("main" function)
|
2. Prepare application sources |
---|
function mcr_run(size_A_str, size_B_str)
%===============================================================================
% MATLAB MPMD EXAMPLE: RUNTIME CODE
%
% This function will be compiled by Matlab Compiler. This function
% - summarizes some configuration steps, e. g. defining dependencies
% - calls the Matlab function implemented by the user
%
% This function calls matmul_serial.m (matrix-matrix-multiplication with
% threading support) as dependency 1 and a plotting function as dependency 2.
%
% Each MPI process will run this code by evaluating its own rank.
%===============================================================================
%% =============================================================================
% Each process will use the MPI rank as task identifier.
% Get my MPI rank from the MPI environment.
%% =============================================================================
myrank = str2num(getenv('PMI_RANK'));
%% =============================================================================
% Create a log file for this task to avoid cluttered output of all MPI tasks in
% the Slurm job output file.
%% =============================================================================
fp = fopen(sprintf('data/JOB_%s_TaskID_%d.log', getenv('SLURM_JOB_ID'), myrank), 'w');
%===============================================================================
% All processes say hello.
% Show some information, also useful for troubleshooting
%===============================================================================
fprintf(fp, 'MPI process rank=%d: I run on compute node %s and will handle task %d.\n', ...
myrank, getenv('HOSTNAME'), myrank);
fprintf(fp, 'MATLAB temporary directory = %s\n', tempdir);
fprintf(fp, 'MCR_CACHE_ROOT = %s\n', getenv('MCR_CACHE_ROOT'));
fprintf(fp, 'MCR root dir = %s\n', ctfroot);
%===============================================================================
% Run user code.
% Scale result by taskID to show MPMD effect.
%===============================================================================
% get sizes of input matrices for computation of C = A*B from commandline
% arguments
size_A = str2num(size_A_str);
size_B = str2num(size_B_str);
[C, comptime] = matmul_serial(size_A, size_B);
C = C * myrank;
plot_save_matrix(C, ['data/matmul_result_taskID_' num2str(myrank)]);
fprintf(fp, '(rank=%d) computation time = %.4f s.\n', myrank, comptime);
%% =============================================================================
% close log
%% =============================================================================
fclose(fp);
| - This is the "main" function. It will be compiled by MATLAB Compiler. In this example, mcr_run.m will act as purely serial standalone application.
- It can be adjusted to any usecase.
- Main purposes are:
- organizing the parallel environment,
- calling application code,
- displaying debug information.
|
function [C, comptime] = matmul_serial(size_A, size_B)
%===============================================================================
% MATLAB EXAMPLE: SERIAL HELLO WORLD
% -> matrix-matrix multiplication C = A*B
%
% INPUT
% size_A, size_B ... 2-element row vectors defining sizes of A and B
% OUTPUT
% C ................ result
% comptime ......... computation time (matrix product only)
%===============================================================================
%===============================================================================
% Check input
%===============================================================================
if nargin~=2
error('Invalid number of input arguments!');
end
if size_A(2)~=size_B(1)
error(sprintf('Dimension mismatch of A (%d columns) and B (%d rows)!',...
size_A(2), size_B(1)));
end
%===============================================================================
% Work
%===============================================================================
% Hello message from compute node
fprintf('Hello from MATLAB process PID=%d running on node %s!\n',...
feature('getpid'),...
strtrim(evalc('system(''hostname'');')));
% generate well-defined matrices
NA = prod(size_A);
NB = prod(size_B);
A = reshape( linspace( 1,NA, NA), size_A );
B = reshape( linspace(NB, 1, NB), size_B );
% compute
tic;
C = A*B;
comptime = toc;
fprintf('serial computation of matrix-matrix product:\n');
fprintf('\ttime = %.2f s\n', comptime);
function plot_save_matrix(A, fname)
%===============================================================================
% Plot matrix A and save figure to file fname.
%===============================================================================
fig = figure('visible','off');
imagesc(A);
colorbar;
saveas(gcf, [fname '.png']);
| - dependencies of application mcr_run.m:
- matmul_serial.m: similar to example shown in section Common Batch Jobs
- plot_save_matrix.m: auxiliary plotting function
|
|
---|
function mcr_build(srcpath, mainfunc, threadingmode)
%===============================================================================
% MATLAB MCR EXAMPLE: BUILD CODE
%
% This function calls Matlab Compiler to build an executable from user code.
%
% INPUT
% srcpath ............ base path to sources (m-files)
% mainfunc ........... name of main M-function to be compiled
% -> file name of function only (no path)
% -> without extension ".m"
% -> according m-file must be located in srcpath
% threadingmode ...... set multithreading mode (optional)
% -> 'disableThreading', 'enableThreading' (default)
%===============================================================================
%===============================================================================
% Construct basic compiler command.
%===============================================================================
if nargin < 3 || (nargin==3) && strcmp(lower(threadingmode), 'enablethreading')
cmd = ['mcc' ...
' -m' ...
' -R -nodisplay' ...
' -o ' mainfunc ...
' ' srcpath '/' mainfunc '.m'];
elseif (nargin==3) && strcmp(lower(threadingmode), 'disablethreading')
cmd = ['mcc' ...
' -m' ...
' -R -nodisplay' ...
' -R -singleCompThread' ...
' -o ' mainfunc ...
' ' srcpath '/' mainfunc '.m'];
end
%===============================================================================
% Find dependencies and add them to the command.
% -> all subdirectories in srcpath are assumed to be dependencies
%===============================================================================
fcont = dir(srcpath);
for d = fcont'
if d.isdir && ~strcmp(d.name,'.') && ~strcmp(d.name,'..')
cmd = [cmd ' -a ' fullfile(srcpath, d.name)];
end
end
cmd = strrep(cmd, '//', '/');
%===============================================================================
% Compile!
%===============================================================================
fprintf('Compiling %s via following mcc-command:\n\t%s\n\n', mainfunc, cmd);
eval(cmd);
movefile(mainfunc, ['./bin/' mainfunc '.exe']);
| This is a generic build script which can be used for any use case. It should not be necessary to modify it. |
4. Compile application and set options to disable/enable multithreading in compiled application |
---|
> module switch spack/23.1.0
> module load <MATLAB MODULE> # EDIT HERE (see supported releases)
> matlab -nodisplay -r "mcr_build('./src/', 'mcr_run', '-singleCompThread');quit"
> matlab -nodisplay -r "mcr_build('./src/', 'mcr_run', 'disableThreading');quit"
> matlab -nodisplay -r "mcr_build('./src/', 'mcr_run', 'enableThreading');quit"
> matlab -nodisplay -r "mcr_build('./src/', 'mcr_run');quit"
| The "standalone" application will be created. In this example we will get "mcr_run.exe". This step can be done on any computer providing a Matlab installation and all necessary toolboxes. If you work on the Linux Cluster, please load the desired MATLAB module and then call the command to compile. It is allowed to run this compilation procedure on the login nodes. |
5. (A) Prepare Slurm batch script |
---|
> module switch spack/23.1.0
> module avail matlab-mcr
#!/bin/bash
#SBATCH -o ./%x.%j.%N.out
#SBATCH -e ./%x.%j.%N.err
#SBATCH -D ./
#SBATCH -J mcr_mpmd
#SBATCH --get-user-env
#SBATCH --export=NONE
#SBATCH --clusters=cm2_tiny
#SBATCH --partition=cm2_tiny
#SBATCH --nodes=1
#SBATCH --tasks-per-node=10
#SBATCH --cpus-per-task=1
#SBATCH --time=0:10:00
#===============================================================================
# USER-DEFINED INPUT
#===============================================================================
WORKDIR=$HOME/MATLAB/EXAMPLES/doku.lrz.de/MCR/mcr_mpmd_example # Edit here
APPNAME=./bin/mcr_run.exe # Edit here
MCRMODULE=<MATLAB-MCR MODULE> # Edit here
SIZE_MATRIX_A="[2000 1000]"
SIZE_MATRIX_B="[1000 8000]"
#===============================================================================
# SET MODULES AND ENVIRONMENT VARIABLES
#===============================================================================
module load slurm_setup
# Load Matlab Compiler Runtime.
module rm matlab
module switch spack/23.1.0
module load $MCRMODULE
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/runtime/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/bin/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/os/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/opengl/lib/glnxa64
# Set MCR cache path to SCRATCH. Don't use HOME directory!
export MCR_CACHE_ROOT=${SCRATCH}/MCR_MPMD_JOBID${SLURM_JOB_ID}
# Set general temp. path (some MATLAB releases want to have TMP).
export TMP=$SCRATCH
#===============================================================================
# RUN APPLICATION
# -> MPI call needs to be configured, depending on the APP (Matlab script or
# function? Does it need arguments?)
#===============================================================================
cd $WORKDIR
mpiexec $APPNAME "${SIZE_MATRIX_A}" "${SIZE_MATRIX_B}" # Edit here
| First panel: - show available MCR modules on LRZ systems
Second panel: - Job script to run application on CoolMUC-2
|
5. (B) Adjust Slurm batch script on SuperMUC-NG
|
#!/bin/bash
#SBATCH -o ./%x.%j.%N.out
#SBATCH -e ./%x.%j.%N.err
#SBATCH -D ./
#SBATCH -J mcr_mpmd
#SBATCH --get-user-env
#SBATCH --export=NONE
#SBATCH --partition=test
#SBATCH --account=MY_PROJECT_ID # Edit here
#SBATCH --ear=off
#SBATCH --nodes=1
#SBATCH --tasks-per-node=10
#SBATCH --cpus-per-task=1
#SBATCH --time=0:10:00
#===============================================================================
# USER-DEFINED INPUT
#===============================================================================
WORKDIR=$HOME/MATLAB/EXAMPLES/doku.lrz.de/MCR/mcr_mpmd_example # Edit here
APPNAME=./bin/mcr_run.exe # Edit here
MCRMODULE=<MATLAB-MCR MODULE> # Edit here
SIZE_MATRIX_A="[2000 1000]"
SIZE_MATRIX_B="[1000 8000]"
#===============================================================================
# SET MODULES AND ENVIRONMENT VARIABLES
#===============================================================================
module load slurm_setup
# Load Matlab Compiler Runtime.
module rm matlab
module switch spack/23.1.0
module load $MCRMODULE
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/runtime/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/bin/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/os/glnxa64
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MCRROOT}/sys/opengl/lib/glnxa64
# Workaround on SuperMUC-NG to link to missing libraries
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lrz/sys/spack/release/23.1.0/opt/skylake_avx512/libxt/1.1.5-gcc-dzla4qt/lib
# Set MCR cache path to SCRATCH. Don't use HOME directory!
export MCR_CACHE_ROOT=${SCRATCH}/MCR_MPMD_JOBID${SLURM_JOB_ID}
# Set general temp. path (some MATLAB releases want to have TMP).
export TMP=$SCRATCH
#===============================================================================
# RUN APPLICATION
# -> MPI call needs to be configured, depending on the APP (Matlab script or
# function? Does it need arguments?)
#===============================================================================
cd $WORKDIR
mpiexec $APPNAME "${SIZE_MATRIX_A}" "${SIZE_MATRIX_B}" # Edit here
| |
6. Run job |
For example:
> sbatch mcrjob.supermuc-ng.slurm
|
|