Simcenter StarCCM+ (Fluid Dynamics)

Simcenter StarCCM+ is a general purpose Computational Fluid Dynamics (CFD) code. Simcenter StarCCM+is part of the Siemens PLM software portfolio since April 2016 (formerly owned by cd-adapco also known as Computational Dynamics-Analysis & Design Application Company Ltd). As a general purpose CFD code Simcenter StarCCM+ provides a wide variety of physical models for turbulent flows, acoustics, Eulerian and Lagrangian multiphase flow modeling, radiation, combustion and chemical reactions, heat and mass transfer including CHT (conjugate heat transfer in solid domains). 

Further information about Simcenter StarCCM+, licensing of the Siemens PLM software and related terms of software usage at LRZ, the Siemens PLM mailing list, access to the Siemens PLM software documentation and LRZ user support can be found on the main Siemens PLM documentation page.

Getting Started

Once you are logged into one of the LRZ cluster systems, you can check the availability (i.e. installation) of StarCCM+ software by:

> module avail starccm

Load the prefered StarCCM+ version environment module, e.g.:

> module load starccm/2023.3.1

One can use StarCCM+ in interactive GUI mode for the only purpose of pre- and/or postprocessing on the Login Nodes (Linux: SSH Option "-Y" or X11-Forwarding; Windows: using PuTTY and XMing for X11-forwarding). This interactive usage is mainly intended for making quick simulation setup changes, which require GUI access. And since StarCCM+ is loading the mesh into the login nodes memory, this approach is only applicable to comparable small cases. It is NOT permitted to run computationally intensive StarCCM+ simulation runs or postprocessing sessions with large memory consumption on Login Nodes. Alternatively StarCCM+ can be run, e.g. for memory intensive postprocessing and OpenGL acceleration of the GUI on the Remote Visualization Systems.

The Simcenter StarCCM+ GUI is started by:

> starccm+

The 3d results visualization program StarView+ can be launched by:

      > starview+

Mixed vs. Double Precision Solver of StarCCM+

Siemens PLM is providing installation packages for mixed precision and higher accuracy double precision simulations. The latter comes for the price of approx. 20% higher execution times and approx. twice as large simulation results files. The LRZ module system is providing access to both versions of StarCCM+.

Access to StarCCM+ mixed precision solvers e.g. by:

module load starccm/2023.3.1                  # loading the mixed precision StarCCM+ module

Access to StarCCM+ double precision solvers e.g. by:

module load starccm_dp/2023.3.1               # loading the double precision StarCCM+ module

Simcenter StarCCM+ Parallel Execution (Batch Mode)

All parallel StarCCM+ simulations on LRZ Linux Clusters and SuperMUC-NG are submitted as non-interactive batch jobs to the appropriate scheduling system (SLURM) into the different pre-defined parallel execution queues. Further information about the batch queuing systems and the queue definitions, capabilities and limitations can be found on the documentation pages of the corresponding HPC system (LinuxCluster, SuperMUC-NG)

For job submission to a batch queuing system a corresponding small shell script needs to be provided, which contains:

  • Batch queueing system specific commands for the job resource definition
  • Module command to load the Simcenter StarCCM+ environment module
  • Start command for parallel execution of starccm+ with all appropriate command line parameters, including a controling StarCCM+ java macro.

The intended syntax and available command line options for the invocation of the starcmm+ solver command can be found out by:

> starccm+ -help

The configuration of the parallel cluster partition (list of node names and corresponding number of cores) is provided to the starccm+ command from the batch queuing system (SLURM) by the provision of an automatically generated environment variable $STAR_HOSTLIST, based on the information provided by the cluster user in the job resource definition. The number of StarCCM+ solver processes is submitted to the starccm+ solver command by the SLURM environment variable $SLURM_NTASKS.

Furthermore we recommend to LRZ cluster users to write for longer simulation runs regular backup files, which can be used as the basis for a job restart in case of machine or job failure. A good practice for a 48 hour StarCCM+ simulation (max. time limit) would be to write backup files every 6 or 12 hours. Further information you can find in the StarCCM+ documentation.

CoolMUC-2 : Simcenter StarCCM+ Job Submission on LRZ Linux Clusters running SLES15 using SLURM

StarCCM+ can be provided on CoolMUC-2 (CM2) with support for both the default Intel MPI and OpenMPI on CM2 queues (cm2_tiny, cm2_std) with Infiniband interfaces.

Similar SLURM batch script syntax can be applied for either using Power-on-Demand licensing or the access to a license server which provides floating licenses for StarCCM+.

Using a Power-on-Demand License

In the following an example of a job submission batch script for StarCCM+ on CoolMUC-2 (SLURM queue = cm2_tiny) in the batch queuing system SLURM is provided. The example is formulated for the use of POD (Power-on-Demand) licensing. POD keys can be obtained either through the TUM campus license or directly from Siemens PLM.

#!/bin/bash
#SBATCH -o ./myjob.%j.%N.out
#SBATCH -D ./
#SBATCH -J starccm_cm2_tiny
#SBATCH --clusters=cm2_tiny
#SBATCH --partition=cm2_tiny
#SBATCH --get-user-env
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28
# --- multiples of 28 for cm2 / cm2_tiny clusters ---
#SBATCH --mail-type=NONE
#SBATCH --mail-user=Max.Mustermann@lrz.de
#SBATCH --export=NONE
#SBATCH --time=00:10:00
#
module load slurm_setup

module av starccm
module load starccm/2023.3.1        # <-- mixed precision version 
# module load starccm_dp/2023.3.1   # <-- double precision version
module list

echo "=================== Verification of the StarCCM+ Hostlist =========================="
echo $STAR_HOSTLIST
cat $STAR_HOSTLIST
echo "=================== Verification of the StarCCM+ Hostlist =========================="

# cat /proc/cpuinfo
# Provide your StarCCM+ simulation file name and Java macro name here:
SIM_FILE=Linux_StarCCM_Test.sim
START_MACRO=MeshSimSave.java
#
echo "===========================  Start of StarCCM+  ===================================="
echo "starccm+ -power -podkey <Name_of_your_POD_license_key> -licpath 1999@flex.cd-adapco.com -rsh /usr/bin/ssh -mpi intel -mpiflags \"-bootstrap ssh\" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE"
#
starccm+ -power -podkey <Name_of_your_POD_license_key> -licpath 1999@flex.cd-adapco.com -rsh /usr/bin/ssh -mpi intel -mpiflags "-bootstrap ssh" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE
echo "===========================  End of StarCCM+  ======================================"

Assumed that the above SLURM script has been saved under the filename "starccm_POD_cm2_tiny.sh", the SLURM batch job has to be submitted by issuing the following command on one of the Linux Cluster login nodes:

sbatch starccm_POD_cm2_tiny.sh

Using a LRZ Floating License

In the following an example of a job submission batch script for StarCCM+ on CoolMUC-2 (SLURM queue = cm2_std) in the batch queuing system SLURM is provided. Correspondingly smaller jobs using a smaller number of compute nodes can be submitted to the CM2 cluster queue cm2_tiny by adjusting the provided SLURM script accordingly (change of --cluster and --partition statements and omitting the --qos statement - see example for cm2_tiny queue above). The example is formulated for the use of a StarCCM+ floating license being provided by the LRZ internal license server license1.lrz.de (User-ID authentication for license check-out). Consequently befor the use of StarCCM+ floating licenses the User-ID of the user has to be registered in the LRZ license server (please send a LRZ Service Request).

#!/bin/bash
#SBATCH -o ./myjob.%j.%N.out
#SBATCH -D ./
#SBATCH -J starccm_cm2_std
#SBATCH --clusters=cm2
#SBATCH --partition=cm2_std
#SBATCH --qos=cm2_std
#SBATCH --get-user-env
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=28
# --- multiples of 28 for cm2 ---
#SBATCH --mail-type=NONE
#SBATCH --mail-user=Max.Mustermann@lrz.de
#SBATCH --export=NONE
#SBATCH --time=00:10:00
#
module load slurm_setup

module av starccm
module load starccm/2023.3.1        # <-- mixed predicion version
# module load starccm_dp/2023.3.1   # <-- double precision version
module list

echo "=================== Verification of the StarCCM+ Hostlist =========================="
echo $STAR_HOSTLIST
cat $STAR_HOSTLIST
echo "=================== Verification of the StarCCM+ Hostlist =========================="

# cat /proc/cpuinfo
# Provide your StarCCM+ simulation file name and Java macro name here:
SIM_FILE=Linux_StarCCM_Test.sim
START_MACRO=MeshSimSave.java
#
echo "===========================  Start of StarCCM+  ===================================="
# Syntax of the start of StarCCM+ by using Intel MPI 2018/2019:
echo "starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags \"-bootstrap ssh\" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE"
#
starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags "-bootstrap ssh" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE
#------------------------------------------------------------------------------------------
# Syntax of the start of StarCCM+ by using OpenMPI:
#   Version 2023.3.1: use commandline options "-mpi openmpi -fabric ofi"
# echo "starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi openmpi -fabric ofi -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE"
#
starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi openmpi -fabric ofi -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE
echo "===========================  End of StarCCM+  ======================================"

Assumed that the above SLURM script has been saved under the filename "starccm_floatlic_cm2_tiny.sh", the SLURM batch job has to be submitted by issuing the following command on one of the Linux Cluster login nodes (lxlogin1,2,3,4):

sbatch starccm_floatlic_cm2_std.sh

CoolMUC-3 : Simcenter StarCCM+ Job Submission on LRZ Linux Clusters running SLES12 using SLURM

Using a Power-on-Demand License

In the following an example of a job submission batch script for StarCCM+ on CoolMUC-3 (SLURM queue = mpp3) in the batch queuing system SLURM is provided. The example is formulated for the use of POD (Power-on-Demand) licensing. POD keys can be obtained either through the TUM campus license or directly from Siemens PLM.

#!/bin/bash
#SBATCH -o ./myjob.%j.%N.out
#SBATCH -D ./
#SBATCH -J starccm_mpp3_slurm
#SBATCH --clusters=mpp3
#SBATCH --get-user-env
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
# --- multiples of 64 for mpp3 ---
#SBATCH --mail-type=NONE
#SBATCH --mail-user=Max.Mustermann@lrz.de
#SBATCH --export=NONE
#SBATCH --time=00:10:00
#
module load slurm_setup

module av starccm
module load starccm/2023.1.1        # <-- mixed precision version
# module load starccm_dp/2023.1.1   # <-- double precision version 
module list

echo "=================== Verification of the StarCCM+ Hostlist =========================="
echo $STAR_HOSTLIST
cat $STAR_HOSTLIST
echo "=================== Verification of the StarCCM+ Hostlist =========================="

# cat /proc/cpuinfo
# Provide your StarCCM+ simulation file name and Java macro name here:
SIM_FILE=Linux_StarCCM_Test.sim
START_MACRO=MeshSimSave.java
#
echo "===========================  Start of StarCCM+  ===================================="
echo "starccm+ -power -podkey <Name_of_your_POD_license_key> -licpath 1999@flex.cd-adapco.com -rsh /usr/bin/ssh -mpi intel -mpiflags \"-bootstrap slurm\" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE"
#
starccm+ -power -podkey <Name_of_your_POD_license_key> -licpath 1999@flex.cd-adapco.com -rsh /usr/bin/ssh -mpi intel -mpiflags "-bootstrap slurm" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE
echo "===========================  End of StarCCM+  ======================================"

Assumed that the above SLURM script has been saved under the filename "starccm_POD_mpp3_slurm.sh", the SLURM batch job has to be submitted by issuing the following command on one of the Linux Cluster login nodes:

sbatch starccm_POD_mpp3_slurm.sh

Using a LRZ Floating License

In the following an example of a job submission batch script for StarCCM+ on CoolMUC-3 (SLURM queue = mpp3) in the batch queuing system SLURM is provided. The example is formulated for the use of a StarCCM+ floating license being provided by the LRZ internal license server license1.lrz.de (User-ID authentication for license check-out). Consequently befor the use of StarCCM+ floating licenses the User-ID of the user has to be registered in the LRZ license server (please send a LRZ Service Request).

#!/bin/bash
#SBATCH -o ./myjob.%j.%N.out
#SBATCH -D ./
#SBATCH -J starccm_mpp3_slurm
#SBATCH --clusters=mpp3
#SBATCH --get-user-env
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=64
# --- multiples of 64 for mpp3 ---
#SBATCH --mail-type=NONE
#SBATCH --mail-user=Max.Mustermann@lrz.de
#SBATCH --export=NONE
#SBATCH --time=00:10:00
#
module load slurm_setup

module av starccm
module load starccm/2023.1.1        # <-- mixed precision version
# module load starccm_dp/2023.1.1   # <-- double precision version
module list

echo "=================== Verification of the StarCCM+ Hostlist =========================="
echo $STAR_HOSTLIST
cat $STAR_HOSTLIST
echo "=================== Verification of the StarCCM+ Hostlist =========================="

# cat /proc/cpuinfo
# Provide your StarCCM+ simulation file name and Java macro name here:
SIM_FILE=Linux_StarCCM_Test.sim
START_MACRO=MeshSimSave.java
#
echo "===========================  Start of StarCCM+  ===================================="
echo "starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags \"-bootstrap slurm\" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE"
#
starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags "-bootstrap slurm" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $START_MACRO -batch-report $SIM_FILE
echo "===========================  End of StarCCM+  ======================================"

Assumed that the above SLURM script has been saved under the filename "starccm_floatlic_mpp3_slurm.sh", the SLURM batch job has to be submitted by issuing the following command on one of the Linux Cluster login nodes:

sbatch starccm_floatlic_mpp3_slurm.sh

Warning: Do NOT use additionally mpirun, mpiexec or any srun command to start the parallel processes. This is done by a MPI wrapper by the starccm+ startup script in the background. Also, do not try to change the default Intel MPI to any other MPI version to run StarCCM+ in parallel. On the LRZ cluster systems only the usage of Intel MPI is supported and known to work propperly with Simcenter StarCCM+.

SuperMUC-NG : Simcenter StarCCM+ Job Submission on SNG running SLES15 using SLURM

In the following an example of a job submission batch script for StarCCM+ on SuperMUC-NG (Login node: skx.supermuc.lrz.de, SLURM partition = test) in the batch queuing system SLURM is provided.
Please note that POD licensing is not supported on SuperMUC-NG, since it is not possible to reach any external license server from SuperMUC-NG compute nodes due to additional hardening of this supercomputer machine.
LRZ can provide a rather small number of available StarCCM+ floating licenses on request. Alternatively interested users in using StarCCM+ on SuperMUC-NG need to provide their own StarCCM+ licenses on a hosted license server in the LRZ network. For that the migration of existing license pools to this LRZ license server need to be arranged with Siemens PLM as a software vendor for StarCCM+. If that is the licensing solution for starCCM+ on SuperMUC-NG you would like to go for, please contact our LRZ Service Desk accordingly.

#!/bin/bash
#SBATCH -o ./myjob_%x.%j.%N.out
#SBATCH -D ./
#SBATCH -J starccm_sng
#SBATCH --partition=test
# ---- test | micro | general | large | fat
#SBATCH --get-user-env
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
# ---- multiples of 48 for SuperMUC-NG ----
#SBATCH --mail-type=NONE
#SBATCH --mail-user=Max.Mustermann@lrz.de
#SBATCH --no-requeue
#SBATCH --export=NONE
#SBATCH --time=0:15:00
#SBATCH --account=<...your project ID ...>
#SBATCH --switches=1@24:00:00
#########################################################
#-----------------------------------------
module load slurm_setup 

module av starccm
# module load starccm/2023.3.1
module load starccm_dp/2023.3.1
module list

echo "=================== Verification of the StarCCM+ Hostlist =========================="
echo $STAR_HOSTLIST
cat $STAR_HOSTLIST
echo "=================== Verification of the StarCCM+ Hostlist =========================="
 
SIM_FILE=Linux_StarCCM_Test.sim
START_MACRO=MeshSimSave.java

echo "=========================  Start von StarCCM+  ===================================="
echo starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags \"-bootstrap ssh\" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $SIM_FILE -batch $START_MACRO -batch-report
date
starccm+ -licpath 1999@license1.lrz.de -rsh /usr/bin/ssh -mpi intel -mpiflags "-bootstrap ssh" -printpids -machinefile $STAR_HOSTLIST -np $SLURM_NTASKS -batch $SIM_FILE -batch $START_MACRO -batch-report
date
echo "=========================  Ende von StarCCM+  ===================================="

Assumed that the above SLURM script has been saved under the filename "starccm_sng_slurm.sh", the SLURM batch job has to be submitted by issuing the following command on one of the SuperMUC-NG login nodes:

sbatch starccm_sng_slurm.sh