What is Moose Framework?
Moose comes already with a lot of modules, but can be extended as is the case for Golem.
The basic installation steps are as follows (please also consult the Moose Framework documentation page).
This above procedure already assumes a correct build environment. On the LRZ HPC cluster, such environment must be arranged for. And some adaptations are necessary, e.g. in order to use our tuned performance libraries (e.g. usage of HDF5, Intel MKL and MPI, etc.). As quite usual for PETSc applications, the whole pipeline down to Moose must be build with the same tool chain (same compiler, compiler settings, MPI, ...).
On ColllMUC-2, for instance, the following procedure works reasonably well.
Some (10) tests may fail. Some 100 are skipped maybe. As long as this is not essential for your workflow, you can live with that.
The hardware-specific GCC optimization flags must be changed when using a different compiler, or using different hardware.
For building Moose apps, the environment must be restored (compilers, tool chain, libraries ... please consider module collections (
module help), can be placed also in an own user-defined module file), and
MOOSE_DIR must be set to the moose top directory. For instance, building (separately) the moose apps (although
MOOSE_DIR is not necessary in this case), might work as follows.
Few tests may fail again. Some are skipped. Please check, whether that's critical for your workflows.
Finally, there are also examples,
moose/examples. Good starting point to learn the workflows of Moose, and to have some reference on how to setup solvers and cases.
For the run-time application, only the run-time libraries are necessary (boost may only be used as compile-time library; but loading it does not harm ... gcc is maybe also not relevant).
The framework includes most libraries in the executable app's
RPATH. However, Intel MKL/MKI and HDF5 modules also provide run-time optimization settings via environment variables. So, loading these modules is recommended.
Moose applications have the option
--help. Use this to learn about run-time adaptations, and monitoring capabilities. As PETSc applications, Moose apps also accept the PETSc run-time cmd parameters. A Slurm job script can be kept rather short. E.g.
HPC relevant Topics
- Create two module collections - one for build-time, and one for run-time.
- Compile the whole chain (PETSc-Libmesh-Moose app) with AVX support for the respective hardware. Computational frameworks like PETSc usually benefit from this.
- Moose apps are MPI programs and are usually started via
srun --mpi=pmi2, or the like. However, some applications may also have the option
--n-threads=<# threads per rank>. Hybrid MPI/Thread execution of applications on the LRZ clusters is recommended for efficient use on NUMA nodes (see the Usage section above).
- Measure Performance: Specifically when you start with a new case, this is important! Start with few time steps. Check correct MPI rank/thread placement to the CPU cores. Try to assess the run-time and memory consumption requirements (if memory becomes a bottleneck on the nodes, consider to use distributed meshes). Perform some scaling test with the test-case at hand, in oder to assess the possibility for accelerating your computations. (A parallel efficiency of 70% and more are ok. Please also look on the Slurm queue limits!)
Assessing the total runtime of a simulation case might prove difficult, because of the adaptive time-step integration. But starting from a certain time-step is possible, and thus stop and restart is a solution to recursively extend the simulation's total time-integration time.
- Pre/Post Processing: Most file formats used in Libmesh/Moose can be analysed with ParaView.
Why don't you provide Moose as a centrally installed Module at LRZ?
- Still only few users with diverging requirements.
- Moose/Apps experience ongoing rapid development. When users start using some fixed-version, or a bigger community starts using the apps by Moose, we can revise our decision.
- Education: Moose is a framework meant to support the development of apps. This is, on the one hand, often not easy to provide as a central module. On the other hand, users should learn to know the complete setup of their tools (including the build of PETSc and Libmesh ... the moose developers have already simplified the business). This is best done when users practice the installation by themselves. For support requests, please contact our Service Desk.
- But for OpenFOAM, there are central modules. And OpenFOAM ist also a framework. That is true. But: More users (much larger community) are using the software as is (no development). Well settled environment management for build and run-time. Well settled release cycle and versioning strategy. Industrial support. We are not going to discriminate Moose here. We just have to make reasonable decisions accounting for limited man power for the support of software.