This page give you a short introduction how to use spack at LRZ based on a preconfigured installation. It make use of the LRZ provided software stack. This documentation focuses on the LRZ specific configuration. If you haven't used spack before, it is probably advisable to go through a tutorial https://spack-tutorial.readthedocs.io/en/latest/

Note that details in the tutorial may differ due to different versions of spack.

Activate Spack

To get started with user space spack just do a

> module load user_spack

which loads a setup script that adds the spack function to your shell. Currently, we only support bash-like shells. The spack version matches the version the default software stack has been built. If you need the user spack for a different version, you can check the availability with `module av user_spack`.

Version and Documentation

To see if everything works correctly try a

> spack -V
0.13.4

and obtain the spack release version in use. LRZ will typically stick to a release version of spack for each software stack, shown here for the release stack 20.1. If you haven't used spack before, you may want to check out the documentation at

https://spack.readthedocs.io/

Make sure that you select the documentation for the spack version in use, since the software evolves quickly and new features may not be available in the version used for the software stack.

Basic Spack Commands

Here are the most important commands to find your way around in the spack installation.

spack list

To check if a package is available in spack you can use the list command

> spack list kokk
==> 2 packages.
kokkos  kokkos-kernels

So two packages match the search string.

spack info

Details are given by the info command

> spack info kokkos
Package:   kokkos

Description:
    Kokkos implements a programming model in C++ for writing performance
    portable applications targeting all major HPC platforms.

Homepage: https://github.com/kokkos/kokkos

Tags: 
    None

Preferred version:  
    2.9.00     https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz

Safe versions:  
    develop    [git] https://github.com/kokkos/kokkos.git on branch develop
    2.9.00     https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz
    2.8.00     https://github.com/kokkos/kokkos/archive/2.8.00.tar.gz
...

spack versions

Available versions are listed by the versions command

> spack versions kokkos
==> Safe versions (already checksummed):
  develop  2.9.00  2.8.00  2.7.24  2.7.00  2.5.00  2.04.11  2.04.04  2.04.00  2.03.13  2.03.05  2.03.00  2.02.15  2.02.07
==> Remote versions (not yet checksummed):
  3.1.01  3.1.00  3.0.00  2.6.00

spack find

The command find checks if a package is already installed

> spack find -l elpa
==> 9 installed packages
-- linux-sles15-haswell / gcc@8.4.0 -----------------------------
o5ty6jb elpa@2017.11.001

-- linux-sles15-haswell / intel@19.0.5.281 ----------------------
vfxqn5c elpa@2018.11.001  zhoqg3p elpa@2019.05.002  bwykpas elpa@2019.11.001  nygb6zs elpa@2020.05.001
bhl4qmn elpa@2018.11.001  d2yg2xi elpa@2019.05.002  qztc5oz elpa@2019.11.001  i3gjkmf elpa@2020.05.001

shows the installed elpa versions and the compiler that has been used (gcc/intel). The -l option provides the unique hash for each package, which is also included in the name of the installation directory (check with > spack find -lp elpa).

spack compilers

The compilers command lists the available compilers in spack

> spack compilers
==> Available compilers
-- gcc sles15-x86_64 --------------------------------------------
gcc@9.3.0  gcc@8.4.0  gcc@7.5.0

-- intel sles15-x86_64 ------------------------------------------
intel@19.1.1.217  intel@19.0.5.281  intel@19.0.4.243  intel@18.0.4

These are the preconfigured ones. You may add your own using > spack compiler add /path/to/compiler

Installing Software with Spack

When installing software with spack in your home directory, you can make use of the resources and versions provided by LRZ.

spack spec

The spec command shows you the dependency tree of a given package

> spack spec -I libvdwxc      
Input spec
--------------------------------
 -   libvdwxc

Concretized
--------------------------------
 -   libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell
[^]      ^fftw@3.3.8%gcc@8.4.0+mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-haswell
[^]          ^intel-mpi@2019.7.217%gcc@8.4.0 arch=linux-sles15-haswell

The libvdwxc library depends on fftw und intel-mpi. The library itself is not yet installed (-) but both fftw and intel-mpi are already available via the upstream LRZ installation ([^]). A package that you have installed locally in your home directory with spack is marked by [+].

spack install

> spack install libvdwxc
==> intel-mpi is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/intel-mpi/2019.7.217-gcc-2vt6dak
==> fftw is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/fftw/3.3.8-gcc-vnifeuh
==> Installing libvdwxc
==> Searching for binary cache of libvdwxc
==> Finding buildcaches in /lrz/sys/spack/cache/build_cache
==> No binary for libvdwxc found: installing from source
==> Fetching file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
curl: (37) Couldn't open file /lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
==> Fetching from file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz failed.
==> Fetching file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz
curl: (37) Couldn't open file /lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz
==> Fetching from file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz failed.
==> Fetching https://launchpad.net/libvdwxc/stable/0.4.0/+download/libvdwxc-0.4.0.tar.gz
############################################################################################################################# 100.0%
==> Staging archive: /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk/libvdwxc-0.4.0.tar.gz
==> Created stage in /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk
==> No patches needed for libvdwxc
==> Building libvdwxc [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed libvdwxc
  Fetch: 0.63s.  Build: 24.30s.  Total: 24.93s.
[+] /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava

line 2+3 : packages are taken from the LRZ installation
line 8-13: spack checks if the source tar ball is available in the LRZ cache (not availabe)
line 14: download from external site
line 16-17: a build directory is created in $TMPDIR
line 18-24: library is built and installed
line 26: installation location of the library in $HOME/spack/opt/...

Providing source files

After a package has been downloaded, the soure files are stored in $HOME/spack/cache

Spack will search this cache next time you install the software and obtain the source from there:

> spack uninstall -ay libvdwxc
...
> spack install libvdwxc
...
==> Using cached archive: /path/to/home/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz

Here, the source tar ball is stored indexed by its sh256 checksum. Sometimes you will have to supply the source tar ball by hand, either because it is commercial software or the download is not reachable due to firewall restrictions. You could follow the checksum caching scheme here, but more user friendly is to place the source in $HOME/spack/cache/<package name>/<package>-<version>.<suffix>. You can deduce the correct naming scheme from the error message of an unsuccessful installation attempt.

Example:

> mkdir spack/cache/libvdwxc/
> mv /$HOME/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
> spack uninstall -ay libvdwxc
==> Successfully uninstalled libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell/4eofava
> spack install libvdwxc
...
==> Using cached archive: /path/to/home/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz

Generating Modules for Your Spack Software

The module command triggers the generation of tcl environment modules.

> spack module tcl refresh libvdwxc
==> You are about to regenerate tcl module files for:

-- linux-sles15-haswell / gcc@8.4.0 -----------------------------
4eofava libvdwxc@0.4.0

==> Do you want to proceed? [y/n] y
==> Regenerating tcl module files
> module show libvdwxc
-------------------------------------------------------------------
/path/to/home/spack/modules/x86_avx2/linux-sles15-haswell/libvdwxc/0.4.0-gcc8-impi:

module-whatis   {Portable C library of density functionals with van der Waals interactions for density functional theory}
conflict        libvdwxc
prepend-path    PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/bin
prepend-path    LD_LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib
prepend-path    LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib
prepend-path    PKG_CONFIG_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib/pkgco>
prepend-path    CMAKE_PREFIX_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/
setenv          LIBVDWXC_BASE /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava
------------------------------------------------------------------

line 1: spack module tcl refresh will regenerate all module file. We have configured spack such that modules will only be generated for explicitly installed packages.
line 11: modules are installed in $HOME/spack/modules/$LRZ_INSTRSET/<architecture>/<package>/<version>
line 15-20: the default module generated by spack sets the most important path and environment variables. If pgk-config support is available for a package the path is also included.

The path to the modules is added to $MODULEPATH when you load the user_spack module, but only if it already exists. So your new modules may only be visible the next time you load user_spack. You can check if the path is included with the command

> module use
Search path for module files (in search order):
  /path/to/home/spack/modules/x86_avx2/linux-sles15-haswell
  /lrz/sys/spack/staging/20.1.1/modules/haswell/linux-sles15-haswell
  /lrz/sys/spack/staging/20.1.1/modules/x86_64/linux-sles15-x86_64

There, the path to spack/modules should appear in the first line.

If you want to use these modules without loading spack_user first, you can add a

module use $HOME/spack/modules/$LRZ_INSTRSET/linux*

to your ~/.bashrc file (~/.cshrc, correspondingly).

Creating Your Own Package

Sometimes package.py files, which describe how to build a package, don't use the most recent version, are broken, or don't exist in the first place for your package. When you fix existing packages or create new packages you need to store them in your private repository.

spack repo

The first step ist to create your own repository

> mkdir ~/spack/repos
> spack repo create ~/spack/repos/mine
==> Created repo with namespace 'mine'.
==> To register it with spack, run this command:
  spack repo add /path/to/home/spack/repos/mine

Now we have a repository with the name 'mine'. Next we need to register it, following the instructions in the last line

> spack repo add ~/spack/repos/mine
==> Added repo with namespace 'mine'.
> spack repo list
==> 4 package repositories.
mine        /path/to/home/spack/repos/mine
lrz         /lrz/sys/spack/release/20.1/repos/lrz
backport    /lrz/sys/spack/release/20.1/repos/backport
builtin     /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/var/spack/repos/builtin

This package repo mine was now added to the list of available repo from spack itself builtin and from the LRZ provided repos lrz and backport.

spack edit / spack create

If you want to fix a package from the preconfigured repos, you copy to package to your own repo

> cp -r /path/to/repo/packages/libxc ~/spack/repos/mine/packages/
> spack edit libxc
  1 # Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
  2 # Spack Project Developers. See the top-level COPYRIGHT file for details.
  3 #
  4 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
  5 
  6 from spack import *
  7 
  8 
  9 class Libxc(AutotoolsPackage):
 10     """Libxc is a library of exchange-correlation functionals for
"~/spack/repos/mine/packages/libxc/package.py" 91L, 3618C

The spack edit command opens the package file from the 'mine' repo in vim or $EDITOR.

If you create a new package with spack create, the package will be generated by default in the first repo, thus, in mine. For instructions and details or spack create please see the documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html .

Configuring Your Spack Instance

LRZ provides a configuration that is very similar to the one the software stack was built with. You may want to change some or all of these settings to serve your needs, e.g. for the package selection, generation of modules, etc. Your individual configuration files are stored in the directory ~/.spack/. With the spac repo add command, you may already have created your own ~/.spack/repos.yaml file there. User config files take precedence over system provided config file, that is they are loaded after the system config files and overwrite their settings.

spack config

Configuration files can be viewed and modified with the config command. Taking up our repos expample

> spack config blame repos
---                                                                               repos:
/path/to/home/.spack/linux/repos.yaml:2                               - /path/to/home/spack/repos/mine
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:2  - /lrz/sys/spack/release/20.1/repos/lrz
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:3  - /lrz/sys/spack/release/20.1/repos/backport
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:4  - $spack/var/spack/repos/builtin

we see that the 'mine' repo stems form the private config file in ~/.spack/linux/, whereas the other repos are configured in the spack site directory. All other config files reside in the same .../etc/spack/linux/ directory. You can either create new config files from scatch in ~/.spack/linux/ or copy them from the site directory and edit them. The command

> spack config --scope=site/linux edit --print-file modules
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/modules.yaml

gives you the full path. The --scope=site/linux option points to the site configuration (default are the user's config files).

You can edit your config files directly with spack. Check out

> spack config edit -h     
usage: spack config edit [-h] [--print-file] [SECTION]

positional arguments:
  SECTION       configuration section to edit. options: compilers, mirrors, repos, packages, modules, config, upstream

For the syntax and structure of these configuration files, please refer to the spack documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html

Example: Generating modules for upstream packages

For packages that have been implicitly installed in the upstream spack stack no modules are generated by default. You can configure your `user_spack` such that you generate modules in your $HOME directory. Call

> spack config edit modules

and enter

modules:
  tcl:
    blacklist_implicits: False
    hash_length: 7

(ends up in `~/.spack/modules.yaml`). Now modules are generated for all installed modules (blacklisting of implitcits turned off) and each newly created module gets a hash suffix of length 7 to avoid naming conflicts. Modules are generated with

> spack module tcl refresh --upstream-modules

For a more fine grained control of the modules to be generated you could use the following scheme

modules:
  tcl:
    blacklist_implicits: False 
    hash_length: 7
    blacklist: [ '%intel', '%gcc' ]
    whitelist:
    - cmake

The blacklist blocks all packages installed with intel and gcc compilers (essentially all) and the whitelist unblocks all versions of cmake.