Building software in user space with spack

This page give you a short introduction how to use spack at LRZ based on a preconfigured installation. It make use of the LRZ provided software stack. This documentation focuses on the LRZ specific configuration. If you haven't used spack before, it is probably advisable to go through a tutorial https://spack-tutorial.readthedocs.io/en/latest/

Note that details in the tutorial may differ due to different versions of spack.

Activate Spack

To get started with user space spack just do a

> module load user_spack

which loads a setup script that adds the spack function to your shell. Currently, we only support bash-like shells. The spack version matches the version the default software stack has been built. If you need the user spack for a different version, you can check the availability with `module av user_spack`.

Version and Documentation

To see if everything works correctly try a

> spack -V
0.13.4

and obtain the spack release version in use. LRZ will typically stick to a release version of spack for each software stack, shown here for the release stack 20.1. If you haven't used spack before, you may want to check out the documentation at 

https://spack.readthedocs.io/

Make sure that you select the documentation for the spack version in use, since the software evolves quickly and new features may not be available in the version used for the software stack.

Basic Spack Commands

Here are the most important commands to find your way around in the spack installation.

spack list

To check if a package is available in spack you can use the list command

> spack list kokk
==> 2 packages.
kokkos  kokkos-kernels

So two packages match the search string.

spack info

Details are given by the info command

> spack info kokkos
Package:   kokkos

Description:
    Kokkos implements a programming model in C++ for writing performance
    portable applications targeting all major HPC platforms.

Homepage: https://github.com/kokkos/kokkos

Tags: 
    None

Preferred version:  
    2.9.00     https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz

Safe versions:  
    develop    [git] https://github.com/kokkos/kokkos.git on branch develop
    2.9.00     https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz
    2.8.00     https://github.com/kokkos/kokkos/archive/2.8.00.tar.gz
...

spack versions

Available versions are listed by the versions command

> spack versions kokkos
==> Safe versions (already checksummed):
  develop  2.9.00  2.8.00  2.7.24  2.7.00  2.5.00  2.04.11  2.04.04  2.04.00  2.03.13  2.03.05  2.03.00  2.02.15  2.02.07
==> Remote versions (not yet checksummed):
  3.1.01  3.1.00  3.0.00  2.6.00

spack find

The command find checks if a package is already installed

> spack find -l elpa
==> 9 installed packages
-- linux-sles15-haswell / gcc@8.4.0 -----------------------------
o5ty6jb elpa@2017.11.001

-- linux-sles15-haswell / intel@19.0.5.281 ----------------------
vfxqn5c elpa@2018.11.001  zhoqg3p elpa@2019.05.002  bwykpas elpa@2019.11.001  nygb6zs elpa@2020.05.001
bhl4qmn elpa@2018.11.001  d2yg2xi elpa@2019.05.002  qztc5oz elpa@2019.11.001  i3gjkmf elpa@2020.05.001

shows the installed elpa versions and the compiler that has been used (gcc/intel). The -l option provides the unique hash for each package, which is also included in the name of the installation directory (check with > spack find -lp elpa).

spack compilers

The compilers command lists the available compilers in spack

> spack compilers
==> Available compilers
-- gcc sles15-x86_64 --------------------------------------------
gcc@9.3.0  gcc@8.4.0  gcc@7.5.0

-- intel sles15-x86_64 ------------------------------------------
intel@19.1.1.217  intel@19.0.5.281  intel@19.0.4.243  intel@18.0.4

These are the preconfigured ones. You may add your own using > spack compiler add /path/to/compiler

Installing Software with Spack

When installing software with spack in your home directory, you can make use of the resources and versions provided by LRZ.

spack spec

The spec command shows you the dependency tree of a given package

> spack spec -I libvdwxc      
Input spec
--------------------------------
 -   libvdwxc

Concretized
--------------------------------
 -   libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell
[^]      ^fftw@3.3.8%gcc@8.4.0+mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-haswell
[^]          ^intel-mpi@2019.7.217%gcc@8.4.0 arch=linux-sles15-haswell

The libvdwxc library depends on fftw und intel-mpi. The library itself is not yet installed (-) but both fftw and intel-mpi are already available via the upstream LRZ installation ([^]). A package that you have installed locally in your home directory with spack is marked by [+].

spack install

> spack install libvdwxc
==> intel-mpi is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/intel-mpi/2019.7.217-gcc-2vt6dak
==> fftw is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/fftw/3.3.8-gcc-vnifeuh
==> Installing libvdwxc
==> Searching for binary cache of libvdwxc
==> Finding buildcaches in /lrz/sys/spack/cache/build_cache
==> No binary for libvdwxc found: installing from source
==> Fetching file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
curl: (37) Couldn't open file /lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
==> Fetching from file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz failed.
==> Fetching file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz
curl: (37) Couldn't open file /lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz
==> Fetching from file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz failed.
==> Fetching https://launchpad.net/libvdwxc/stable/0.4.0/+download/libvdwxc-0.4.0.tar.gz
############################################################################################################################# 100.0%
==> Staging archive: /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk/libvdwxc-0.4.0.tar.gz
==> Created stage in /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk
==> No patches needed for libvdwxc
==> Building libvdwxc [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed libvdwxc
  Fetch: 0.63s.  Build: 24.30s.  Total: 24.93s.
[+] /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava
  • line 2+3 : packages are taken from the LRZ installation
  • line 8-13: spack checks if the source tar ball is available in the LRZ cache (not availabe)
  • line 14: download from external site
  • line 16-17: a build directory is created in $TMPDIR
  • line 18-24: library is built and installed
  • line 26: installation location of the library in $HOME/spack/opt/...

Providing source files

After a package has been downloaded, the soure files are stored in  $HOME/spack/cache

Spack will search this cache next time you install the software and obtain the source from there:

> spack uninstall -ay libvdwxc
...
> spack install libvdwxc
...
==> Using cached archive: /path/to/home/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz

Here, the source tar ball is stored indexed by its sh256 checksum. Sometimes you will have to supply the source tar ball by hand, either because it is commercial software or the download is not reachable due to firewall restrictions. You could follow the checksum caching scheme here, but more user friendly is to place the source in $HOME/spack/cache/<package name>/<package>-<version>.<suffix>. You can deduce the correct naming scheme from the error message of an unsuccessful installation attempt.

Example:

> mkdir spack/cache/libvdwxc/
> mv /$HOME/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
> spack uninstall -ay libvdwxc
==> Successfully uninstalled libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell/4eofava
> spack install libvdwxc
...
==> Using cached archive: /path/to/home/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz

Generating Modules for Your Spack Software

The module command triggers the generation of tcl environment modules.

> spack module tcl refresh libvdwxc
==> You are about to regenerate tcl module files for:

-- linux-sles15-haswell / gcc@8.4.0 -----------------------------
4eofava libvdwxc@0.4.0

==> Do you want to proceed? [y/n] y
==> Regenerating tcl module files
> module show libvdwxc
-------------------------------------------------------------------
/path/to/home/spack/modules/x86_avx2/linux-sles15-haswell/libvdwxc/0.4.0-gcc8-impi:

module-whatis   {Portable C library of density functionals with van der Waals interactions for density functional theory}
conflict        libvdwxc
prepend-path    PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/bin
prepend-path    LD_LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib
prepend-path    LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib
prepend-path    PKG_CONFIG_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib/pkgco>
prepend-path    CMAKE_PREFIX_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/
setenv          LIBVDWXC_BASE /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava
------------------------------------------------------------------
  • line 1: spack module tcl refresh will regenerate all module file. We have configured spack such that modules will only be generated for explicitly installed packages.
  • line 11: modules are installed in $HOME/spack/modules/$LRZ_INSTRSET/<architecture>/<package>/<version>
  • line 15-20: the default module generated by spack sets the most important path and environment variables. If pgk-config support is available for a package the path is also included.

The path to the modules is added to $MODULEPATH when you load the user_spack module, but only if it already exists. So your new modules may only be visible the next time you load user_spack. You can check if the path is included with the command


> module use
Search path for module files (in search order):
  /path/to/home/spack/modules/x86_avx2/linux-sles15-haswell
  /lrz/sys/spack/staging/20.1.1/modules/haswell/linux-sles15-haswell
  /lrz/sys/spack/staging/20.1.1/modules/x86_64/linux-sles15-x86_64


There, the path to spack/modules should appear in the first line.

If you want to use these modules without loading spack_user first, you can add a

module use $HOME/spack/modules/$LRZ_INSTRSET/linux*

to your ~/.bashrc file (~/.cshrc, correspondingly).

Software already existing in the Software Stack

Sometimes you wish to create a module for software we have indeed installed, but don't provide a module for it (for whatsoever reason). Still, all is not lost. You can create a module also from those.

Ideally, you figured out with "spack find -pL <software name>" whether this software (library, e.g.) is installed, where it is installed, and its hash. Please, check the correctness of OS version, hardware, compiler! For instance, on CoolMUC-2, let's check for udunits:

> module load user_spack
> spack find -pL udunits
==> 4 installed packages
-- linux-sles15-haswell / gcc@11.2.0 ----------------------------
y2lpepnvxom7pqefxejnmgiqqzrbvhj3 udunits@2.2.28  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/udunits/2.2.28-gcc-y2lpepn

-- linux-sles15-haswell / intel@21.4.0 --------------------------
42k5yhzgvhf526bk6zunvswtiq6h3jsb udunits@2.2.28  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/udunits/2.2.28-intel-42k5yhz

-- linux-sles15-skylake_avx512 / gcc@11.2.0 ---------------------
g3vblls2sjgk7stxnbe4aqerh4dshonl udunits@2.2.28  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/skylake_avx512/udunits/2.2.28-gcc-g3vblls

-- linux-sles15-skylake_avx512 / intel@21.4.0 -------------------
4ktifb7jmaqqll6kzxkfww3h3kbiudxz udunits@2.2.28  /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/skylake_avx512/udunits/2.2.28-intel-4ktifb7

We are on SLES 15, Haswell (CoolMUC-2), and look for gcc/11.2.0 compiled libraries (if it matters ... ). So, y2lpepnvxom7pqefxejnmgiqqzrbvhj3 is the desired hash (you can also use the first 7 characters or so, as long as it is unique).

Let's assume, we wish our modules be located in ~/modules (it makes sense to differentiate also the spack version ... but we skip this for sake of illustration). So, we can create the module file as follows:

> mkdir -p ~/modules
> spack -c"modules:tcl:blacklist_implicits:False" -c"modules:default:roots:tcl:$HOME/modules" module tcl refresh --upstream-modules /y2lpepnvxom7pqefxejnmgiqqzrbvhj3

Some explanations:

1) The -c options are necessary to overwrite the defaults. Some modules are explicitly blacklisted by us. No need to live with that for you! Furthermore, user_spack makes a different directory the default module path for each user. That's for convenience ... if you don't want to think much about that. But if you do, you can also change that, as shown. (Please feel free to parse the spack documentation for more useful features!)

2) We vote for using the hashes, although in many cases, also the package names work ... maybe but for the price that you create modules also for OS versions/hardware/compilers you don't need.

Usage is then simple:

> module use -p ~/modules
> module load udunits

We recomment also to consider module collections (see module help!) for convenient restoring complete environments (including the insertion of module search paths). Especially for handling many modules, this can really accelerate your workflow!

Creating Your Own Package

Sometimes package.py files, which describe how to build a package, don't use the most recent version, are broken, or don't exist in the first place for your package. When you fix existing packages or create new packages you need to store them in your private repository.

spack repo

The first step ist to create your own repository

> mkdir ~/spack/repos
> spack repo create ~/spack/repos/mine
==> Created repo with namespace 'mine'.
==> To register it with spack, run this command:
  spack repo add /path/to/home/spack/repos/mine

Now we have a repository with the name 'mine'. Next we need to register it, following the instructions in the last line

> spack repo add ~/spack/repos/mine
==> Added repo with namespace 'mine'.
> spack repo list
==> 4 package repositories.
mine        /path/to/home/spack/repos/mine
lrz         /lrz/sys/spack/release/20.1/repos/lrz
backport    /lrz/sys/spack/release/20.1/repos/backport
builtin     /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/var/spack/repos/builtin

This package repo  mine was now added to the list of available repo from spack itself builtin and from the LRZ provided repos lrz and backport.

spack edit / spack create

If you want to fix a package from the preconfigured repos, you copy to package to your own repo

> cp -r /path/to/repo/packages/libxc ~/spack/repos/mine/packages/
> spack edit libxc
  1 # Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
  2 # Spack Project Developers. See the top-level COPYRIGHT file for details.
  3 #
  4 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
  5 
  6 from spack import *
  7 
  8 
  9 class Libxc(AutotoolsPackage):
 10     """Libxc is a library of exchange-correlation functionals for
"~/spack/repos/mine/packages/libxc/package.py" 91L, 3618C

The spack edit command opens the  package file from the 'mine' repo in vim or $EDITOR.

If you create a new package with spack create, the package will be generated by default in the first repo, thus, in mine. For instructions and details or spack create please see the documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html .

Configuring Your Spack Instance

LRZ provides a configuration that is very similar to the one the software stack was built with. You may want to change some or all of these settings to serve your needs, e.g. for the package selection, generation of modules, etc. Your individual configuration files are stored in the directory ~/.spack/. With the spac repo add command, you may already have created your own ~/.spack/repos.yaml file there. User config files take precedence over system provided config file, that is they are loaded after the system config files and overwrite their settings.

spack config

Configuration files can be viewed and modified with the config command. Taking up our repos expample

> spack config blame repos
---                                                                               repos:
/path/to/home/.spack/linux/repos.yaml:2                               - /path/to/home/spack/repos/mine
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:2  - /lrz/sys/spack/release/20.1/repos/lrz
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:3  - /lrz/sys/spack/release/20.1/repos/backport
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:4  - $spack/var/spack/repos/builtin

we see that the 'mine' repo stems form the private config file in ~/.spack/linux/, whereas the other repos are configured in the spack site directory. All other config files reside in the same .../etc/spack/linux/ directory. You can either create new config files from scatch in ~/.spack/linux/ or copy them from the site directory and edit them. The command

> spack config --scope=site/linux edit --print-file modules
/dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/modules.yaml

gives you the full path.  The --scope=site/linux option points to the site configuration (default are the user's config files).

You can edit your config files directly with spack. Check out

> spack config edit -h     
usage: spack config edit [-h] [--print-file] [SECTION]

positional arguments:
  SECTION       configuration section to edit. options: compilers, mirrors, repos, packages, modules, config, upstream

For the syntax and structure of these configuration files, please refer to the spack documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html

Example: Generating modules for upstream packages

For packages that have been implicitly installed in the upstream spack stack no modules are generated by default. You can configure your `user_spack` such that you generate modules in your $HOME directory. Call

> spack config edit modules

and enter


modules:
  tcl:
    blacklist_implicits: False
    hash_length: 7

(ends up in `~/.spack/modules.yaml`). Now modules are generated for all installed modules (blacklisting of implitcits turned off) and each newly created module gets a hash suffix of length 7 to avoid naming conflicts. Modules are generated with

> spack module tcl refresh --upstream-modules

For a more fine grained control of the modules to be generated you could use the following scheme

modules:
  tcl:
    blacklist_implicits: False 
    hash_length: 7
    blacklist: [ '%intel', '%gcc' ]
    whitelist:
    - cmake

The blacklist blocks all packages installed with intel and gcc compilers (essentially all) and the whitelist unblocks all versions of cmake.

Installing Software at a central place (e.g. a shared DSS container)

If you want to install a custom version of a spack package using user_spack but you want also others to be able to use the installation, you can tweak the installation and module path for user_spack. For this we assume that everyone already has access to a shared folder /path/to/your/shared/folder (see also FAQ: Sharing files with other users ).

We create the necessary folders:

module load user_spack
mkdir -p ~/.spack # just to be sure
mkdir -p /path/to/your/shared/folder/user_spack/opt/
mkdir -p /path/to/your/shared/folder/user_spack/modules/

Then we edit the local spack config:

> spack config edit config

and add 

config:
  install_tree:
    root: /path/to/your/shared/folder/user_spack/opt/

Similarly:

> spack config edit modules

and add

modules:
  default:
    enable:
      - tcl
    roots:
      tcl: /path/to/your/shared/folder/user_spack/modules/

Now we install our package (which is not already installed in the central installation). Here we use zlib without shared libraries for demonstation:

> spack install zlib~shared
==> Installing zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln
==> No binary for zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln found: installing from source
==> Using cached archive: file:///lrz/sys/spack/cache/_source-cache/archive/b3/b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'edit'
==> zlib: Executing phase: 'build'
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln
  Stage: 0.27s.  Edit: 0.64s.  Build: 0.64s.  Install: 0.05s.  Total: 1.65s
[+] /path/to/your/shared/folder/user_spack/opt/linux-sles15-haswell/zlib/1.2.13-gcc-12.2.0-hjfs3ik

We generate the module(s):

> spack modules tcl refresh -y

and we should see the modules:

> module use -p /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell # "linux-sles15-haswell" could also be a bit different depending on the system your are on and the user_spack version. The correct path to use contains the folder called "zlib"
> module av zlib
---------------------------------------------------- /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell -----------------------------------------------------
zlib/1.2.13-gcc12

As we assumed that your colleagues already have access to the shared folder, all they should have to do is

module use -p /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell # as explained above, the subfolder of the "modules" folder can differ for your setup

and then load the wanted module. If it does not work, please re-check the permissions of /path/to/your/shared/folder/user_spack/ and its subfolders and correct if necessary.