FAQ: Python and R on SuperMUC-NG by using conda

SuperMUC-NG does not allow for outgoing internet connections and therefore conda does not work out of the box (same for pip or R-CRAN).

There is, however, a way to transfer conda environments from your local machine or the SuperMUC-Cloud to the SuperMUC File system easily:

Method 1:

Using conda pack:

Step 1: On the machine with internet connection (local linux workstation or SuperMUC-Cloud instance)

  • Install conda, either
    • on your local machine (linux x86_64) or
    • in a VM on the SuperMUC-Cloud (see: Compute Cloud how to get a VM)
  • Install all the packages that you want into a new environment (e.g. my_env)
  • install the conda-pack package
    $ conda install conda-pack
  • pack environment my_env into my_env.tar.gz
    $ conda pack -n my_env

  • then copy the tar file to SuperMUC-NG:
    $ scp my_env.tar.gz skx.supermuc.lrz.de:~

Step 2: On SuperMUC-NG (no internet connection to outside)

  • Unpack environment into directory my_env
    $ mkdir -p my_env
    $ tar -xzf my_env.tar.gz -C my_en

  • Activate the environment:
    $ source my_env/bin/activate
    This adds my_env/bin to your path.

  • When using the environment for the first time, update the executable paths and prefixes with the following command:

    (my_env)$ conda-unpack
  • Run python in the environment
    (my_env)$ python

  • Deactivate the environment to remove it from your path
    (my_env)$ source my_env/bin/deactivate

    If later on you need additional packages you have to go back to step 1

Method 2

Alternative - via local HTTP proxy and reverse SSH tunnels:

There is a chance to download conda packages directly to the NG filesystem via a local HTTP proxy, and reverse SSH tunnels. A decription is to be found here: https://medium.com/@brianray_7981/securely-install-jupyter-python-without-internet-connection-in-4-easy-steps-7d6907fe022e

An adaptation is to use the Intel Python installations on the SuperMUC-NG login nodes (e.g. "module load python/3.5_intel", and then "conda create -p $WORK_<project ID>/ic35/envs/test --clone root --offline" (if you want to install it into your WORK-directory)). A session that was tested under a Linux PC (local):

local> pip install --upgrade --user proxy.py
local> .local/bin/proxy.py --port 8899                              # if this port number is already occupied, please take another once >1024, which is free
local> ssh -R 8899:localhost:8899 smng                           # incl. ProxyJumps, ...
sng> module load python/3.5_intel                                  # 3.5_intel was tested successfully
sng> conda create -p $HOME/ic35/envs/test --clone root --offline
sng> cat > .condarc << EOT
- defaults
- r

show_channel_urls: True
allow_other_channels: True

ssl_verify: False
sng> source activate $HOME/ic35/envs/test
(conda) sng> conda install r r-irkernel

# example with jupyter notebook:
(conda) sng> jupyter notebook --ip= --port=8891      # also here, if port is already occupied, take another free one
(conda) sng> ~C                                                                          # "~C" is a OpenSSH escape sequence to reach the SSH shell; it should not be visible on the screen; if it is, please press <Enter> once to clean the shell
ssh> -L 8891:localhost:8891                                                  # port forwarding for the jupyter notebook

# local: start browser with localhost:8891/...

sng> echo 'alias initConda='module load python/3.5_intel; source activate $HOME/ic35/envs/test; export PS1="(conda) \u@\h:\w> "'' >> .bashrc

# for all following conda sessions
sng> initConda
(conda) <user>@login01:~>                                                  # go on with your work