FAQ: Python and R on SuperMUC-NG by using conda
SuperMUC-NG does not allow for outgoing internet connections and therefore conda does not work out of the box (same for pip or R-CRAN).
There is, however, a way to transfer conda environments from your local machine or the SuperMUC-Cloud to the SuperMUC File system easily:
Method 1:
Using conda pack:
Step 1: On the machine with internet connection (local linux workstation or SuperMUC-Cloud instance)
- Install conda, either
- on your local machine (linux x86_64) or
- in a VM on the SuperMUC-Cloud (see: Compute Cloud how to get a VM)
- Install all the packages that you want into a new environment (e.g. my_env)
- install the conda-pack package
$ conda install conda-pack
pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
then copy the tar file to SuperMUC-NG:
$ scp my_env.tar.gz skx.supermuc.lrz.de:~
Step 2: On SuperMUC-NG (no internet connection to outside)
Unpack environment into directory my_env
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_en
Activate the environment:
$ source my_env/bin/activate
This adds my_env/bin to your path.When using the environment for the first time, update the executable paths and prefixes with the following command:
(my_env)$ conda-unpack
- Run python in the environment
(my_env)$ python
Deactivate the environment to remove it from your path
(my_env)$ source my_env/bin/deactivate
If later on you need additional packages you have to go back to step 1
Method 2
Alternative - via local HTTP proxy and reverse SSH tunnels:
There is a chance to download conda packages directly to the NG filesystem via a local HTTP proxy, and reverse SSH tunnels. A decription is to be found here: https://medium.com/@brianray_7981/securely-install-jupyter-python-without-internet-connection-in-4-easy-steps-7d6907fe022e
An adaptation is to use the Intel Python installations on the SuperMUC-NG login nodes (e.g. "module load python/3.5_intel", and then "conda create -p $WORK_<project ID>/ic35/envs/test --clone root --offline" (if you want to install it into your WORK-directory)). A session that was tested under a Linux PC (local):
local> pip install --upgrade --user proxy.py
local> .local/bin/proxy.py --port 8899 # if this port number is already occupied, please take another once >1024, which is free
local> ssh -R 8899:localhost:8899 smng # incl. ProxyJumps, ...
sng> module load python/3.5_intel # 3.5_intel was tested successfully
sng> conda create -p $HOME/ic35/envs/test --clone root --offline
sng> cat > .condarc << EOT
channels:
- defaults
- r
show_channel_urls: True
allow_other_channels: True
proxy_servers:
http: http://127.0.0.1:8899
https: http://127.0.0.1:8899
ssl_verify: False
EOT
sng> source activate $HOME/ic35/envs/test
(conda) sng> conda install r r-irkernel
# example with jupyter notebook:
(conda) sng> jupyter notebook --ip=127.0.0.1 --port=8891 # also here, if port is already occupied, take another free one
(conda) sng> ~C # "~C" is a OpenSSH escape sequence to reach the SSH shell; it should not be visible on the screen; if it is, please press <Enter> once to clean the shell
ssh> -L 8891:localhost:8891 # port forwarding for the jupyter notebook
# local: start browser with localhost:8891/...
sng> echo 'alias initConda='module load python/3.5_intel; source activate $HOME/ic35/envs/test; export PS1="(conda) \u@\h:\w> "'' >> .bashrc
# for all following conda sessions
sng> initConda
(conda) <user>@login01:~> # go on with your work