FAQ: Accessing SVN, GIT and Mercurial repositories on SuperMUC-NG


The SuperMUC firewall permits only incoming SSH connections.

Access to subversion (SVN) server

You can use port forwarding to establish a connection between the subversion server and SuperMUC, i.e., you may use one of the following procedures.

  • You will be prompted for your SuperMUC password (or your ssh passphrase). If you are unlucky, the port selected by you (e.g., 10022) is already used by someone else - in this case; you will see an error message printed out in advance of the motd; you then need to change your port to a different value.
  • You might need to delete the localhost entry from ~/.ssh/known_hosts if ssh complains about the host key.

If you need to change ssh ports, you will probably also need to invoke "svn switch --relocate ..." on your SVN sandboxes because the port number will be encoded in the stored location.

Using SVN with an HTTPS svn server

To establish the port forwarding for the SSL/TLS port, issue the following command to connect from your workstation you normally use to SSH to the SuperMUC:

ssh -l <LoginName> -R <arbitraryPortNumber>:<svnServer>:443 skx.supermuc.lrz.de
Example:
ssh -l hk00xyz -R 10443:pmviewer.svn.sourceforge.net:443 skx.supermuc.lrz.de

After successful login to SuperMUC, you may then access your repository via ("module load subversion"):

svn <svnCommand> https://<remoteLoginName>@localhost:<ForwardedPortNumber>/<svnDirectoryPath>
Example:               
svn list https://mySVNUser@localhost:10443/svnroot/pmviewer
svn co   https://mySVNUser@localhost:10443/svnroot/pmviewer pmviewer

Using SVN+SSH repository access

To establish the port forwarding for the SSH port, issue the following command to connect from your workstation you normally use to SSH to the system SuperMUC:

ssh -l <your_userID_on_SuperMUC-NG> -R <arbitraryPortNumber>:<machine-withSVNrepo.>:22 skx.supermuc.lrz.de
Example:
ssh -l hk00xyz -R 10022:mySVNmachine.myhost.de:22 skx.supermuc.lrz.de
Repository:
mySVNmachine.myhost.de:/my/svn/repo

After successful login to supermuc, you have to set up a new protocol in your ~/.subversion/config file. Therefore you enter the following last line to the tunnel section in the config file:

[tunnels]
### Configure svn protocol tunnel schemes here.  By default, only
### the 'ssh' scheme is defined.  You can define other schemes to
### be used with 'svn+scheme://hostname/path' URLs.  A scheme
### ...
myssh = ssh -p 10022

Now you may use the svn+ssh command as usual, with the exception that the newly defined myssh protocol is used instead of the standard ssh protocol:

svn <svnCommand> svn+myssh://<remoteLoginName>@localhost/<svnDirectoryPath>
Example:               
svn list svn+myssh://mySVNUser@localhost/my/svn/repo
svn co   svn+myssh://mySVNUser@localhost/my/svn/repo

GIT

Git supports four protocols to access remote repositories: Local, HTTP(S), SSH, and git. The method required to use Git on the SuperMUC-NG depends on the chosen protocol:

Note: No module load is needed as system git is available.

As mentioned above, The SuperMUC firewall permits only incoming SSH connections; following workarounds are possible to manage the git workflow when using HTTP(S), SSH, and git protocol.

Via HTTP(S) Protocol

The most popular way to use Git now is via HTTP or HTTPS protocol because of its simplicity and authentication features. 

For GitHub and GitLab servers, please use the following workaround, 

Establish a proxy by using a feature of OpenSSH; on your local desktop/laptop/PC/MAC, do,

local> ssh -N -D 1080 localhost

Note: For this method to work, you would want to check if, on your local machine, remote access to your own machine is enabled.  

This will create a "loopback" SSH connection from your laptop/desktop to itself. "-N" will not execute any command but will act as a SOCKS proxy on port 1080 (-D 1080)

Open a second terminal/shell, login to the SuperMUC-NG, and forward the port on the remote machine(e.g., 8888) to the port on your laptop/desktop(1080) where the "just established" SOCKS proxy is listening on:

local> ssh -R 8888:localhost:1080 skx.supermuc.lrz.de

By doing so, you have a SOCKS proxy listening on port 8888 of the SuperMUC-NG. You can use this proxy for accessing the remote git repositories (such as GitLab repo or GitHub repo). The option "-c http.sslVerify=false" is to circumvent the SSL certificate issue of HTTPS.

In order to do so, you can now add "-c http.proxy='socks5://localhost:8888'" to your git commands.

For example, for the Github repositories and GitLab repositories,

skx> git -c http.proxy='socks5://localhost:8888' -c http.sslVerify=false clone https://github.com/spack/spack.git

or

skx> git -c http.proxy='socks5://localhost:8888' -c http.sslVerify=false clone https://gitlab.lrz.de/<path>/<other_path>.git

Note: For gitlab.lrz.de, the user can directly git clone without any of these steps.  But to access/clone the repository hosted on an external GitLab server, you can use the step mentioned above.

In order to avoid typing this in every git call, you can add the entry in the ~/.gitconfig file. Here set the respective port to be used whenever git talks to a remote repository via HTTP by,

skx> cat ~/.gitconfig 
[http]

        sslVerify = false

        proxy = socks5://localhost:8888

Git-protocol does not allow push over this protocol. Generally, if you are only interested in read-only git repositories, then git-protocol is the fastest way to manage GIT workflow.

Note: By the end of 2020, GitHub will not support git-protocol.

local> ssh -R 12345:github.com:9418 <userID_on_SuperMUC-NG>@skx.supermuc.lrz.de

skx> git clone git://localhost:12345/<somepath>.git

or example for llvm repo,

skx> git clone git://localhost:12345/llvm-mirror/llvm.git

Important note: If you get the warning ("remote port forwarding failed for listen port 12345") then port 12345 is already in use by someone else. Replace 12345 with, e.g., 13345 or some other number >10,000.

Via SSH Protocol (port 22)

Let's assume we like to clone LLVM from GitHub with the SSH protocol, i.e.; we like to access the repository git@github.com:llvm-mirror/llvm.git. In order to do so, establish a tunnel from a port (e.g., 12355) on the SuperMUC-NG to port 22 on the repositories host (github.com) by,

local> ssh -R 12355:github.com:22 <userID_on_SuperMUC-NG>@skx.supermuc.lrz.de

or example for llvm repo,

skx> git clone ssh://git@localhost:12355/llvm-mirror/llvm.git

SSHFS

The other simple way is to mount a remote (SuperMUC-NG) filesystem locally.

local> mkdir mnt 
local> sshfs -o follow_symlinks <userID_on_SuperMUC-NG>@skx.supermuc.lrz.de: mnt
local> cd mnt
local> git clone ...      # or svn or hg or ...
... # do your work whatever it be
... # like git push or git pull
...
local> cd && fusermount -u mnt # on MAC-OS "cd && umount mnt"

Mercurial

local> ssh -l <your_userID_on_SuperMUC-NG> -R 12345:www.mercurial-scm.org:443 skx.supermuc.lrz.de
skx> module load mercurial
skx> hg clone --insecure https://localhost:12345/repo/hg/ mercurial-repo

"--insecure" is necessary to circumvent the SSL certificate issue of HTTPS (as the requested DNS is now "localhost")