ssh - Secure Shell on LRZ HPC Systems

Introduction

LRZ offers secure access to the HPC systems from outside via SSH ("secure shell"):

Password authentication using user name and password which was set in the LRZ IDM Portal. How to do this is described in the section "Basic usage" below. Most users will use this method.

For advanced users with special requirements LRZ also provides ssh-key login. If you want to use ssh keys, see chapter 3 below. We strongly discourage this method due to additional security risks like stolen and compromized ssh keys.

The following features of secure shell are relevant for users of LRZ HPC systems:

  • encrypted transfer of passwords and data
  • tunneling of services (in particular X11 graphics display) over the secure shell network port
  • usage of the version 2 protocol of ssh. Version 1 is not supported at LRZ, since not sufficiently secure.

Secure shell has superseded other, insecure protocols like rsh and telnet, which are not supported anymore at LRZ.

Two-Factor Authentication

SSH login to the HPC systems requires two-factor authentication. Please read here to learn more about it:

Usage

Installation on client system (your desktop)

While secure shell is installed by default on LRZ systems, it may be necessary to install it on your system.

  • Linux Systems: Most distributions contain an ssh client, but it may not have been installed. Please use the distribution-specific installation program to install the client
  • Windows Systems: Various free implementations are available on the internet, for example PuTTY or MobaXterm. An alternative is to install the Windows subsystem for Linux (WSL).
  • If you have a C compiler on your system, you can also install e.g. portable openssh from source.

Basic usage

Basic usage of ssh is very simple: Issue the command

$ ssh <LRZaccount>@<targetLRZsystem> 

You will be asked for your SIM password.  If the authentication is successfully concluded, you have a login shell on the target system. The target system typically must be specified with its full domain name.

A secure copy mechanism is also provided:

$ scp <myfile> <LRZaccount>@<targetLRZsystem>:<remotedirectory>
$ scp <LRZaccount>@<targetLRZsystem>:<remotedirectory> <myfile>

scp cannot be invoked on the LRZ system, because all outgoing connections are blocked. You have to use it on your local computer. Windows users can also use WinSCP or MobaXterm

The upper command will, for example, copy the local file <myfile> to the directory ~/<remotedirectory> on the remote system. This directory must exist prior to the copy attempt.

For additional security we provide the public system keys for SuperMUC-NG and Linux Cluster login nodes which can be added into the ~/.ssh/known_hosts file (you may need to generate the .ssh directory if it does not exist yet) on your local machine.

Policies for SSH public-key authentication

Passphrases must not be empty

When a key pair is generated (see below), the generator asks for a passphrase. It is not allowed to use an empty passphrase for keypairs that store private and public keys, respectively, on different systems. We consider an empty passphrase as a violation of elementary security policies and will shut out any user doing so from further usage of LRZ systems.

The only exception allowed is for keypairs that are only internally used, and if additional protective steps are taken to disable authentication from outside systems (see limitations on authorized keys below).

Limitations on encryption algorithms

On LRZ HPC systems, authentication by key pair will only be accepted if one of the following algorithms was used to generate key pairs:

  1. ECDSA
  2. ED25519 

We only accept the maximum key length of 521 bits (ECDSA only) and highly recommend to set 100 rounds (ECDSA and ED25519, increasing efforts to crack passwords of stolen keys). See How-to-do details in configuration section below.

Outgoing connections

Due to security reasons, outgoing connections using ssh are disabled. On the Linux Cluster, this applies for the default port 22, on SuperMUC-NG outgoing connections are more generally disabled.


Configuration of SSH public-key authentication for access to LRZ systems from outside

The variant described here deploys the portable openssh implementation for this purpose. Before using secure shell, you will need to configure the software. The configuration starts on your own workstation/desktop (aka the client), other items need to be performed on the target LRZ system itself. While the ssh command will also work without these customization steps, usage is made easier by performing it. Here is the sequence of commands you need to perform (using the example of ecdsa); the leftmost column of the table indicates where the respective commands must be executed.

Client (your desktop) is a LINUX system

Where?Commands to executeFurther details

Client system
(your desktop)

LINUX system
ssh-keygen -t ecdsa -b 521 -a 100

Generate an ECDSA key with maximum key size (521 bits) and 100 rounds. The command will respond with

Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/myaccount/.ssh/id_ecdsa)

to which you respond by typing the ENTER key. Next, you are prompted for a passphrase:

Enter passphrase (empty for no passphrase):

Passphrase

You have to respond with a non-empty passphrase (a long sentence, but easy to remember for you).

Once this is done, both the files ~/.ssh/id_ecdsa and ~/.ssh/id_ecdsa.pub (the private and public keys, respectively) exist.

Client system
(your desktop)

LINUX system
ssh-copy-id -i /home/myaccount/.ssh/id_ecdsa <user>@<targetsystem>

Please insert correct LRZ user name and target system name. Enter password when prompted!

The command performs following automatic actions:

  • If not existing, creates ~/.ssh directory on target system.
  • If not existing, creates the file ~/.ssh/authorized_keys on target system.
  • Changes the permissions of your home, ~/.ssh and ~/.ssh/authorized_keys  on target system (removes group writability).
  • Copies the public key  (which is in /home/myaccount/.ssh/id_ecdsa.pub) to the LRZ target system and adds it to list of authorized keys by appending to ~/.ssh/authorized_keys (you may have multiple entries in that file).

If successful, the command's response will contain:

Number of key(s) added: 1

Client (your desktop) is a WINDOWS system

Where?Steps / Commands to executeFurther details
Client system
(your desktop)

Setup SSH and connect to LRZ target system, e. g. using PuTTY, and open a terminal window.

We recommend to install an SSH tool, such as PuTTY or the OpenSSH client to establish ssh connectivity.

Please refer to Getting started with PuTTY and How to Use PuTTY on Windows for details.

Please use correct LRZ user name and target system name.

LRZ
target system

Prepare your home and .ssh directories:

mkdir -p ~/.ssh
chgrp $(id -g -n) ~/.ssh
chmod o-rwx,g-w ~
chmod go-rwx ~/.ssh

command details: 

create .ssh directory if not existing
change group(s) of .ssh directory associated with your user account
change permissions of your home directory
change permissions of .ssh directory

Client system
(your desktop)
Generate SSH keys.

SSH key pairs may be generated via the PuTTY tool puttygen.exe (select pkk file version 2).

Please refer to Using public keys for SSH authentication and PuTTYgen - Key Generator for PuTTY on Windows for details.

Client system
(your desktop)

Copy public SSH key to LRZ target system.

You may use the secure copy command from PuTTY in terminal:

WINDOWS system
pscp c:\path_to_key\name_of_key.pub <user>@<targetsystem>:.ssh/
Please refer to Using PSCP to transfer files securely for details using PuTTY.
LRZ
target system

Append public SSH key to authorized keys:

cd ~/.ssh
cat name_of_key.pub >> authorized_keys
rm name_of_key.pub

command details:

change to .ssh directory
append public key to the end of the file "authorized_keys" (will be created if not existing)
delete public key file, not needed anymore

Per-session activity

At the beginning of your day's session please perform the following commands (LINUX system):

Where?Commands to executeFurther details

Client system

(your desktop)

LINUX system
eval $(ssh-agent -s)

or

LINUX system
eval `ssh-agent -c`
Starts the ssh agent for your shell. The first variant will work for bash/ksh/sh, the second one for csh/tcsh.

Note: modern Linux distributions already do this for you. So you can omit this step if you already see a ssh-agent process running on your system.

Client system

(your desktop)

LINUX system
ssh-add
You will be prompted for the passphrase(s) associated with your key(s). Please enter the value configured by you.

Subsequent ssh commands can then be executed in the same shell without needing to re-authenticate. For other shells, you need to set the environment variables SSH_AUTH_SOCK and SSH_AGENT_PID to the same values as in the shell that started the agent, and export them (this can be automated).

LRZ internal configuration

This section is only relevant for setting ssh up to work without passwords (and without ssh agent) inside the cluster. This is needed for e.g. batch process startup in some commercial applications). None of the keys (public or private) generated here shall be transferred outside the cluster, since this may incur security problems. The following commands must therefore be executed on the LRZ target system.


Where?Commands to executeFurther details

LRZ
target system

(login node)

cd $HOME/.ssh
ssh-keygen -t ecdsa

For example, generate an ECDSA key. The command will respond with

Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/myaccount/.ssh/id_ecdsa)

to which you respond by typing the ENTER key. Next, you are prompted for a passphrase:

Enter passphrase (empty for no passphrase):

to which you should respond by typing ENTER (no passphrase).

cd ~/.ssh
cat id_ecdsa.pub >> authorized_keys
Add internal public key to list of authorized keys.

Special topics

This section discusses special usage patterns that are relevant if a system only allows one-way connections (such as the SuperMUC, which does not allow outgoing traffic). As a consequence, tools (such as distributed source control packages) which use ssh as the underlying transport protocol may fail.

Port forwarding

If outgoing traffic is disallowed for a <PORT>, you can set up port forwarding using an incoming connection e.g.,

ssh -R <NEW_PORT>:<URL>:<PORT> <Your Login>@<Login Node>.lrz.de

the shell generated on the target system will then allow you to specify localhost:<NEW_PORT> in place of the original <URL>:<PORT> because the ssh connection forwards the former to the latter through a tunnel connection.

Note that <NEW_PORT> must not be in use by someone else; you will see a ssh-generated error message if this is the case. If your software stores <NEW_PORT> away somewhere, you will need to do additional configuration changes if you need to change the value of <NEW_PORT>.

If incoming traffic is disallowed for a <PORT>, the following command can be used:

ssh -L <NEW_PORT>:localhost:<PORT> <Your Login>@<Login Node>.lrz.de

On a separate local shell, it is then possible to access the service running on <PORT> of the remote machine via localhost:<NEW_PORT>.

Data access via sshfs

If port forwarding fails and your software only requires access to data on disk, using a ssh-tunnelled mount may be sufficient. The sshfs command, which typically is delivered with modern Linux distributions can be used for this purpose:

sshfs [-o workaround=rename]  <Your Login>@<Login Node>.lrz.de:<Path name> <Mount Point>

will mount a path on the remote system to a local mount point; if the remote path is omitted the HOME directory will be used. File access permissions should be taken over from the remote system. Once the mount is established, you can run your software locally and access data via the mount; note that some packages may need additional workaround options such as the one optionally specified above. You can remove the mount via the command

fusermount -u <Mount Point> 

Troubleshooting and optional features

Login fails

If the secure shell command does not work as expected, please

  • Try a ping command to the target system
  • Try a host command with the target system as the argument (local DNS configuration).
  • Check whether any maintenance measures have been announced for the target system.
  • Check which authentication method is triggered (password or key); a verbosity option (see below) can help with this.
  • Check whether suitably restrictive permissions are set on your $HOME as well as your $HOME/.ssh directory (see above).
  • Finally, try one of the -v, -vv or -vvv option (increasing verbosity) on the ssh command.
  • If you decide to contact the LRZ Service Desk, please supply the output of -vvv.

Connections are disrupted after some time

If ssh-based connections (including sshfs mounts) tend to hang if left unused for a time, adding the option

 ServerAliveInterval 100

to your (workstation-local) .ssh/config file may help.

X11 does not work

If opening new X windows does not work, please check the following:

  • Either, specify the -Y switch in your ssh command line, or, add an entry of the form
     ForwardX11Trusted yes

       
    to your .ssh/config file. This setting assumes that you trust our administrators as well as our security procedures, since technically this opens a tunnel that allows an attack on your X11 display. You can consider using the -X switch (or the ForwardX11 configuration file setting) instead, but a great many X11 clients will not work at all then.

  • If you have a Microsoft Windows client, you need to activate X11 tunnelling in the PuTTY configuration, as well as have a X Window server (e.g. Xming) running locally.
  • If you got an error message like

    error in locking authority file
    /dss/dsshome1/.../.Xauthority
    at login, please check your HOME filesystem quota. If this command indicates an overflow, you need to delete files from your HOME directory. Multiple simultaneous logins to the target system may also provoke the above error message (in this case due to file locking). In the latter case, please exit the failed shell and start a new one.

Optional measure to enhance your security

In order to permit connections from particular sources, each key contained in the authorized keys file on the HPC system may be prepended with a non-trivial from clause. In case of a stolen private key, a third party will not be able to use it to access the HPC system from a different computer. For example, the default entry in authorized_keys

ecdsa-sha2-nistp521 <key value> <user>@<host>

will be changed to:

from="<IP address of your computer>" ecdsa-sha2-nistp521 <key value> <user>@<host>