Page tree
Skip to end of metadata
Go to start of metadata

Introduction

LRZ offers two options to access the HPC systems from outside via SSH ("secure shell"):

(1) Password authentication using user name and password which was set in the LRZ IDM Portal. How to do this is described in the section "Basic usage" below.

OR

(2) Public/private key encrypted communication for advanced users with additional requirements.

The following features of secure shell are relevant for users of LRZ HPC systems:

  • encrypted transfer of passwords and data
  • tunneling of services (in particular X11 graphics display) over the secure shell network port
  • usage of the version 2 protocol of ssh. Version 1 is not supported at LRZ, since not sufficiently secure.

Secure shell has superseded other, insecure protocols like rsh and telnet, which are not supported anymore at LRZ.

Policies for SSH public-key authentication

Passphrases must not be empty

When a key pair is generated (see below), the generator asks for a passphrase. It is not allowed to use an empty passphrase for keypairs that store private and public keys, respectively, on different systems. We consider an empty passphrase as a violation of elementary security policies and will shut out any user doing so from further usage of LRZ systems.

The only exception allowed is for keypairs that are only internally used, and if additional protective steps are taken to disable authentication from outside systems (see limitations on authorized keys below).

Limitations on encryption algorithms

On LRZ HPC systems, authentication by key pair will only be accepted if one of the following algorithms was used to generate key pairs:

  1. ECDSA
  2. ED25519 

We only accept the maximum key length of 521 bits (ECDSA only) and highly recommend to set 100 rounds (ECDSA and ED25519, increasing efforts to crack passwords of stolen keys). See How-to-do details in configuration section below.

Limitations on authorized keys

Restrictions apply to the file "authorized_keys". Any key which violates the following rules will be forcibly disabled by an automated procedure that is regularly executed.


In order to permit connections from particular sources, each key contained in the authorized keys file must be prepended with a non-trivial from clause. See How-to-do details in configuration section below. The following guidelines apply:


from clauseCompliance
with rule
Remarks
from="172.16.0.1"
(tick)IP addresses must be used.
from="hostname.de"
(error)
from="172.16.0.?"
(tick)It is permitted to supply a subnet using at most one wildcard!
The use of wildcards "?" or "*" is restricted to the last octet of the IP address.
"?" replaces a single digit, e. g. 172.16.0.? represents the IP address range 172.16.0.[0-9].
"*" replaces a multiple-digit number, e. g. 172.16.0.* represents the IP address range 172.16.0.[0-255].
from="172.16.0.*"
(tick)
from="172.16.*.110"
(error)
from="172.16.*.*"
(error)
from="*"
(error)
from="172.16.0.?,172.16.0.110"
(tick)It is permitted to specify a list of entries without whitespaces.
from="172.16.0.110,!172.16.0.150"
(tick)Allows connections from 172.16.0.110. The exclamation mark "!" disallows connections from a source.
from="172.16.0.0/24"
(error)CIDR notation of IP addresses is not supported.
from="*.*.vpn.lrz.de"
(tick)Exception:
Use this pattern when connected to LRZ VPN. You do not need to add the dedicated VPN IP address which has been assigned to your VPN connection.



Each key must be generated by a supported encryption algorithm.

Outgoing connections

Outgoing connections using ssh are disabled. On the Linux Cluster, this applies for the default port 22, on SuperMUC-NG outgoing connections are more generally disabled.

Usage

Basic usage

Basic usage of ssh is very simple: Issue the command

ssh -Y -l <LRZ account> <target LRZ system> 


If you have not (yet) configured ssh for public-key authentication, you will be asked for your SIM password. Otherwise, you will be asked for your ssh key passphrase.

If the authentication is successfully concluded, you have a login shell on the target system. The target system typically must be specified with its full domain name.


For additional security we provide the public system keys for SuperMUC-NG and Linux Cluster login nodes which can be added into the ~/.ssh/known_hosts file (you may need to generate the .ssh directory if it does not exist yet) on your local machine.


A secure copy mechanism is also provided:

scp <my file> <LRZ account>@<target LRZ system>:<remote directory>

will, for example, copy the local file <my file> to the directory ~/<remote directory> on the remote system. This directory must exist prior to the copy attempt.

Installation on client system (your desktop)

While secure shell is installed by default on LRZ systems, it may be necessary to install it on your system.

  • Linux Systems: Most distributions contain an ssh client, but it may not have been installed. Please use the distribution-specific installation program to install the client
  • Windows Systems: Various free implementations are available on the internet, for example PuTTY. An alternative is to install the Windows subsystem for Linux (WSL).
  • If you have a C compiler on your system, you can also install e.g. portable openssh from source.

Configuration of SSH public-key authentication for access to LRZ systems from outside

The variant described here deploys the portable openssh implementation for this purpose. Before using secure shell, you will need to configure the software. The configuration starts on your own workstation/desktop (aka the client), other items need to be performed on the target LRZ system itself. While the ssh command will also work without these customization steps, usage is made easier by performing it. Here is the sequence of commands you need to perform (using the example of ecdsa); the leftmost column of the table indicates where the respective commands must be executed.

Client (your desktop) is a LINUX system

Where?Commands to executeFurther details

Client system
(your desktop)

LINUX system
ssh-keygen -t ecdsa -b 521 -a 100

Generate an ECDSA key with maximum key size (521 bits) and 100 rounds. The command will respond with

Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/myaccount/.ssh/id_ecdsa)

to which you respond by typing the ENTER key. Next, you are prompted for a passphrase:

Enter passphrase (empty for no passphrase):

Passphrase

You have to respond with a non-empty passphrase (a long sentence, but easy to remember for you).

Once this is done, both the files ~/.ssh/id_ecdsa and ~/.ssh/id_ecdsa.pub (the private and public keys, respectively) exist.

Client system
(your desktop)

LINUX system
ssh-copy-id -i /home/myaccount/.ssh/id_ecdsa <user>@<targetsystem>

Please insert correct LRZ user name and target system name. Enter password when prompted!

The command performs following automatic actions:

  • If not existing, creates ~/.ssh directory on target system.
  • If not existing, creates the file ~/.ssh/authorized_keys on target system.
  • Changes the permissions of your home, ~/.ssh and ~/.ssh/authorized_keys  on target system (removes group writability).
  • Copies the public key  (which is in /home/myaccount/.ssh/id_ecdsa.pub) to the LRZ target system and adds it to list of authorized keys by appending to ~/.ssh/authorized_keys (you may have multiple entries in that file).

If successful, the command's response will contain:

Number of key(s) added: 1
LRZ
target system
vi ~/.ssh/authorized_keys

Edit the new authorized key entry to contain a from clause. For example, if you want to be able to log in from your workstation with IP 129.166.15.81, the line for the corresponding public key would look as follows:

from="129.166.15.81" ecdsa-sha2-nistp521 <key value> <user>@<host>

where placeholders are given for the actual key value, user name, and host on which the private key was generated.

Client (your desktop) is a WINDOWS system

Where?Steps / Commands to executeFurther details
Client system
(your desktop)

Setup SSH and connect to LRZ target system, e. g. using PuTTY, and open a terminal window.

We recommend to install an SSH tool, such as PuTTY or the OpenSSH client to establish ssh connectivity.

Please refer to Getting started with PuTTY and How to Use PuTTY on Windows for details.

Please use correct LRZ user name and target system name.

LRZ
target system

Prepare your home and .ssh directories:

mkdir -p ~/.ssh
chgrp $(id -g -n) ~/.ssh
chmod o-rwx,g-w ~
chmod go-rwx ~/.ssh

command details: 

create .ssh directory if not existing
change group(s) of .ssh directory associated with your user account
change permissions of your home directory
change permissions of .ssh directory

Client system
(your desktop)
Generate SSH keys.

SSH key pairs may be generated via the PuTTY tool puttygen.exe.

Please refer to Using public keys for SSH authentication and PuTTYgen - Key Generator for PuTTY on Windows for details.

Client system
(your desktop)

Copy public SSH key to LRZ target system.

You may use the secure copy command from PuTTY in terminal:

WINDOWS system
pscp c:\path_to_key\name_of_key.pub <user>@<targetsystem>:.ssh/
Please refer to Using PSCP to transfer files securely for details using PuTTY.
LRZ
target system

Append public SSH key to authorized keys:

cd ~/.ssh
cat name_of_key.pub >> authorized_keys
rm name_of_key.pub

command details:

change to .ssh directory
append public key to the end of the file "authorized_keys" (will be created if not existing)
delete public key file, not needed anymore

LRZ
target system
vi ~/.ssh/authorized_keys

Edit the new authorized key entry to contain a from clause. For example, if you want to be able to log in from your workstation with IP 129.166.15.81, the line for the corresponding public key would look as follows:

from="129.166.15.81" ecdsa-sha2-nistp521 <key value> <user>@<host>

where placeholders are given for the actual key value, user name, and host on which the private key was generated.

Per-session activity

At the beginning of your day's session please perform the following commands (LINUX system):

Where?Commands to executeFurther details

Client system

(your desktop)

LINUX system
eval $(ssh-agent -s)

or

LINUX system
eval `ssh-agent -c`
Starts the ssh agent for your shell. The first variant will work for bash/ksh/sh, the second one for csh/tcsh.

Note: modern Linux distributions already do this for you. So you can omit this step if you already see a ssh-agent process running on your system.

Client system

(your desktop)

LINUX system
ssh-add
You will be prompted for the passphrase(s) associated with your key(s). Please enter the value configured by you.

Subsequent ssh commands can then be executed in the same shell without needing to re-authenticate. For other shells, you need to set the environment variables SSH_AUTH_SOCK and SSH_AGENT_PID to the same values as in the shell that started the agent, and export them (this can be automated).

LRZ internal configuration

This section is only relevant for setting ssh up to work without passwords (and without ssh agent) inside the cluster. This is needed for e.g. batch process startup in some commercial applications). None of the keys (public or private) generated here shall be transferred outside the cluster, since this may incur security problems. The following commands must therefore be executed on the LRZ target system.


Where?Commands to executeFurther details

LRZ
target system

(login node)

ssh-keygen -t ecdsa

For example, generate an ECDSA key. The command will respond with

Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/myaccount/.ssh/id_ecdsa)

to which you respond by typing the ENTER key. Next, you are prompted for a passphrase:

Enter passphrase (empty for no passphrase):

to which you should respond by typing ENTER (no passphrase).

cd ~/.ssh
cat id_ecdsa.pub >> authorized_keys
Add internal public key to list of authorized keys.
vi ~/.ssh/authorized_keys

Edit each entry in authorized_keys to contain a from clause as described above:

from="list-of-IP-addresses" ecdsa-sha2-nistp521 <key value> <user>@<host>

For usage of the Linux Cluster, the lists of IP addresses need to look as follows:

Cluster segmentReplace "list-of-IP-addresses" with ...

cm2
cm2_tiny
serial
inter
Housing Cluster

IP list
"10.7.1.*,10.7.2.*,10.7.3.*,10.7.4.*,10.7.5.*,10.7.6.*,10.4.12.*,10.4.13.*,10.4.14.*,10.4.15.*,10.4.16.*,10.4.17.*,10.4.18.*"
mpp3

Special topics

This section discusses special usage patterns that are relevant if a system only allows one-way connections (such as the SuperMUC, which does not allow outgoing traffic). As a consequence, tools (such as distributed source control packages) which use ssh as the underlying transport protocol may fail.

Port forwarding

If outgoing traffic is disallowed for a <PORT>, you can set up port forwarding using an incoming connection e.g.,

ssh -R <NEW_PORT>:<URL>:<PORT> <Your Login>@<Login Node>.lrz.de

the shell generated on the target system will then allow you to specify localhost:<NEW_PORT> in place of the original <URL>:<PORT> because the ssh connection forwards the former to the latter through a tunnel connection.

Note that <NEW_PORT> must not be in use by someone else; you will see a ssh-generated error message if this is the case. If your software stores <NEW_PORT> away somewhere, you will need to do additional configuration changes if you need to change the value of <NEW_PORT>.

If incoming traffic is disallowed for a <PORT>, the following command can be used:

ssh -L <NEW_PORT>:localhost:<PORT> <Your Login>@<Login Node>.lrz.de

On a separate local shell, it is then possible to access the service running on <PORT> of the remote machine via localhost:<NEW_PORT>.

Data access via sshfs

If port forwarding fails and your software only requires access to data on disk, using a ssh-tunnelled mount may be sufficient. The sshfs command, which typically is delivered with modern Linux distributions can be used for this purpose:

sshfs [-o workaround=rename]  <Your Login>@<Login Node>.lrz.de:<Path name> <Mount Point>

will mount a path on the remote system to a local mount point; if the remote path is omitted the HOME directory will be used. File access permissions should be taken over from the remote system. Once the mount is established, you can run your software locally and access data via the mount; note that some packages may need additional workaround options such as the one optionally specified above. You can remove the mount via the command

fusermount -u <Mount Point> 

Troubleshooting

Login fails

If the secure shell command does not work as expected, please

  • Try a ping command to the target system
  • Try a host command with the target system as the argument (local DNS configuration).
  • Check whether any maintenance measures have been announced for the target system.
  • Check which authentication method is triggered (password or key); a verbosity option (see below) can help with this.
  • Check whether suitably restrictive permissions are set on your $HOME as well as your $HOME/.ssh directory (see above).
  • Finally, try one of the -v, -vv or -vvv option (increasing verbosity) on the ssh command.
  • If you decide to contact the LRZ Service Desk, please supply the output of -vvv.

I use public-key authentication and my authorized_keys file was unexpectedly modified by LRZ

Observation

Shortly after adding a key to the ~/.ssh/authorized_keys file, the public-key authentication does not work anymore. Instead, I have been asked to enter my password at login. Inspecting the file reveals that the key in question has been commented out.

ExplanationLRZ reserves the right to modify user's authorized_keys file by disabling keys which are not compliant to our rules. See section on policies for public-key authentication (on this page).
Example
authorized_keys file provided by the user
# my comment: For Connection from my host 172.16.0.1
from="172.16.0.1" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# When I connect via LRZ VPN
from="*.*.vpn.lrz.de ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
#
# Some other keys
ssh-rsa THISISAVERYLONGPUBLICKEY user@mycomputer
from="172.16.0.1" ssh-rsa THISISAVERYLONGPUBLICKEY user@mycomputer
ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
from="*" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
from="172.16.?.1" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
from="172.16.*.*" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
from="hostname.de" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
authorized_keys file audited by LRZ
# my comment: For Connection from my host 172.16.0.1
from="172.16.0.1" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# When I connect via LRZ VPN
from="*.*.vpn.lrz.de ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
#
# Some other keys
# LRZ: commented the next line because does not respect limited format.
#ssh-rsa THISISAVERYLONGPUBLICKEY user@mycomputer
# LRZ: commented the next line because does not respect limited format.
#from="172.16.0.1" ssh-rsa THISISAVERYLONGPUBLICKEY user@mycomputer
# LRZ: commented the next line because does not respect limited format.
#ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# LRZ: commented the next line because of the wildcard in: *
#from="*" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# LRZ: commented the next line because of the wildcard in: 172.16.?.1
#from="172.16.?.1" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# LRZ: commented the next line because of the wildcard in: 172.16.*.*
#from="172.16.*.*" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycomputer
# LRZ: commented the next line because of using hostname(s): hostname.de
#from="hostname.de" ecdsa-sha2-nistp521 THISISAVERYLONGPUBLICKEY== user@mycompute
# LRZ: Unfortunaly this file violated our policy (7 times), so it has been sanitized
# by an automated script
# (please see https://doku.lrz.de/display/PUBLIC/ssh+-+Secure+Shell+on+LRZ+HPC+Systems).

Note: Due to continuous development of the automated procedure, the wording in the LRZ comments of this example and your file might slightly differ.

Self help

You may check the public keys in your authorized_keys file on your own. LRZ will provide a command line tool soon.

Connections are disrupted after some time

If ssh-based connections (including sshfs mounts) tend to hang if left unused for a time, adding the option

 ServerAliveInterval 100

to your (workstation-local) .ssh/config file may help.

X11 does not work

If opening new X windows does not work, please check the following:

  • Either, specify the -Y switch in your ssh command line, or, add an entry of the form
     ForwardX11Trusted yes

       
    to your .ssh/config file. This setting assumes that you trust our administrators as well as our security procedures, since technically this opens a tunnel that allows an attack on your X11 display. You can consider using the -X switch (or the ForwardX11 configuration file setting) instead, but a great many X11 clients will not work at all then.

  • If you have a Microsoft Windows client, you need to activate X11 tunnelling in the PuTTY configuration, as well as have a X Window server (e.g. Xming) running locally.
  • If you got an error message like

    error in locking authority file
    /dss/dsshome1/.../.Xauthority
    at login, please check your HOME filesystem quota. If this command indicates an overflow, you need to delete files from your HOME directory. Multiple simultaneous logins to the target system may also provoke the above error message (in this case due to file locking). In the latter case, please exit the failed shell and start a new one.
  • No labels