Page tree
Skip to end of metadata
Go to start of metadata

Please note that the Data Transfer System for SuperMUC-NG is still under development. So this document, as well as the available data transfer options will increase over the next few months. So check back from time to time to see what's new.

Overview

SuperMUC-NG provides a high performance network connection to the outside world. In order to perform easy and fast data staging to/from SuperMUC-NG, we provide various data transfer methods. However, because of limited support resources, there is a distinction on what is available to transfer data to/from arbitrary external sites and to/from our GCS and PRACE partner sites. In the following document, we want to give you the necessary information on using the available data transfer options.

SuperMUC-NG DTN Setup

The diagram below shows the SuperMUC-NG DTN topology. As you can see, we operate currently four Login nodes, two Globus Online and two Special Purpose Data Transfer Nodes (DTNs), each one connected via a 100GE network link to our Data Center Network. The Data Center Network is connected via four 100GE links to the German Research Network (DFN), that are operated in a pair wise active/passive mode. The DFN then provides high bandwidth connections to the internet and via the European Research Network (Geant) to other Research Networks all over the world.

Note that the Data Transfer Nodes (DTNs) only provide server side functionality. Initiating Transfers to/from the DTNs is always done by the user from the Login Nodes or via the Globus Online Webinterface. User Login to the DTNs is not possible.


Data Transfer to/from arbitrary external sites

In the following we will outline the two data transfer methods, we currently support from/to arbitrary external endpoints.

In order to be able to transfer data to/from an arbitrary external site, the IP(s) of the external endpoint(s) have to be registered as trusted IP(s) by your project in the SuperMUC-NG firewall. If not already done so, please ask the Master User of your project to submit a Service Request to register your IP(s) in the SuperMUC-NG firewall.

Globus Online

The preferred data transfer method on SuperMUC-NG to/from arbitrary external sites is to use Globus Online. LRZ operates an dedicated Globus Endpoint for SuperMUC NG called LRZ SuperMUC-NG Globus DTN. In order to get started with Globus Online check out this short tutorial. Note that you can use your SuperMUC-NG credentials to log in to Globus Online and must use it to authenticate against the LRZ SuperMUC-NG Globus DTN endpoint. Just search for LRZ or Leibniz in the list of organisations on the login page(s).

In general Globus Online performs best if you transfer multiple (>8) large (>10GB) files so that it can efficiently parallelise data transfers. This is especially important over long distance links.


If you already used Globus on SuperMUC, you may ask yourself now one or more of the following questions:
Is it really that simple? => Yes. It is!
What about getting certificates and linking them to my user account?  => No more need to hassle with certificates. Your SuperMUC-NG login credentials will be all you need.
What about that complicated globus-url-copy command? => Gone. Check out the new Globus Online CLI. 

Secure Copy

For smaller amounts of data you also can use secure copy (scp), secure ftp (sftp) or RSYNC (rsync) using ssh as remote shell from your external endpoint to the SuperMUC-NG login nodes. Note that the direction from external to SNG is important as the firewall of SuperMUC-NG does not permit outgoing ssh connections.

When choosing this data transfer method, please bear in mind that sshscp and sftp are very limited in the achievable performance, especially over high latency WAN links. See also https://fasterdata.es.net/data-transfer-tools/say-no-to-scp/ on this topic.

Data Transfer to/from GCS sites

The three GCS Sites (HLRS in Stuttgart, JSC in Jülich and LRZ in Garching) operate a network of Data Transfer Nodes (DTNs), that have been optimised to enable you to transfer data with up to 10GB/s between the three German National Flagship Supercomputers. 

UNICORE File Transfer (UFTP)

UFTP is a data streaming library and file transfer tool which is based on a client/server architecture. In order to transfer between JSC, HLRS and LRZ, all three sites operate Unicore authentification services and data access servers (uftpd). To transfer files, the user has to log in to one of the corresponding hosts at the site and then authenticates with a public/private ssh key to the authentication service of another site.

At LRZ, the two authenitification services are located at

https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW
https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW

Setting up the Client

To transfer files with a client at LRZ on SuperMUC-NG to another site, you need to log in to

skx-arch.supermuc.lrz.de

Then you have to load the uftp-client

module use -a /lrz/sys/share/modules/extfiles
module load uftp-client

For more information on the uftp-client, see the examples below or please refer to https://www.unicore.eu/docstore/uftpclient-1.3.2/uftpclient-manual.html

SSH Key Management

In order to use uftp, you need to create and use seperate ssh keys. NOTE: Due to security concerns you are not allowed to use these keys for anything else than uftp transfers, especially not for SSH-based access (including SCP and SFTP) to any system. To create a new ssh key for uftp, use the following commands:

Here the transfer is to JSC, for transfer to HLRS replace "jsc"-> "hlrs"
mkdir -p ~/.uftp
ssh-keygen -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_jsc

and chose a secure(!) passphrase. Keys without passphrase violate the security regulations of LRZ and their use is strictly forbidden.

Transfer between LRZ and JSC

Login to judac.fz-juelich.de and copy the contents of ~/.uftp/id_uftp_to_jsc.pub (located at LRZ) into the file ~/.uftp/authorized_keys (located at JUDAC). Now you should be able to get some informations from the uftp service at JSC by executing on skx-arch

uftp info -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC

where you need to replace YOUR_USERNAME_AT_JSC with your username at JUDAC/JSC. The output should look similar to this:

Client identity: CN=YOUR_USERNAME_AT_JSC , OU=ssh-local-users
Client auth method: SSHKEY
Auth server type: AuthServer
Server: JUDAC
URL base: https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:
Description: JUDAC
Remote user info: uid=YOUR_USERNAME_AT_JSC ;gid=N/A
Sharing support: enabled
Server status: OK [connected to UFTPD judacsrv.fz-juelich.de:64433]
Server: JUDAC-PRACE
URL base: https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC-PRACE:
Description: JUDAC via PRACE Network
Remote user info: uid= YOUR_USERNAME_AT_JSC;gid=N/A
Sharing support: not available
Server status: OK [connected to UFTPD judacsrv.fz-juelich.de:64433]

If you receive a warning like

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/dss/dsshome1/lrz/sys/grid/uftp-client-1.3.2/lib/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

this can be SAFELY IGNORED and will disappear with never versions.


To list the contents of a remote directory, use

uftp ls -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/AT/JUDAC

To download a file from JUDAC to LRZ:

uftp cp -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC /LOCAL/PATH/AT/LRZ

To upload a file from LRZ to JUDAC, you just need to reverse the order of the last two arguments:

uftp cp -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC /LOCAL/PATH/AT/LRZ https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC 

You can also use more streams to potentially speed up the transfer using the option "-t 10":

uftp cp -t 10 -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC /LOCAL/PATH/AT/LRZ https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC 

Transfer between LRZ and JSC (with client at JUDAC)

To use the client at JUDAC, please also refer to https://apps.fz-juelich.de/jsc/hps/judac/uftp.html . The procedure to enable access to LRZ is similar to the one in the reverse direction:

At JUDAC create a ssh key with passphrase using

mkdir ~/.uftp
ssh-keygen -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_lrz

At LRZ copy the contents of ~/.uftp/id_uftp_to_lrz.pub (located at JUDAC) into the file ~/.uftp/authorized_keys (located at LRZ) and replace the authentification URL https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC with https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW or https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW, so e.g.

uftp cp -t 10 -i ~/.uftp/id_uftp_to_lrz -u YOUR_USERNAME_AT_LRZ /LOCAL/PATH/AT/JUDAC  https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW:/PATH/TO/FILE/AT/LRZ

Transfer between LRZ and HLRS

To transfer from/to HLRS, create another(!) ssh key at skx-arch:

mkdir -p ~/.uftp
ssh-keygen -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_hlrs

To enable the public key for transfer, open a service request at HLRS and tell them your public key (the contents of ~/.uftp/id_uftp_to_hlrs.pub) and your username and ask them to enable your key for uftp. A sample command on skx-arch then should look like


uftp cp -t 10 -i ~/.uftp/id_uftp_to_hlrs -u YOUR_USERNAME_AT_HLRS /LOCAL/PATH/AT/LRZ https://gridftp-fr1.hww.hlrs.de:9000/rest/auth/HLRS:/PATH/TO/FILE/AT/HLRS 



Grid Community Toolkit (GridFTP)

Data Transfer to/from PRACE sites

Coming Soon.

Grid Community Toolkit (GridFTP)

Beyond Data Transfer: Sharing and Public Access

If you want to share data generated on SuperMUC-NG with external parties or even make this data publicly available, you can have a look at our LRZ Data Science Storage (DSS). GCS has funded 20PB of DSS storage for SuperMUC-NG. For details on how SuperMUC-NG projects can apply for storage space on DSS see: Data Science Storage for SuperMUC.


  • No labels