Please note that the Data Transfer System for SuperMUC-NG is still under development. So this document, as well as the available data transfer options will increase over the next few months. So check back from time to time to see what's new.
SuperMUC-NG provides a high performance network connection to the outside world. In order to perform easy and fast data staging to/from SuperMUC-NG, we provide various data transfer methods. However, because of limited support resources, there is a distinction on what is available to transfer data to/from arbitrary external sites and to/from our GCS and PRACE partner sites. In the following document, we want to give you the necessary information on using the available data transfer options.
SuperMUC-NG DTN Setup
The diagram below shows the SuperMUC-NG DTN topology. As you can see, we operate currently four Login nodes, two Globus Online and two Special Purpose Data Transfer Nodes (DTNs), each one connected via a 100GE network link to our Data Center Network. The Data Center Network is connected via four 100GE links to the German Research Network (DFN), that are operated in a pair wise active/passive mode. The DFN then provides high bandwidth connections to the internet and via the European Research Network (Geant) to other Research Networks all over the world.
Note that the Data Transfer Nodes (DTNs) only provide server side functionality. Initiating Transfers to/from the DTNs is always done by the user from the Login Nodes or via the Globus Online Webinterface. User Login to the DTNs is not possible.
Data Transfer to/from arbitrary external sites
In the following we will outline the two data transfer methods, we currently support from/to arbitrary external endpoints.
In order to be able to transfer data to/from an arbitrary external site, the IP(s) of the external endpoint(s) have to be registered as trusted IP(s) by your project in the SuperMUC-NG firewall. If not already done so, please ask the Master User of your project to submit a Service Request to register your IP(s) in the SuperMUC-NG firewall.
The preferred data transfer method on SuperMUC-NG to/from arbitrary external sites is to use Globus Online. LRZ operates an dedicated Globus Endpoint for SuperMUC NG called LRZ SuperMUC-NG Globus DTN. In order to get started with Globus Online check out this short tutorial. Note that you can use your SuperMUC-NG credentials to log in to Globus Online and must use it to authenticate against the LRZ SuperMUC-NG Globus DTN endpoint. Just search for LRZ or Leibniz in the list of organisations on the login page(s).
In general Globus Online performs best if you transfer multiple (>8) large (>10GB) files so that it can efficiently parallelise data transfers. This is especially important over long distance links.
If you already used Globus on SuperMUC, you may ask yourself now one or more of the following questions:
Is it really that simple? => Yes. It is!
What about getting certificates and linking them to my user account? => No more need to hassle with certificates. Your SuperMUC-NG login credentials will be all you need.
What about that complicated globus-url-copy command? => Gone. Check out the new Globus Online CLI.
For smaller amounts of data you also can use secure copy (
scp), secure ftp (
sftp) or RSYNC (
ssh as remote shell from your external endpoint to the SuperMUC-NG login nodes. Note that the direction from external to SNG is important as the firewall of SuperMUC-NG does not permit outgoing ssh connections.
When choosing this data transfer method, please bear in mind that
sftp are very limited in the achievable performance, especially over high latency WAN links. See also https://fasterdata.es.net/data-transfer-tools/say-no-to-scp/ on this topic.
Data Transfer to/from GCS sites
The three GCS Sites (HLRS in Stuttgart, JSC in Jülich and LRZ in Garching) operate a network of Data Transfer Nodes (DTNs), that have been optimised to enable you to transfer data with up to 10GB/s between the three German National Flagship Supercomputers.
UNICORE File Transfer (UFTP)
UFTP is a data streaming library and file transfer tool which is based on a client/server architecture. In order to transfer between JSC, HLRS and LRZ, all three sites operate Unicore authentification services and data access servers (
uftpd). To transfer files, the user has to log in to one of the corresponding hosts at the site and then authenticates with a public/private ssh key to the authentication service of another site.
At LRZ, the two authenitification services are located at
Setting up the Client
To transfer files with a client at LRZ on SuperMUC-NG to another site, you need to log in to
Then you have to load the uftp-client
For more information on the uftp-client, see the examples below or please refer to https://www.unicore.eu/docstore/uftpclient-1.3.2/uftpclient-manual.html
SSH Key Management
In order to use uftp, you need to create and use seperate ssh keys. NOTE: Due to security concerns you are not allowed to use these keys for anything else than uftp transfers, especially not for SSH-based access (including SCP and SFTP) to any system. To create a new ssh key for uftp, use the following commands:
and chose a secure(!) passphrase. Keys without passphrase violate the security regulations of LRZ and their use is strictly forbidden.
Transfer between LRZ and JSC
judac.fz-juelich.de and copy the contents of
~/.uftp/id_uftp_to_jsc.pub (located at LRZ) into the file
~/.uftp/authorized_keys (located at JUDAC). Now you should be able to get some informations from the uftp service at JSC by executing on
where you need to replace YOUR_USERNAME_AT_JSC with your username at JUDAC/JSC. The output should look similar to this:
If you receive a warning like
this can be SAFELY IGNORED and will disappear with never versions.
To list the contents of a remote directory, use
To download a file from JUDAC to LRZ:
To upload a file from LRZ to JUDAC, you just need to reverse the order of the last two arguments:
You can also use more streams to potentially speed up the transfer using the option "-t 10":
Transfer between LRZ and JSC (with client at JUDAC)
To use the client at JUDAC, please also refer to https://apps.fz-juelich.de/jsc/hps/judac/uftp.html . The procedure to enable access to LRZ is similar to the one in the reverse direction:
At JUDAC create a ssh key with passphrase using
At LRZ copy the contents of
~/.uftp/id_uftp_to_lrz.pub (located at JUDAC) into the file
~/.uftp/authorized_keys (located at LRZ) and replace the authentification URL https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC with https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW or https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW, so e.g.
Transfer between LRZ and HLRS
To transfer from/to HLRS, create another(!) ssh key at
To enable the public key for transfer, open a service request at HLRS and tell them your public key (the contents of
~/.uftp/id_uftp_to_hlrs.pub) and your username and ask them to enable your key for uftp. A sample command on skx-arch then should look like
Grid Community Toolkit (GridFTP)
Data Transfer to/from PRACE sites
Grid Community Toolkit (GridFTP)
Beyond Data Transfer: Sharing and Public Access
If you want to share data generated on SuperMUC-NG with external parties or even make this data publicly available, you can have a look at our LRZ Data Science Storage (DSS). GCS has funded 20PB of DSS storage for SuperMUC-NG. For details on how SuperMUC-NG projects can apply for storage space on DSS see: Data Science Storage for SuperMUC.