...

Warnung
In order to be able to transfer data to/from an arbitrary external site, the IP(s) of the external endpoint(s) have to be registered as trusted IP(s) by your project in the SuperMUC-NG firewall. If not already done so, please ask the Master User of your project to submit a Service Request to register your IP(s) in the SuperMUC-NG firewall.

...

Codeblock

language	text

https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW
https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW

which can be both reached via the alias:

Codeblock

language	text

https://datagw.supermuc.lrz.de:9000/rest/auth/DATAGW

Setting up the Client

To transfer files with a client at LRZ on SuperMUC-NG to another site, you need to log in to

...

Codeblock

language	text

module use -a /lrz/sys/share/modules/extfiles
module load uftp-client

For more information on the uftp-client, see the examples below or please refer to https://www.unicore.eu/docstore/uftpclient-1.3.2/uftpclient-manual.html

...

and chose a secure(!) passphrase. Keys without passphrase violate the security regulations of LRZ and their use is strictly forbidden.

Transfer between LRZ and JSC

Login to judac.fz-juelich.de and copy the contents of ~/.uftp/id_uftp_to_jsc.pub (located at LRZ) into the file ~/.uftp/authorized_keys (located at JUDAC). Now you should be able to get some informations from the uftp service at JSC by executing on skx-arch

...

language	text

...

Encrypted transfers

Hinweis
For performance reasons all data transfers are NOT encrypted by default (but authentication is, of course). If you want your transfer to be completely encrypted, use the additional command line argument "-E" or "--encrypt" in the copy command (see also "uftp cp --help"). But be aware that this can have massive impact on the transfer speed.

Transfer between LRZ and JSC

Login to judac.fz-juelich.de and copy the contents of ~/.uftp/id_uftp_to_jsc

...

where you need to replace YOUR_USERNAME_AT_JSC with your username at JUDAC/JSC. The output should look similar to this:

.pub (located at LRZ) into the file ~/.uftp/authorized_keys (located at JUDAC). Now you should be able to get some informations from the uftp service at JSC by executing on skx-arch

Codeblock

language	text

Client identity: CN=uftp info -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC , OU=ssh-local-users
Client auth method: SSHKEY
Auth server type: AuthServer
Server: JUDAC
URL base: https://uftp.fz-juelich.de:9112/UFTP_https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:
Description: JUDAC
Remote user info: uid=

where you need to replace YOUR_USERNAME_AT_JSC

...

with your username at JUDAC/JSC. The output should look similar to this:

Codeblock

language	text

Client identity: CN=YOUR_USERNAME_AT_JSC , OU=ssh-local-users
Client auth method: SSHKEY
Auth server type: AuthServer
Server: JUDAC-PRACE
URL base: https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC-PRACE:
Description: JUDAC via PRACE Network
Remote user info: uid= YOUR_USERNAME_AT_JSC ;gid=N/A
Sharing support: not availableenabled
Server status: OK [connected to UFTPD judacsrv.fz-juelich.de:64433]

Hinweis

If you receive a warning like

Codeblock

language	text

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/dss/dsshome1/lrz/sys/grid/uftp-client-1.3.2/lib/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

this can be SAFELY IGNORED and will disappear with never versions.

To list the contents of a remote directory, use

Codeblock

language	text

uftp ls -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/AT/JUDAC

...

Server: JUDAC-PRACE
URL base: https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC-PRACE:
Description: JUDAC via PRACE Network
Remote user info: uid= YOUR_USERNAME_AT_JSC;gid=N/A
Sharing support: not available
Server status: OK [connected to UFTPD judacsrv.fz-juelich.de:64433]

To list the contents of a remote directory, use

Codeblock

language	text

uftp ls -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/AT/JUDAC

To download a file from JUDAC to LRZ:

Codeblock

language	text

uftp cp -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC /LOCAL/PATH/AT/LRZ

To upload a file from LRZ to JUDAC, you just need to reverse the order of the last two arguments:

Codeblock

language	text

uftp cp -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC /LOCAL/PATH/AT/LRZ https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC /LOCAL/PATH/AT/LRZ

...

You can also use more streams to potentially speed up the transfer using the option "-t 10":

Codeblock

language	text

uftp cp -t 10 -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC /LOCAL/PATH/AT/LRZ https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC

You can also use more streams to potentially speed up the transfer using the option "-t 10":

Codeblock

language	text

uftp cp -t 10 -i ~/.uftp/id_uftp_to_jsc -u YOUR_USERNAME_AT_JSC /LOCAL/PATH/AT/LRZ https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC:/PATH/TO/FILE/AT/JUDAC

Transfer between LRZ and JSC (with client at JUDAC)

...

Transfer between LRZ and JSC (with client at JUDAC)

To use the client at JUDAC, please also refer to https://apps.fz-juelich.de/jsc/hps/judac/uftp.html . The procedure to enable access to LRZ is similar to the one in the reverse direction:

At JUDAC create a ssh key with passphrase using

Codeblock

language	text

mkdir ~/.uftp
ssh-keygen -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_lrz

At LRZ copy the contents of ~/.uftp/id_uftp_to_lrz.pub (located at JUDAC) into the file ~/.uftp/authorized_keys (located at LRZ) and replace the authentification URL https://uftp.fz-juelich.de:9112/jscUFTP_Auth/hpsrest/judac/uftp.html . The procedure to enable access to LRZ is similar to the one in the reverse direction:

At JUDAC create a ssh key with passphrase using

...

language	text

...

auth/JUDAC with https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW or https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW, so e.g.

Codeblock

language	text

uftp cp -t 10 -i ~/.uftp/id_uftp_to_lrz

At LRZ copy the contents of ~/.uftp/id_uftp_to_lrz.pub (located at JUDAC) into the file ~/.ssh/authorized_keys (located at LRZ) and replace the authentification URL https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/JUDAC with https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW or https://datagw04.supermuc.lrz.de:9000/rest/auth/DATAGW, so e.g.

Codeblock

language	text

uftp cp -t 10 -i -u YOUR_USERNAME_AT_LRZ /LOCAL/PATH/AT/JUDAC  https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW:/PATH/TO/FILE/AT/LRZ

Transfer between LRZ and HLRS

To transfer from/to HLRS, create another(!) ssh key at skx-arch:

Codeblock

language	text

mkdir -p ~/.uftp
ssh-keygen -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_lrz -u YOUR_USERNAME_AT_LRZ /LOCAL/PATH/AT/JUDAC  https://datagw03.supermuc.lrz.de:9000/rest/auth/DATAGW:/PATH/TO/FILE/AT/LRZ

Transfer between LRZ and HLRS

...

hlrs

To enable the public key for transfer, open a service request at HLRS and tell them your public key (the contents of ~/.uftp/id_uftp_to_hlrs.pub) and your username and ask them to enable your key for uftp. A sample command on skx-arch then should look like

Codeblock

language	text

mkdiruftp cp -t 10 -pi ~/.uftp
ssh-keygen/id_uftp_to_hlrs -a 100 -t ed25519 -f ~/.uftp/id_uftp_to_hlrs

To enable the public key for transfer, open a service request at HLRS and tell them your public key (the contents of ~/.uftp/id_uftp_to_hlrs.pub) and your username and ask them to enable your key for uftp. A sample command on skx-arch then should look like

Codeblock

language	text

uftp cp -t 10 -i ~/.uftp/id_uftp_to_hlrs -u YOUR_USERNAME_AT_HLRS /LOCAL/PATH/AT/LRZ https://gridftp-u YOUR_USERNAME_AT_HLRS /LOCAL/PATH/AT/LRZ https://gridftp-fr1.hww.hlrs.de:9000/rest/auth/HLRS:/PATH/TO/FILE/AT/HLRS

...

Codeblock
gsiftp://datagw03.supermuc.lrz.de gsiftp://datagw04.supermuc.lrz.de

which can be both reached via the alias:

Codeblock
gsiftp://datagw.supermuc.lrz.de

Associate your DN from your personal certificate with your LRZ-username

In the following we assume that you successfully obtained your signed certificate as a .p12 file which is called "SignedGridCert.p12". As a next step, you need to extract you your DN (Distingiushed Name) from the certificate. This can be done via

Codeblock

language	text

openssl pkcs12 -in SignedGridCert.p12 -nodes | openssl x509 -noout -subject
Enter Import Password:
subject=C = DE, O = GridGermany, OU = Leibniz-Rechenzentrum, CN = John Doe -nameopt RFC2253 | sed s/"subject="//
Enter Import Password: 
CN=John Doe,OU=Leibniz-Rechenzentrum,O=GridGermany,C=DE

Afterwards, please follow the instructions on https://www.lrz.de/services/compute/grid_en/certificate_en/person-certificate_en/register_cert_en/ to associate your DN with your LRZ-Account.

Until the association becomes valid it may take up to thirty minutes.

Hinweis
Please note the reverse order in the DN. From the example above, the DN you need to enter into the IDM portal would be `CN=John Doe,OU=Leibniz-Rechenzentrum,O=GridGermany,C=DE`
Hinweis
To use GridFTP with HLRS and To use GridFTP with HLRS and JSC you also need to associate your DN with your corresponding usernames at these sites. For JSC, this can be done in https://judoor.fz-juelich.de/ under "Change data", for HLRS you need to contact the colleagues directly.

...

Then you need to load the GridFTP module:

Codeblock
module useload gridftp-a /lrz/sys/share/modules/extfiles module load gridftp-client

Now you need to generate a proxy certificate with a limited lifetime. This is done via

Codeblock
grid-proxy-init

...

client

Now you need to generate a proxy certificate with a limited lifetime. This is done via

Codeblock
grid-proxy-init

After entering your passphrase you should have obtained a proxy certificate which is valid for several hours.

Encrypted transfers

Hinweis
For performance reasons all data transfers are NOT encrypted by default (but authentication is, of course). Integrity protection and encryption are optional. To integrity protect the data, use the "-dcsafe" option in the copy command. For encrypted data transfer, use the "-dcpriv" option (see also "globus-url-copy -help"). But be aware that this can have massive impact on the transfer speed.

Copying data between the sites

...

Codeblock
globus-url-copy -vb -p 6 gsiftp://datagw03datagw.supermuc.lrz.de/PATH/TO/FILE/AT/LRZ gsiftp://judacsrv.fz-juelich.de/PATH/TO/FILE/AT/JSC

...

Codeblock
globus-url-copy -vb -p 6 gsiftp://gridftp-fr1.hww.de:2812/PATH/TO/FILE/AT/HLRS gsiftp://datagw04datagw.supermuc.lrz.de/PATH/TO/FILE/AT/LRZ

...

Data Transfer to/from PRACE sites

Hinweis
Coming Soon.

...

Beyond Data Transfer: Sharing and Public Access

...

Page tree

Versions Compared

Old Version 14550

New Version Current

Key

Setting up the Client

Transfer between LRZ and JSC

Encrypted transfers

Transfer between LRZ and JSC

Transfer between LRZ and JSC (with client at JUDAC)

Transfer between LRZ and JSC (with client at JUDAC)

Transfer between LRZ and HLRS

Transfer between LRZ and HLRS

Associate your DN from your personal certificate with your LRZ-username

Encrypted transfers

Copying data between the sites

Data Transfer to/from PRACE sites

Beyond Data Transfer: Sharing and Public Access

Page tree

Page History

Versions Compared

Old Version 14550

New Version Current

Key

Setting up the Client

Transfer between LRZ and JSC

Encrypted transfers

Transfer between LRZ and JSC

Transfer between LRZ and JSC (with client at JUDAC)

Transfer between LRZ and JSC (with client at JUDAC)

Transfer between LRZ and HLRS

Transfer between LRZ and HLRS

Associate your DN from your personal certificate with your LRZ-username

Encrypted transfers

Copying data between the sites

Data Transfer to/from PRACE sites

Beyond Data Transfer: Sharing and Public Access