Usually every 5 to 6 years we need to refresh the storage hardware that drives the LRZ Data Science Storage. Usually this also means that we need to move the DSS containers on the old pool to a new pool. While we work hard to make this transition as smooth as possible it is not completely transparent to data curators and users though. In the following we describe the current migration process step by step and outline the required data curator and user responses in each of theses steps.

The DSS Data Migration Procedure

Let's say we have a DSS Container called pr74qo-dss-0000 on the DSS pool LRZ_PILOT (which path is /dss/dssfs01) and this container needs to be migrated to the DSS pool SUPERMUC_NG (which path is /dss/dssfs02).

StepWhat will happenRequired Data Curator ActionsRequired User Actions
1In the first step we will create a Read Only twin of the container at /dss/dssfs01/pr74qo/pr74qo-dss-0000 on the new pool under /dss/dssfs02/pr74qo/pr74qo-dss-0000 and will once pull over all data stored at /dss/dssfs01/pr74qo/pr74qo-dss-0000 to /dss/dssfs02/pr74qo/pr74qo-dss-0000If you use Globus Sharing on this container, please note down all Globus Sharing ACLs you defined as these will be lost during the migration.None
2

In the second step we will usually announce a system maintenance with downtime as it is required that all IO on the container source path /dss/dssfs01/pr74qo/pr74qo-dss-0000 is stopped before we can proceed with this step.

So after all IO has stopped on the container source path we will migrate all NFS exports and the Globus Shared Endpoint pointing from the container source path to the container target path /dss/dssfs02/pr74qo/pr74qo-dss-0000

Then we will switch the container target path from Read-Only to Read-Write whereby each file in the container still remains a twin of the file on the source container until the file has been written to the first time on the target container. 

After that, we will migrate any backups from the container source path to the container target path.

And finally we will trigger final sync from the container source to the container target that will pull over all data that has been changed since the first sync, carried out in step 1.

Last but not least we will remove the "twin relationship" between the source and the target container path completely and the migration maintenance is finished.

NoneStop all IO on the container
3After the migration maintenance, data curators and users need to perform some adjustments.If you used Globus Sharing on this container, please recreate the Globus Sharing ACLs you saved in step1.

Make sure your workloads are now pointing to the container target path instead of the container source path

If you use NFS mounts of the container please adjust your NFS mounts to point to the new NFS export server and container target path

Related articles