DSS documentation for data curators
1. Introducing DSSWeb
As a data curator of one or more data projects, you can manage your DSS storage via our DSSWeb self-service system. The system provides you with a Web Interface (GUI), a Command Line Interface (CLI) and a HTTP REST API (API).
You can think of DSSWeb as your virtual storage system administrator, which accepts your request, such as:
- Create a 10TB DSS container for project XYZ and back it up every day
- Give user Max access to DSS container XYZ
- Export DSS Container XYZ to my virtual machine with IP X.X.X.X for 10 days
and then automatically executes the necessary steps in order to fulfil your request.
IMPORTANT: The DSSWeb Portal is only accessible within the MWN for security reasons. Data Curators outside of the MWN can use LRZ's VPN Service or login to the Cloud CLI from one of the LRZ Compute Nodes.
1.1. Introducing the DSSWeb Management Portal
The DSSWeb Management User Interface is accessible via https://dssweb.dss.lrz.de. Your user needs to have data curator rights of a lest one data project, to log in.
1.2. Introducing the DSSWeb Management CLI
The DSSWeb Command Line Interface is currently available as Software as a Service (SaaS) in the LRZ cloud (which we recommend to use) and as an on-premise software solution. In the following, we will describe how to get started with both versions of the CLI.
1.2.1. Installing the CLI on premise
In order to install the CLI on premise, the installation of git and python3.4 (or higher) is required. Use the following steps to install the CLI on a *NIX system:
$ git clone https://gitlab.lrz.de/a2822bp/dssweb-client.git && cd dssweb-client $ python3 -m venv env # create a new virtual environment called 'env', where x is your current python 3 version (env)$ source env/bin/activate # activate your virtual environment (env)$ pip install -r requirements.txt # install all dependencies to your virtual environment (env)$ python setup.py install # install dsscli to your virtual environment
After successful installation you should be able to start the CLI in interactive mode using the dsscli command.
$ dsscli (dsscli) help Shell commands (type help <topic>): =================================== cmdenvironment help load pyscript run set shortcuts edit history py quit save shell show Application commands (type help <topic>): ========================================= complete dss container update login project show dss container create dss nfsexport create pool list dss container list dss nfsexport list pool show dss container show dss nfsexport show id pool show id dss container show id help project list ...
You can also use the CLI in non-interactive mode, like a regular shell command by calling the dsscli command directly with the particular arguments for the action you want to perform.
$ dsscli -h usage: dsscli [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug] DSSWeb Client optional arguments: --version show program's version number and exit -v, --verbose Increase verbosity of output. Can be repeated. -q, --quiet Suppress output except warnings and errors. --log-file LOG_FILE Specify a file to log output. Disabled by default. -h, --help Show help message and exit. --debug Show tracebacks on errors. Commands: ... help print detailed help for another command (cliff) login Request username and password and obtain a token from DSSWeb API. ...
1.2.2. Updating the CLI
As new features become available, we will update the code in the gitlab repository, so make sure to update the software from time to time. In order to update the software, just follow the installation procedure again. It will overwrite the old version of the software.
1.2.3. Accessing the CLI in the cloud
In order to access the interactive CLI, which is offered as Software as a Service in the LRZ cloud, just use SSH to login to dsscli.dss.lrz.de. In order to be able to log in, your user will need to have data curator rights for one or more data projects.
$ ssh XXXXXX@dsscli.dss.lrz.de XXXXXX@dsscli.dss.lrz.de's password: (dsscli) help Shell commands (type help <topic>): =================================== cmdenvironment help load pyscript run set shortcuts edit history py quit save shell show Application commands (type help <topic>): ========================================= complete dss container update login project show dss container create dss nfsexport create pool list dss container list dss nfsexport list pool show dss container show dss nfsexport show id pool show id dss container show id help project list ...
For your convenience, you can provide one or more SSH public keys in order to be able to login in without having to provide a password. Just use the edit command, which will allow you to edit the authorized_keys file of your user.
Another advantage of the SaaS version is that we will provide you with updated versions of the CLI automatically.
1.2.4. General usage of the CLI
The CLI is structured in several subcommands, that allow you to list, show, create and change various items of a data project like containers, access invitations, NFS exports, etc. To get a list of available subcommands, just call the help command in interactive mode or use the -h switch in non-interactive mode.
In order to get help on a particular command, you can just call the subcommand, followed by the -h switch.
(dsscli) dss container list -h <---- You can also use dsscli dss container list -h for non-interactive version
usage: dss container list [-h] [-f {csv,json,table,value,yaml}] [-c COLUMN]
[--max-width <integer>] [--fit-width]
[--print-empty] [--noindent]
[--quote {all,minimal,none,nonnumeric}]
[--poolname [POOLNAME]]
[projectname]
List all available DSS data containers.
positional arguments:
projectname
optional arguments:
-h, --help show this help message and exit
--poolname [POOLNAME]
output formatters:
output formatter options
-f {csv,json,table,value,yaml}, --format {csv,json,table,value,yaml}
the output format, defaults to table
-c COLUMN, --column COLUMN
specify the column(s) to include, can be repeated
table formatter:
--max-width <integer>
Maximum display width, <1 to disable. You can also use
the CLIFF_MAX_TERM_WIDTH environment variable, but the
parameter takes precedence.
--fit-width Fit the table to the display width. Implied if --max-
width greater than 0. Set the environment variable
CLIFF_FIT_WIDTH=1 to always enable
--print-empty Print empty table if there is no data to show.
json formatter:
--noindent whether to disable indenting the JSON
CSV Formatter:
--quote {all,minimal,none,nonnumeric}
when to include quotes, defaults to nonnumeric
Note that the output of the commands is usually optimised for human readability. However, if you want to script certain things, you might be interested using the -f switch to change the output formatting to something, which is easier to parse by a script.
1.3. Introducing the DSSWeb Management API
Currently the API still lacks some important documentation information. Though, we are working hard on extending this documentation in the future. You can access the current documentation via https://dssweb.dss.lrz.de/api/docs/.
If you are interested in using our API, please raise an Incident for 'Datenhaltung → Data Science Storage' at the LRZ Servicedesk.
2. Understanding Data Projects and Data Pools
In the LRZ Data Science Management Concept, we use Data Projects as organizational envelopes around the data, which is stored in the context of this data project. This is particularly useful as it implements a link between the data and the people who are formally responsible for the data and therefore protects us from orphaned data.
In order to keep this link alive, as master user of such a data project, LRZ will request you to confirm prolongation of the data project, every year. If you fail to do so, we will shut down the project and delete the data that is stored in the context of it.
A particular data project is managed by a group of users, which is called the data curator group of the project. By default only the master users of the project belong to this group, but upon request via the Servicedesk, LRZ can also add arbitrary other users to the list of data curators for a data project.
In order to be able to store data in the context of a data project, the project also needs grants on one or more data pools. Data pools are physical storage systems which are integrated into the LRZ Data Science Management Ecosystem. Currently only Data Science Storage systems are supported, but it is planned to add other systems like an Archive for example in the future. Grants specify which fraction of a particular data pool (a quota), a particular data project is allowed to use. These grants are managed by LRZ.
Note that data pools are usually optimized for a particular data pattern or workload. This information is provided you via the description and hints fields of the data pool to give you some guidance on container placement.
2.1. Getting information about the data projects of a data curator
2.1.1. Using the GUI
When you log in to the DSSWeb GUI, you will be automatically redirected to the Project Overview site, which will give you all necessary information about the data project(s), you are curator of. In order to drill down to other information, you'll first have to select the project you want to work with. You can always come back to the Projects site and select a different project, if you need to switch context.
2.2. Getting information about the grants for a particular data project
2.2.1. Using the GUI
Once you've selected a project to work with, you'll be automatically redirected to the Project Detail site.
Now click on the DSS Pools Icon and you will be redirected to the DSS Pool Overview site, which will list you all available storage grants for your project.
The diagram will show you, how much of your storage grant has already been allocated to containers by you or your data curator colleagues.
3. Understanding Containers
Containers are the organizational elements, in which your actual data will live. Depending on the type of the underlying storage system, containers could be implemented in very different ways. For the Data Science Storage (DSS) systems, a container is basically a directory in a POSIX file system, with some additional magic attached to it. For a future archive system or an object storage system, a container will most likely be implemented in a different way. However, what is important to note is the basic concept of a container, which will be equal across all potential underlying storage systems:
- A container is some kind of storage element with a certain size, which lets you store data in it.
- A container should be used to encapsulate data, which semantically belongs together.
- You can invite arbitrary users, which exist in the LRZ Identity Management System to access the data in the container.
3.1. Understanding Data Science Storage Containers
As already discussed, data containers on a Data Science Storage system are implemented as a directory on a POSIX file system, which is owned by a POSIX group, dedicated to the container. Each container has a certain limitation on the capacity and number of files, which can be stored in the container. The members of the container group are managed by the data curators via an invitation process. The access to the container is managed via POSIX rights and NFSv4 type ACLs. Also each container is associated with a so called ID Streamlining Mode. This policy defines how hard the system will try to make sure that access rights and ACLs stay in alignment with the access rights concept. Last but not least, each DSS type container is associated with a so called protection policy, which defines if and how the data in a container is protected by regular backups to the LRZ Backup- and Archive system.
3.1.1. Understanding Data Science Storage Container ID Streamlining Modes
In contrast to other file system types, such as NTFS, you cannot revoke certain POSIX mode bit and ACL change rights of the owner of a file. Therefore you cannot guarantee, that a user may accidentally or on purpose do something unexpected, that may break the rights enforcement methods in place. However, there are certain measures that can be taken, which increasingly deviate from the POSIX standard. Therefore, we offer several levels of ID Streamlining Modes, so you can pick the right one for your use-case.
- The most basic mode is called
NONE. Using this policy, the container directory will be created with owning userrootand owning group <container group> and sets the POSIX rights torwxrws—. This will make sure that directories and files created within the container will automatically inherit the container group setting and that directories created within the container will inherit thesetgidbit. However, every user can change the permissions of their files and directories using chmod and therefore overwrite the default settings easily. So the use case for this policy is usually that you want to let your users manage the access rights within the container by themselves and we only make sure that access at the container level is restricted to the invited container users. - A more restrictive but still almost POSIX compliant mode is called
NORMAL. This is also the default mode, you should choose if you are in doubt. This implements the same POSIX mode bits as theNONEmode, but in addition sets a set of NFSv4 style ACLs on the container directory, that will automatically be inherited by all files and directories, created in the container, which makes sure that all users belonging to the container group haverw-rights on files andrwxrights on directories. The point where this setting deviates from the POSIX or NFSv4 semantic is that if a user changes the POSIX mode bits usingchmod, we do not replace the ACL by the new mode bits, but instead only update the ACL and therefore keep our access right settings for the container group intact. So basically the only way for a user to overwrite the permissions on a file he owns, is by explicitly editing the NFSv4 style ACL. - A even more restrictive and barely POSIX compliant mode is called
STRICT. This implements the same techniques described for theNORMALmode, but disallows the use ofchmodcompletely. As this happens on the file system level, this also holds true for each program library call, that uses achownPOSIX API call. So users should not be able to remove thesetgidbit. However, by editing the ACLs, they still can overwrite the permissions. - A special version of the mode
NONEis the WORK policy. From a container perspective both are equivalent. However for WORK, DSS will create a dedicated user directory with POSIX rightsrwx------for each invited user.
3.1.2. Understanding Container Access Rights Management
For managing container access rights, we follow an invitation approach as you may already be familiar from cloud storage services like the LRZ Sync+Share service or Dropbox, Google Drive, etc. This means, for example, if data curator Alice wants to give user Bob access to the data container Cont-A, Alice will tell DSSWeb to invite Bobs username(s) to access container Cont-A. DSSWeb will then send an E-Mail, containing a unique invitation link to the E-Mail address assigned to Bobs username in the LRZ Identity Management System. Bob can then accept the invitation by clicking on the invitation link and accepting our data privacy and usage terms. After that, Bobs username will be automatically placed in the container POSIX group of container Cont-A and Bob is ready to access the data in Cont-A. However, if Bob chooses not to accept the invitation for whatever reason, the pending invitation will automatically expire after 30 days in DSSWeb.
In certain constellations, a data curator may want to delegate the access rights management for a single container to another user, but not give him or her data curator rights for the whole data project. In this case, the data curator can set the Container Manager flag for the invitation of the user, which will give him access to the DSSWeb Portal and allow him to create/read/update/delete Invitations and Auto Group Links as well as NFS exports for the particular container he was granted the Container Manager right.
3.1.3. Understanding Container Access Revocation and User Deletion Actions
As you may have noticed, the semantics of Data Containers are designed as collaborative space, in which every user can access all data. So semantically the data is not owned by an individual, but the whole group. However, as we operate on a POSIX file system, the POSIX semantics dictate that for each file there has to be a single user who is the formal owner of the file. Technically this is implemented by storing the UID of the owning user for each file in the metadata of the file.
Now let's look at the case when a data curator wants to revoke access of a certain user to a container. Therefore, we remove the user from the container group and because of the permissions set on the container level, the user cannot access any data in the container anymore. However, there may now still be files in the container which are formally owned by the revoked user. While this is technically insignificant - since the user cannot access the data in the container anymore - it may be misleading and ambiguous to the remaining users. The same holds true if a user, invited to a container gets completely deleted. In this case, even resolving the UID to a user name will not work anymore and you will be presented with a cryptic number as owner of the files in your container.
In order to avoid these kinds of problems, whenever you revoke access for a user or a user get's deleted in the Identity Management System, we automatically start a batch job, that changes ownership of all files, which belong to the revoked or deleted user. Please note that this will not happen if the container ID Streamlining Mode policy is set to NONE. Currently the new owner is automatically determined from the list of data curators, whereby we choose that user, which is the first one in an alphabetical order. Note that this happens asynchronously so there may be a noticeable delay between revocation/deletion and ownership change.
We plan to extend this functionality in future releases, so that you may be able to specify a new owner, when you revoke a particular user, or put all data owned by that user to a tape archive or even delete all data owned by that user. If you have a particular use case, which you wish to see implemented, please open a ticket a the LRZ Servicedesk so that we can discuss it and file a user story in our product backlog.
3.1.4. Understanding Automatic Container Access Group Links
There may be situations where explicitly managing DSS Container Access Rights for each individual user may represent unnecessary overhead, because you want to give one or more particular groups automatically access to a container. So when a new member is added to the group, he should automatically be invited to access the container and when a member is removed from the group he should automatically be revoked access rights.
You can achieve this by defining so called Auto Group Links for a container. We currently support to link groups from DSS, TUMonline, LMU, LinuxCluster and plan to extend this to SuperMUC-NG groups in the future. For more details on this, please check out our Knowledge Base article on this topic.
Please note that invitations, created from Automatic Container Access Group Links have very limited manual management capabilities. The only thing, a data curator can manually do is to resend an invitation mail for such an invitation.
3.1.5. Understanding Data Science Storage Container Protection Policies
In contrast to highly available and therefore more cost intensive LRZ storage offerings like the LRZ NAS Cloud or LRZ Sync+Share, where data is replicated between two independent disk storage systems, so that in case a whole storage system fails we can switch the service to the second system within hours, Data Science Storage is cost and performance optimized. Therefore we do not replicate the data on DSS systems to a second online storage system but just offer traditional tape backup to the LRZ Backup- and Archive system as the only data protection mechanism. As data protection needs may vary from use-case to use-case, we offer several protection policies.
Please be aware that because of the typical size of multiple PBs of a single Data Science Storage system, the Recovery Time Objective for a catastrophic system failure, in which the data of a complete DSS system has to be restored is typically targeted to be one month.
- The most simple data protection policy is called
NONE. Containers which are assigned this protection policy are not protected at all. Use this policy only for data, which can easily be regenerated or recaptured. - For automatic backups, which are performed once a week, use the policy
BACKUP_WEEKLY. This will perform an incremental backup run of your container once a week (usually during the weekend). The retention policies of the backup are 180 days and a maximum of 3 versions per file. For more information about DSS backup retention policies, see DSS Understanding Data Science Storage Container Backup Retention. - For automatic backups, which are performed once a day, use the policy
BACKUP_DAILY. This will perform an incremental backup run of your container every day (usually during night). The retention policies of the backup are 180 days and a maximum of 3 versions per file. For more information about DSS backup retention policies, see DSS Understanding Data Science Storage Container Backup Retention.
For containers which exclusively store static, read-only content, like simulation results, output data from instruments like genomic sequencers, microscopes, NMR spectroscopes, etc, we also provide special ARCHIVE policies. These policies help you to be compliant with the DFG rules of good scientific practice, which require you to retain scientific primary data for 10 years. This is implemented by assigning the backed up data of your container a special retention policy, that keeps backups for 10 years and also keeps two copies of the backed up data in two different locations. Please note the restrictions that apply when using this data protection policy: DSS Understanding Data Science Storage Container Backup Retention.
- For automatic incremental archives of static data, which are performed once a week, use the policy
ARCHIVE_WEEKLY. This will perform an incremental archive run of your container once a week (usually during weekend). - For automatic incremental archives of static data, which are performed once a day, use the policy
ARCHIVE_DAILY. This will perform an incremental archive run of your container once a day (usually during night).
For information on how to restore backed up files, please check this Knowledge Base article.
3.1.6. Understanding Data Science Storage Container Sharing with External Users
In addition to sharing container data with LRZ/TUM/LMU users, you can also share your container data with arbitrary external users, using the 3rd party Globus Sharing service. We have integrated Globus Sharing via their API into the DSSWeb Self-Service portal for your convenience, so you can easily manage Globus Sharing permissions via DSSWeb. However, in order to use Globus Sharing on a particular container, we have to do some special preparations, which are not necessary if you don't need Globus Sharing. Therefore, you have to explicitly enable Globus Sharing on your container. This can either be done at container creation time or anytime afterwards. You can also disable Globus Sharing at any time. After Globus Sharing has been enabled on your container, you can invite arbitrary external users to access your data via Globus by just specifying their Globus User ID or E-Mail address. They will then receive an invitation Mail from Globus that provides the necessary instructions in order to access the shared data.
For details on how Globus Sharing and the integration into DSS and DSSWeb works, please check this Knowledge Base article.
4. Understanding the Asynchronous Task Model
As we already described earlier, DSSWeb takes your high level requests (like create a new container) and automatically executes the necessary steps to fulfil your request. Every time you tell DSSWeb to create or change something, it must talk to one or more external systems, like LRZ SIM, LRZ's Backup and Archive System, the underlying filesystems of DSS and so on and ask them to execute one or more tasks. So under the covers, DSSWeb is a fairly complex, highly distributed system which depends on multiple systems working seamlessly together. However, in such highly distributed systems, we have to accept that things may fail from time to time. Fortunately, these failures often have a temporary and very seldom a permanent nature. For example, think of the situation in which one of the subsystems is not available for whatever reason. This situation is usually fixed very fast and then everything works just fine again.
In order to hide all this complexity and not to bother you with temporary errors, thereby giving you the best possible user experience, we work with an asynchronous task model. That means, whenever you tell DSSWeb to create or change something, it will just store the necessary information in its database and make sure that an asynchronous task is started in the background. So while DSSWeb may still be busy, fulfilling your request, you can already go on and do other suff and then check back later to see if your request was carried out successfully.
You can check the status of an object (like a container), by looking at it's Status and Active Task fields. Typically when you first create an object it goes to a CREATE_PENDING state, meaning the object is currently being setup. When an object reaches a "steady state", meaning it was successfully created, its status is usually ACTIVE. When you change an object, it will go to a CHANGE_PENDING state and as soon as the changes have been carried out it will go back to ACTIVE again. The same semantics often are also true for deletion of objects, meaning the object will got to a DELETE_PENDING state and only after it has been successfully cleaned up on the subsystems, it is removed from the DSSWeb database.
In the GUI, the status is usually represented via Icons. You can get a textual representation of the state by simply put your mouse pointer on such a status icon.
Please note that currently, when an object is in transition, we do not accept any further changes to the object. So whenever an object is in a PENDING state, you have to wait until it is ACTIVE again to apply the next operation to this object.
4.1. Understanding Task Failures
Usually, when an asynchronous task hits a failure in a subsystem, the only thing you should notice is that the task may take longer than normal to execute (there is no special state indicating a problem for an object, it will simply stay in the PENDING state). The implemented semantics are that whenever a task encounters a failure, it will retry after some minutes until the task can finally be finished. DSSWeb monitors the execution times and number of retries of tasks and in case the retry count of a task gets unusually high, it will notify LRZ personal to take a look. (We currently also think about automatically raising an Incident at the LRZ Servicedesk for you, so that you get feedback on the status of the fixing procedures.)
5. Understanding our Software Development Process
Our software development process is loosely based on an agile approach and is highly driven by the demands of our customers. So, if you are missing a particular feature or would like to see something particular to be implemented in DSSWeb, please don't hesitate and reach out to us by opening a request on the LRZ Servicedesk so we can discuss your requirements and record it in our product backlog.
6. Managing Data Curators
When a LRZ project receives the right to use DSS, the project will be automatically imported to DSSWeb and all Master Users of the LRZ project will be designated as initial data curators for the project in DSSWeb and receive a welcome mail from DSSWeb. However, from there on the roles of Master User and Data Curator are not linked anymore. This means that the data curators can manage the list of data curators for their particular project in DSSWeb independent from the Master Users in SIM. On the other side, changes of the Master User in SIM will not be reflected in DSSWeb. For every data project, there can be zero or one primary data curators. If a data curator has been assigned the primary role, whenever a user is removed from the list of invited users or the user is deleted in the LRZ SIM user directory, DSS will change ownership of the files owned by this user to the primary data curator. If no primary data curator is assigned for a project, the data curator to which file ownership will be changed to is arbitrarily chosen from the list of available data curators.
6.1. Showing Data Curators
In the following, we show you how to get an overview of the current data curators of your project.
After selecting a particular project in the Project Overview site in the GUI, you'll see a list of the current data curators of that project. In the following illustration, you can see the data curators of the project pr74qo.
6.2. Adding new Data Curators to a project
In the following, we show how you can add additional data curators to your project. When you add a user as data curator, the user will receive an automatically generated welcome mail from DSSWeb that informs him about the granted right and points him to this documentation.
Adding data curators is currently a synchronous task. Therefore the request/command could take several seconds to complete.
To add a new data curator within a project for which you have data curator rights, click the + button at the bottom of Data Curators section. In the following example, we will add the user with the username di93jur as a data curator in the project pr74qo.
Enter the username of the future data curator. Optionally, you can choose for the data curator to be primary, by selecting Is primary. Click the Add button to create the data curator.
6.3. Updating a Data Curator
In the following, we show you how you can change the primary data curator for a project.
There can at max only be one primary data curator per project. When you assign primary data curator role to a curator in a project where another curator is already primary data curator, the primary role is automatically revoked from the existing primary data curator and transferred to the newly appointed primary data curator.
To set an existing data curator as a primary data curator within a project for which you have data curator rights, click the blue button next to the name of the data curator. In the following example, we will set the data curator with the username di93jur as primary data curator in the project pr74qo.
To distinguish itself from the others, the primary data curator is assigned a green circle next to it, meanwhile the blue button has now disappeared.
6.4. Removing a Data Curator from a project
In the following, we show how you can remove a data curator from a project. When you remove a user as data curator of a project, he will only use the data curator right for this project. Any container access invitations of this user will not be touched. If you also want to revoke access to the data of the project, you must also delete the invitations of the user in the individual containers.
Deleting data curators is currently a synchronous task. Therefore the request/command could take several seconds to complete.
To delete an existing data curator from a project for which you have data curator rights, click the red button next to the name of the data curator. In the following example, we will delete the data curator with the username di93jur in the project pr74qo.
After you click the red button, a new window will appear to inform you about deleting the selected data curator. Click Delete to terminate the process.
As shown in the illustration below, the deleted user (di93jur) has now disappeared from the Data Curators section.
7. Managing Containers
In the following we describe how you can create a new DSS Container for your data project.
7.1. Creating Containers
Please note that in order to create a new DSS container, the following requirements must be met:
- You must be assigned the data curator role for the data project
- Your data project must own a grant for the specific data pool in which you want to create the container
- The size and file limits of the container must be less or equal to the available size and files in your pools grant
Please note that currently you cannot change the data protection mode or ID Streamlining Mode of a container. Since theses changes may require substantially resources to perform a transition or may have unwanted side effects on your data, currently these properties can only be set at container creation time. So choose carefully at container creation time.
If you really require such a transition for an existing container, please contact us via the LRZ Servicedesk.
7.1.1. Using the GUI
Once you've selected a project to work with, you'll be automatically redirected to the Project Detail site.
Now click on the DSS Containers Icon and you will be redirected to the DSS Container Overview site, which will list you all containers for your project.
As long as you have storage grants for an additional container available, you can click on the New DSS Container Icon, to add a new container.
The Fields which are marked with an asterix * are mandatory fields. The other fields are optional. Fill out the form and click on Submit to create your new container.
If all the container could be successfully created, the UI will show you a System Message, that indicates that the container has been created.
If there were any errors, you will receive a System Message that indicates which error occurred instead.
After container creation, you'll see the new container on the DSS Container Overview site, whereby the status icon will indicate that creation is still pending.
Once the creation process has finished, the status icon will automatically change to the active 
Icon.
Please note that all the data curators of a data project will automatically get access to the containers, created for their project. If that is not what you want, you have to revoke access for them manually.
If the used space or used number of files in a container cross the 80, 90 and 99 percent boundary, DSSWeb will inform all data curators of the project about this.
7.2. Showing Containers
Please note that even in order to just access container information via DSSWeb, you currently need to be data curator for the particular project, the container belongs to.
7.2.1. Using the GUI
Once you've selected a project to work with, you'll be automatically redirected to the Project Detail site.
Now click on the DSS Containers Icon and you will be redirected to the DSS Container Overview site, which will list you all containers for your project.
In order to get more details on a particular container, just click on Select and you will be redirected to the DSS Container Details site.
The Used Space diagram will show you, how much of the granted capacity and files for the container has already been used. Please note that this is not a real time display and also it may not be 100% accurate.
7.3. Modifying Containers
In the following we show how certain properties of containers can be changed. Currently you can change the following properties of an existing container:
- The maximal size of the container (Quota GB)
- The maximal number of files of the container (Quota Files)
- The description
- Whether Globus Sharing is enabled or disabled
- The data protection mode
Please note that it is technically not possible to change the data protection mode from BACKUP to ARCHIVE and vice versa. However it is possible to first change the mode to NONE and then to the desired new protection mode. But you have to bear in mind that changing data protection mode to NONE will result in deletion of all BACKUP/ARCHIVE data!
7.3.1. Using the GUI
Once you've selected a project to work with, you'll be automatically redirected to the Project Detail site.
Now click on the DSS Containers Icon and you will be redirected to the DSS Container Overview site, which will list you all containers for your project.
In order to change settings for a particular container, just click on the containers Edit Button and you will be redirected to the DSS Container Update site.
Fill out the form and click on Submit to update your container.
If the container could be successfully changed, the UI will take you to the DSS Container Details site and show you a System Message, that indicates that the container has been updated.
After container update, you'll see the new container on the DSS Container Overview site, whereby the status icon will indicate that update is still pending.
Once the creation process has finished, the status icon will automatically change to the ACTIVE Icon and the Edit Button will be activated again.
Please note that you can only apply new changes to a container if it is in the ACTIVE state. If the container is not in the ACTIVE state, the Edit Button will be automatically disabled.
7.4. Delete Containers
In the following we show how you can delete a DSS container.
Deleting a DSS Container is an operation that is final and cannot be reverted. Please note that all data, including potential backups and archives will be deleted!
7.4.1. Using the GUI
Once you've selected a project to work with, you'll be automatically redirected to the Project Detail site.
Now click on the DSS Containers Icon and you will be redirected to the DSS Container Overview site, which will list you all containers for your project.
In order to delete a particular container, just click on Delete. Then you will see a confirm box. Enter a container name and click Delete.
After container deletion, you'll see this container on the DSS Container Overview site, whereby the status icon will indicate that deletion is still pending.
Once the deletion process has finished, this container will be automatically removed from DSS Container Overview site.
Please note that you can only apply new changes to a container if it is in the ACTIVE state. If the container is not in the ACTIVE state, the Delete Button will be automatically disabled.
8. Managing Container Access Rights
As discussed, Data Science Containers are collaborative spaces. As such, you can grant virtually any user, which is known in the LRZ Identity Management, access to a data container. As the Identity Management Systems of LRZ, TUM, LMU are kind of federated, this also means that you can grant any user, managed by one of these IdMs access to your data containers.
Please note that the various ways in which users can access the data within a container is documented in the DSS documentation for users document.
8.1. Granting Access Rights
In order to allow an user to access a particular data container, you must invite him or her to the container group.
8.1.1. Using the GUI
In order to invite a user to access a particular data container, navigate to the Container Detail site.
On this detail page, you'll find the SIM User tab in the Container User section:
Click on the Add new user button and you will be redirected to the Invitation Create site.
The Fields which are marked with an asterix * are mandatory fields. The other fields are optional. Fill out the form and click on Invite to create your new invitation.
Please note that you can only create Invitations of Access Type RO (Read Only) if the Container has set ID Stremline Mode to STRICT.
Enable the Is Container Manager flag only if you want to delegate the invited user the right to manage all invitations and auto group links for this particular container via DSSWeb himself. See the Understanding Container Access Rights Management section above.
If the invitation could be successfully be created, the UI will take you back to the DSS Container Details site and show you a System Message, that indicates that the invitation has been created.
After invitation creation, you'll see the new invitation on the DSS Container Overview site, whereby the status icon will indicate that creation is still pending.
Once the creation process has finished, the status icon will automatically change to the USER ACCEPT PENDING 
state and once the user has accepted the invitation it will automatically change to the USER ACCEPTED 
state.
Please note that you will not see first name and last name until the invitee has accepted the invitation.
8.2. Showing Access Rights
8.2.1. Using the GUI
In order to display the access rights of a container user, please navigate to the Container Detail site.
On this detail page, you'll find the SIM User tab in the Container User section:
8.3. Modify Access Rights
In the following we show how certain properties of invitations can be changed. Currently you can change the following properties of an existing invitation:
- For invitations that have not yet been accepted:
- Prolong expiration date
- Resend the invitation mail
- The maximum amount of GB, the user is allowed to consume within the container.
- The maximum number of files, the user is allowed to create within the container.
- If the invited user shall be a manager for this container, which will give him the possibility to manage invitations, auto group links and NFS exports for this container.
- For invitations that have already been accepted:
- The access mode to grant. This can either be
READ_WRITEorREAD_ONLY. However, the later can only be used with containers that implement an ID Stremline Mode ofSTRICT. - The maximum amount of GB, the user is allowed to consume within the container.
- The maximum number of files, the user is allowed to create within the container.
- If the invited user shall be a manager for this container, which will give him the possibility to manage invitations, auto group links and NFS exports for this container. (Default: no)
- The access mode to grant. This can either be
8.3.1. Using the GUI
In order to change the access rights of a container user, please navigate to the Container Detail site.
On this detail page, you'll find the SIM User tab in the Container User section.
For each invitation, you'll find four buttons on the right. Depending on the state of the invitation, these buttons may be enabled or disabled.
- The
button prolongs a still pending invitation - The
button resends the invitation mail for a still pending invitation - The
button directs you to the invitation modification form - The
button deletes the invitation
When you click on the prolong or resend button, the action will be immediately carried out and a System Message will appear that shows you the result of the action:
The invitation update button will take you to the Invitation Update form. Fill out the form and click on the Modify Button.
If the invitation was successfully changed, the UI will take you back to the DSS Container Details site and show you a System Message, that indicates that the invitation has been updated.
Depending on the type of the update, the invitation may go to a CHANGE PENDING state until the update has been carried out on the connected systems.
8.4. Revoking Access Rights
In the following, we show how existing invitations can be revoked/deleted. You can revoke accepted as well as pending invitations. If you revoke a pending invitation, the invitation link, sent to the person you invited will be invalidated, so accepting the invitation is no longer possible. If you revoke an already accepted invitation, the user will be removed from the container access group and file ownership of each file, technically owned by the user, will be transfered to a data curator. See Understanding Container Access Revocation and User Deletion Actions for more information on the topic.
8.4.1. Using the GUI
In order to delete the access rights of a container user, please navigate to the Container Detail site.
On this detail page, you'll find the SIM User tab in the Container User section.
For each invitation, you'll find four buttons on the right. Depending on the state of the invitation, these buttons may be enabled or disabled.
- The button prolongs a still pending invitation
- The button resends the invitation mail for a still pending invitation
- The button directs you to the invitation modification form
- The button deletes the invitation
In order to delete an invitation, click on the Delete Button. After that, a confirmation dialog will appear.
If you confirm the deletion by clicking on the Delete Button of the confirmation dialog, the particular invitation will automatically turn into DELETE PENDING state and once, deletion is complete will disappear from the Container User list.
9. Managing DSS Container Access Rights for External Users via Globus
If you need to share data with people who do not have a user account at LRZ, LMU or TUM, you can do so, as discussed above, using the Globus Sharing feature of the Globus Data Management Portal.
Please note that you don't have to (and should not) use Globus Sharing if the person you want to share data with has an active LRZ, LMU or TUM account. Globus Sharing is only a fall back solution for sharing data with people who don't have and can't get a valid user account in the mentioned institutions. If possible, always prefer Internal Access Rights.
Please note that in order to manage Globus Sharing ACLs, you first must enable the particular data container for Globus Sharing. See the DSS Data Container section above for how to do this.
Managing Globus Sharing permissions via DSSWeb has been deprecated in favour of giving you direct access to the Globus Sharing Interface in Globus via the Access Manager role. If you want/need to manage Globus Sharing ACLS via CLI, please use the Globus CLI.
Managing Globus Sharing permissions is not done in DSSWeb directly but via the Globus Online web interface. In order to be able to manage Globus Sharing permissions for a particular container, navigate to the Container Detail site of a container. On this page, you'll find the Globus User tab in the Container User section. When you click on the Manage Globus Sharing Permissions button, DSSWeb will add your user, you used to log in to DSSWeb, as so called Access Manager to the Globus Shared Endpoint and redirect you to the Globus Online web interface. When you log in to the Globus Online web interface, make sure to use the same LRZ user account which you also used to log in to DSSWeb. After that you should be able to manage the access permissions of the containers shared endpoint via the Globus Online web interface. See https://docs.globus.org/how-to/share-files/ step 5 and following on how to do that.
10. Managing DSS Container Auto Group Links
If you already manage a TUMOnline Group and want that everyone in this group automatically gets an invitation for a particular data container, you can do so using the Container Autor Group Links, already discussed above. In the following we will show you how to manage these Auto Group Links.
10.1. Creating Auto Group Links
10.1.1. Using the GUI
In order to create a new auto group link for a particular container, navigate to the Container Detail site of a data container.
On this detail page, you'll find the Auto Group Links tab in the Container User section:
Click on the Create new group link button and you will be redirected to the Group Link Create site.
The Fields which are marked with an asterix * are mandatory fields. The other fields are optional. Fill out the form and click on Create to create your new group link.
If the group link could be successfully be created, the UI will take you back to the DSS Container Details site and show you a System Message, that indicates that the invitation has been created.
After group link creation, you'll see the new group link on the Auto Group Links tab of the DSS Container Overview site.
10.2. Showing Auto Group Links
10.2.1. Using the GUI
In order to display the auto group links of a container, please navigate to the Container Detail site.
On this detail page, you'll find the Auto Group Links tab in the Container User section:
10.3. Updating Auto Group Links
10.3.1. Using the GUI
In order to change the auto group links of a container, please navigate to the Container Detail site.
On this detail page, you'll find the Auto Group Links tab in the Container User section.
For each group link, you'll find buttons on the right. Depending on the state of the group link, these buttons may be enabled or disabled.
- The button directs you to the group link modification form
- The button deletes the group link
The group link update button will take you to the Group Link Update form. Fill out the form and click on the Modify Button.
If the invitation was successfully changed, the UI will take you back to the DSS Container Details site and show you a System Message, that indicates that the group link has been updated.
10.4. Deleting Auto Group Links
10.4.1. Using the GUI
In order to delete an auto group link for a particular container, please navigate to the Container Detail site.
On this detail page, you'll find the Auto Group Links tab in the Container User section.
For each group link, you'll find buttons on the right. Depending on the state of the group link, these buttons may be enabled or disabled.
- The button deletes the group link
In order to delete an group link, click on the Delete Button. After that, a confirmation dialog will appear.
If you confirm the deletion by clicking on the Delete Button of the confirmation dialog, the particular group link will automatically disappear from the Container User list, once the deletion is completed.
11. Managing DSS Container Exports
11.1. Limitations
Plese note that there are several limitations for exporting DSS containers via NFS.
You can only export DSS containers to IPs that are located in the LRZ data center. Exporting to IPs external to LRZ is not supported.
All DSS container exports are configured with root_squash enabled. Therefore it is not possible to have "root access" on the data via an NFS client.
Also technically not forbidden, you should only export DSS containers to IPs that are statically assigned to and trusted by you. NFS exports follow a "host based trust" semantic, which means the DSS NFS server will trust any IP/system to which a DSS container is exported. There is no additional user authentication between NFS server and client enforced. This is especially important if you want to export DSS containers to cloud machines, as these - by default - use a dynamically allocated IP, which may be reused by other machines as soon as you shut down your VM.
11.2. Creating exports
11.2.1. Using the GUI
In order to create a new NFS export for one of your containers via DSSWeb GUI, navigate to your container overview page. And click on the green button + Add new export
After clicking the button you get redirected to NFS export creation window, the form takes the following arguments:
- the IP address of the NFS client to export to
- the client access mode, either read-only (RO) or read-write (RW)
- and an optional expiration date at which the export will be deleted automatically
Upon clicking create, the NFS export creation is triggered and you get redirected back to the container overview page where you should see an entry in NFS export table.
If the export is still creating you should see a spinner in the status column.
Once the export is created there should be a green hook.
11.3. Showing exports
11.3.1. Using the GUI
In order to display the NFS exports of a container user, please navigate to the Container Detail site.
On this detail page, you'll find the NFS Exports section:
11.4. Updating export
11.4.1. Using the GUI
In order to update an existing NFS export for one of your container via DSSWeb GUI to container overview page. Look for the NFS export table and click on the orange button in the Actions column.
In the following table are five NFS exports. Four of them are part of a container config (config column LRZ Remote Visualisation) which means that they cannot be edited manually and hence the edit button is disabled.
NFS exports managed by you are editable. By clicking edit you are redirected to NFS export update view where you can change:
- the access mode read-only or read-write
- the expiration date after which the export is deleted
When you made your changes hit modify to apply your changes.
11.5. Deleting exports
11.5.1. Using the GUI
In order to delete an existing NFS export navigate to your container overview page. And select an export in the NFS export table by clicking on the read delete button. You can only delete NFS exports manually which are not part of a container config.
Upon clicking the read delete button you will see a confirmation dialog to reassure deletion is wanted.
Click delete in the dialog the deletion is triggered and indicated by red spinner in the NFS export table.
Once the deletion process is finished, the entry is cleared from the table.
11.6. NFS helper functions
As NFS relies on the fact, that all users have the same UID and GID on all attached systems, you have to make sure that you fulfil this requirement also on hosts, which are not subscribed to the LRZ LDAP directory system (like VMs on the LRZ HPC Cloud for example). In order to help you with this, we provide two helper functions via the CLI, that generate /etc/passwd and /etc/group stub entries for you for all invited users of a container. You can use this entries to paste it into the respective files on your hosts, that mount a DSS container via NFS.
In order to get stub entries for /etc/passwd for all invited users of the projects, you are data curator of, you can use the dss passwd list function. The function take a projectname and containername as optional arguments that allow you to filter only the entries of a particular project or container.
(dsscli) dss passwd list --containername pr74qo-dss-0000 pr74qo a2822xx:x:88888:2222::/home/a2822xx: a2822yy:x:99999:2222::/home/a2822yy:
Here the UID of the user a2822xx is 88888. Thus the user account of the VM trying to mount the DSS Container has to be named a2822xx as well and having the UID 88888.
In addition, you can use the dss group list function to generate stub entries for /etc/groups. However, this is not really necessary for correct functioning of the DSS NFS mount but has mainly cosmetic reasons so that the GIDs on the NFS client resolve correctly. As with the dss passwd list command, you can use the optional arguments projectname and containername to filter the group entries.
(dsscli) dss group list --containername pr74qo-dss-0000 pr74qo dssusers:x:2222: pr74qo-dss-0000-ro:x:777098166: pr74qo-dss-0000:x:777098167:
Here the GUID is shown. If the export has been read only, the GUID is 777098166, if it has been a read write export the GUID would be 777098167. The user of the VM has to match this GUID to gain the intendet rights and the above mentioned UID and name to gain successful access to the DSS Container mounted in the VM.
12. Managing DSS Container Configs
Container Configs can be used to manage a predefined set of container users and NFS exports per container. After a config is applied successfully to a container its users are invited and NFS exports are enabled.
12.1. Showing Configs
12.1.1. Using the GUI
To list available container configs navigate to your container's overview page. At the bottom of the page you find a section Container Configs. After clicking on the + Add new Config button there is a dropdown menu with all available configs. Select a config in the dropdown to view its description.
To see a list of active container configs navigate back to the container's overview page and view the Container Configs section at the bottom of the page. The table shows the name of active configs, its description, a status (indicating whether config is in the process of creation/deletion or active) and a delete button the remove the config.
NFS exports managed by the config are listed in the NFS Exports section. It is not possible to update or delete those exports manually.
The same applies to container users. Every container user managed by the config is listed in the Container User section. It is not possible to update or delete those users manually
12.2. Applying Configs
12.2.1. Using the GUI
To add configs to your container navigate to a container's overview page. At the bottom of the page you find a section Container Configs. After clicking on the + Add new Config button there is a dropdown menu to choose from available configs. Upon selecting a config click Apply.
You will be redirected to the container's overview page where you can view the status of the newly created config.
12.3. Deleting Configs
12.3.1. Using the GUI
To delete an active config navigate to the bottom of a container's overview page. And click on the red button with a trash can icon.