Introducing DSSWeb

As a data curator of one or more data projects, you can manage your DSS storage via our DSSWeb self-service system. The system provides you with a Web Interface (GUI), a Command Line Interface (CLI) and a HTTP REST API (API). 

You can think of DSSWeb as your virtual storage system administrator, which accepts your request, such as:

• Create a 10TB DSS container for project XYZ and back it up every day
• Give user Max access to DSS container XYZ
• Export DSS Container XYZ to my virtual machine with IP X.X.X.X for 10 days
  

and then automatically executes the necessary steps in order to fulfil your request.

IMPORTANT: The DSSWeb Portal is only accessible within the MWN for security reasons. Data Curators outside of the MWN can use LRZ's VPN Serviceor login to the Cloud CLI from one of the LRZ Compute Nodes.

Introducing the DSSWeb Management Portal

The DSSWeb Management User Interface is accessible via https://dssweb.dss.lrz.de. Your user needs to have data curator rights of a lest one data project, to log in.

Introducing the DSSWeb Management CLI

The DSSWeb Command Line Interface is currently available as Software as a Service (SaaS) in the LRZ cloud (which we recommend to use) and as an on-premise software solution. In the following, we will describe how to get started with both versions of the CLI.

Installing the CLI on premise

In order to install the CLI on premise, the installation of git and python3.4 (or higher) is required. Use the following steps to install the CLI on a *NIX system:

$ git clone https://gitlab.lrz.de/a2822bp/dssweb-client.git && cd dssweb-client
$ python3 -m venv env  # create a new virtual environment called 'env', where x is your current python 3 version
(env)$ source env/bin/activate	# activate your virtual environment
(env)$ pip install -r requirements.txt	# install all dependencies to your virtual environment
(env)$ python setup.py install	# install dsscli to your virtual environment

After successful installation you should be able to start the CLI in interactive mode using the dsscli command.

$ dsscli 
(dsscli) help

Shell commands (type help <topic>):
===================================
cmdenvironment  help     load  pyscript  run   set    shortcuts
edit            history  py    quit      save  shell  show     

Application commands (type help <topic>):
=========================================
complete               dss container update   login         project show
dss container create   dss nfsexport create   pool list   
dss container list     dss nfsexport list     pool show   
dss container show     dss nfsexport show id  pool show id
dss container show id  help                   project list
...

You can also use the CLI in non-interactive mode, like a regular shell command by calling the dsscli command directly with the particular arguments for the action you want to perform.

$ dsscli -h
usage: dsscli [--version] [-v | -q] [--log-file LOG_FILE] [-h] [--debug]

DSSWeb Client

optional arguments:
  --version            show program's version number and exit
  -v, --verbose        Increase verbosity of output. Can be repeated.
  -q, --quiet          Suppress output except warnings and errors.
  --log-file LOG_FILE  Specify a file to log output. Disabled by default.
  -h, --help           Show help message and exit.
  --debug              Show tracebacks on errors.

Commands:
  ...
  help           print detailed help for another command (cliff)
  login          Request username and password and obtain a token from DSSWeb API.
  ...

Updating the CLI

As new features become available, we will update the code in the gitlab repository, so make sure to update the software from time to time. In order to update the software, just follow the installation procedure again. It will overwrite the old version of the software.

Accessing the CLI in the cloud

In order to access the interactive CLI, which is offered as Software as a Service in the LRZ cloud, just use SSH to login to dsscli.dss.lrz.de. In order to be able to log in, your user will need to have data curator rights for one or more data projects.

$ ssh XXXXXX@dsscli.dss.lrz.de
XXXXXX@dsscli.dss.lrz.de's password: 
(dsscli) help
Shell commands (type help <topic>):
===================================
cmdenvironment  help     load  pyscript  run   set    shortcuts
edit            history  py    quit      save  shell  show     

Application commands (type help <topic>):
=========================================
complete               dss container update   login         project show
dss container create   dss nfsexport create   pool list   
dss container list     dss nfsexport list     pool show   
dss container show     dss nfsexport show id  pool show id
dss container show id  help                   project list

...

For your convenience, you can provide one or more SSH public keys in order to be able to login in without having to provide a password. Just use the edit command, which will allow you to edit the authorized_keys file of your user.

Another advantage of the SaaS version is that we will provide you with updated versions of the CLI automatically.

General usage of the CLI

The CLI is structured in several subcommands, that allow you to list, show, create and change various items of a data project like containers, access invitations, NFS exports, etc. To get a list of available subcommands, just call the help command in interactive mode or use the -h switch in non-interactive mode.

In order to get help on a particular command, you can just call the subcommand, followed by the -h switch.

(dsscli) dss container list -h		<---- You can also use dsscli dss container list -h for non-interactive version
usage: dss container list [-h] [-f {csv,json,table,value,yaml}] [-c COLUMN]
                          [--max-width <integer>] [--fit-width]
                          [--print-empty] [--noindent]
                          [--quote {all,minimal,none,nonnumeric}]
                          [--poolname [POOLNAME]]
                          [projectname]

List all available DSS data containers.

positional arguments:
  projectname

optional arguments:
  -h, --help            show this help message and exit
  --poolname [POOLNAME]

output formatters:
  output formatter options
  -f {csv,json,table,value,yaml}, --format {csv,json,table,value,yaml}
                        the output format, defaults to table
  -c COLUMN, --column COLUMN
                        specify the column(s) to include, can be repeated

table formatter:
  --max-width <integer>
                        Maximum display width, <1 to disable. You can also use
                        the CLIFF_MAX_TERM_WIDTH environment variable, but the
                        parameter takes precedence.
  --fit-width           Fit the table to the display width. Implied if --max-
                        width greater than 0. Set the environment variable
                        CLIFF_FIT_WIDTH=1 to always enable
  --print-empty         Print empty table if there is no data to show.

json formatter:
  --noindent            whether to disable indenting the JSON

CSV Formatter:
  --quote {all,minimal,none,nonnumeric}
                        when to include quotes, defaults to nonnumeric

Note that the output of the commands is usually optimised for human readability. However, if you want to script certain things, you might be interested using the -f switch to change the output formatting to something, which is easier to parse by a script.

Introducing the DSSWeb Management API

Currently the API still lacks some important documentation information. Though, we are working hard on extending this documentation in the future. You can access the current documentation via https://dssweb.dss.lrz.de/api/docs/.

If you are interested in using our API, please raise an Incident for 'Datenhaltung → Data Science Storage' at the LRZ Servicedesk.

Understanding Data Projects and Data Pools

In the LRZ Data Science Management Concept, we use Data Projects as organizational envelopes around the data, which is stored in the context of this data project. This is particularly useful as it implements a link between the data and the people who are formally responsible for the data and therefore protects us from orphaned data.

A particular data project is managed by a group of users, which is called the data curator group of the project. By default only the master users of the project belong to this group, but upon request via the Servicedesk, LRZ can also add arbitrary other users to the list of data curators for a data project.

In order to be able to store data in the context of a data project, the project also needs grants on one or more data pools. Data pools are physical storage systems which are integrated into the LRZ Data Science Management Ecosystem. Currently only Data Science Storage systems are supported, but it is planned to add other systems like an Archive for example in the future. Grants specify which fraction of a particular data pool (a quota), a particular data project is allowed to use. These grants are managed by LRZ.

Getting information about the data projects of a data curator

(dsscli) project list
+----+--------+
| ID | Name   |
+----+--------+
|  1 | pr53da |
|  2 | pr53ve |
|  3 | pr74cu |
|  4 | pr74qo |
|  5 | pr74wa |
+----+--------+

(dsscli) project show pr74qo
+---------------+------------------------------------------+
| Field         | Value                                    |
+---------------+------------------------------------------+
| ID            | 4                                        |
| Name          | pr74qo                                   |
| Curator Group | pr74qo-curators                          |
| Status        | Active                                   |
| Description   | DAT DSS Testprojekt                      |
| Organisation  | BAdW                                     |
| Project Start | 2016-08-03                               |
| Project End   | 2018-12-31                               |
| URL           | http://localhost:8001/v1/dataprojects/4/ |
+---------------+------------------------------------------+

Getting information about the grants for a particular data project

(dsscli) dss pool list pr74qo
+-----+---------+---------------+
|  ID | Project | Name          |
+-----+---------+---------------+
| 250 | pr74qo  | LRZ_PILOT     |
| 251 | pr74qo  | LRZ_PILOT_LWP |
+-----+---------+---------------+

(dsscli) dss pool show pr74qo 'LRZ_PILOT'
+----------------------------+------------------------------------------------------------------------------------------------+
| Field                      | Value                                                                                          |
+----------------------------+------------------------------------------------------------------------------------------------+
| ID                         | 250                                                                                            |
| Name                       | LRZ_PILOT                                                                                      |
| Project                    | pr74qo                                                                                         |
| Description                | LRZ DSS pilot system. 1 x DDN SFA12KX, 290 x 8TB NL SAS for data, 8 x 800GB SSD for meta data. |
| Hints                      | Optimized for large sequential workloads. Mean file size > 4MB. Minimum allocation unit 128KB. |
| Owner                      | LRZ                                                                                            |
| Quota GB                   | 10240                                                                                          |
| Free GB                    | 6144                                                                                           |
| Used GB                    | 4096                                                                                           |
| Used GB Percentage         | 40.0                                                                                           |
| Quota Files                | 2000000                                                                                        |
| Free Files                 | 1660000                                                                                        |
| Used Files                 | 340000                                                                                         |
| Used Files Percentage      | 17.0                                                                                           |
| Quota Containers           | 10                                                                                             |
| Free Containers            | 6                                                                                              |
| Used Containers            | 4                                                                                              |
| Used Containers Percentage | 40.0                                                                                           |
| Is Lightweight Pool        | False                                                                                          |
+----------------------------+------------------------------------------------------------------------------------------------+

Understanding Containers

Containers are the organizational elements, in which your actual data will live. Depending on the type of the underlying storage system, containers could be implemented in very different ways. For the Data Science Storage (DSS) systems, a container is basically a directory in a POSIX file system, with some additional magic attached to it. For a future archive system or an object storage system, a container will most likely be implemented in a different way. However, what is important to note is the basic concept of a container, which will be equal across all potential underlying storage systems:

1. A container is some kind of storage element with a certain size, which lets you store data in it.
2. A container should be used to encapsulate data, which semantically belongs together.
3. You can invite arbitrary users, which exist in the LRZ Identity Management System to access the data in the container.

Understanding Data Science Storage Containers

As already discussed, data containers on a Data Science Storage system are implemented as a directory on a POSIX file system, which is owned by a POSIX group, dedicated to the container. Each container has a certain limitation on the capacity and number of files, which can be stored in the container. The members of the container group are managed by the data curators via an invitation process. The access to the container is managed via POSIX rights and NFSv4 type ACLs. Also each container is associated with a so called ID Streamlining Mode. This policy defines how hard the system will try to make sure that access rights and ACLs stay in alignment with the access rights concept. Last but not least, each DSS type container is associated with a so called protection policy, which defines if and how the data in a container is protected by regular backups to the LRZ Backup- and Archive system.

Understanding Data Science Storage Container ID Streamlining Modes

• The most basic mode is called NONE. Using this policy, the container directory will be created with owning user root and owning group <container group> and sets the POSIX rights to rwxrws—. This will make sure that directories and files created within the container will automatically inherit the container group setting and that directories created within the container will inherit the setgid bit. However, every user can change the permissions of their files and directories using chmod and therefore overwrite the default settings easily. So the use case for this policy is usually that you want to let your users manage the access rights within the container by themselves and we only make sure that access at the container level is restricted to the invited container users.
• A more restrictive but still almost POSIX compliant mode is called NORMAL. This is also the default mode, you should choose if you are in doubt. This implements the same POSIX mode bits as the NONE mode, but in addition sets a set of NFSv4 style ACLs on the container directory, that will automatically be inherited by all files and directories, created in the container, which makes sure that all users belonging to the container group have rw- rights on files and rwx rights on directories. The point where this setting deviates from the POSIX or NFSv4 semantic is that if a user changes the POSIX mode bits using chmod, we do not replace the ACL by the new mode bits, but instead only update the ACL and therefore keep our access right settings for the container group intact. So basically the only way for a user to overwrite the permissions on a file he owns, is by explicitly editing the NFSv4 style ACL.
• A even more restrictive and barely POSIX compliant mode is called STRICT. This implements the same techniques described for the NORMAL mode, but disallows the use of chmod completely. As this happens on the file system level, this also holds true for each program library call, that uses a chown POSIX API call. So users should not be able to remove the setgid bit. However, by editing the ACLs, they still can overwrite the permissions.
• A special version of the mode NONE is the WORK policy. From a container perspective both are equivalent. However for WORK, DSS will create a dedicated user directory with POSIX rights rwx------ for each invited user.

Understanding Container Access Rights Management

For managing container access rights, we follow an invitation approach as you may already be familiar from cloud storage services like the LRZ Sync+Share service or Dropbox, Google Drive, etc. This means, for example, if data curator Alice wants to give user Bob access to the data container Cont-A, Alice will tell DSSWeb to invite Bobs username(s) to access container Cont-A. DSSWeb will then send an E-Mail, containing a unique invitation link to the E-Mail address assigned to Bobs username in the LRZ Identity Management System. Bob can then accept the invitation by clicking on the invitation link and accepting our data privacy and usage terms. After that, Bobs username will be automatically placed in the container POSIX group of container Cont-A and Bob is ready to access the data in Cont-A. However, if Bob chooses not to accept the invitation for whatever reason, the pending invitation will automatically expire after 30 days in DSSWeb.

In certain constellations, a data curator may want to delegate the access rights management for a single container to another user, but not give him or her data curator rights for the whole data project. In this case, the data curator can set the Container Manager flag for the invitation of the user, which will give him access to the DSSWeb Portal and allow him to create/read/update/delete Invitations and Auto Group Links as well as NFS exports for the particular container he was granted the Container Manager right.

Understanding Container Access Revocation and User Deletion Actions

As you may have noticed, the semantics of Data Containers are designed as collaborative space, in which every user can access all data. So semantically the data is not owned by an individual, but the whole group. However, as we operate on a POSIX file system, the POSIX semantics dictate that for each file there has to be a single user who is the formal owner of the file. Technically this is implemented by storing the UID of the owning user for each file in the metadata of the file. 

Now let's look at the case when a data curator wants to revoke access of a certain user to a container. Therefore, we remove the user from the container group and because of the permissions set on the container level, the user cannot access any data in the container anymore. However, there may now still be files in the container which are formally owned by the revoked user. While this is technically insignificant - since the user cannot access the data in the container anymore - it may be misleading and ambiguous to the remaining users. The same holds true if a user, invited to a container gets completely deleted. In this case, even resolving the UID to a user name will not work anymore and you will be presented with a cryptic number as owner of the files in your container.

In order to avoid these kinds of problems, whenever you revoke access for a user or a user get's deleted in the Identity Management System, we automatically start a batch job, that changes ownership of all files, which belong to the revoked or deleted user. Please note that this will not happen if the container ID Streamlining Mode policy is set to NONE. Currently the new owner is automatically determined from the list of data curators, whereby we choose that user, which is the first one in an alphabetical order. Note that this happens asynchronously so there may be a noticeable delay between revocation/deletion and ownership change.

Understanding Automatic Container Access Group Links

There may be situations where explicitly managing DSS Container Access Rights for each individual user may represent unnecessary overhead, because you want to give one or more particular groups automatically access to a container. So when a new member is added to the group, he should automatically be invited to access the container and when a member is removed from the group he should automatically be revoked access rights.

You can achieve this by defining so called Auto Group Links for a container. We currently support to link groups from DSS, TUMonline, LMU, LinuxCluster and plan to extend this to SuperMUC-NG groups in the future. For more details on this, please check out our Knowledge Base article on this topic.

Understanding Data Science Storage Container Protection Policies

• The most simple data protection policy is called NONE. Containers which are assigned this protection policy are not protected at all. Use this policy only for data, which can easily be regenerated or recaptured.
• For automatic backups, which are performed once a week, use the policy BACKUP_WEEKLY. This will perform an incremental backup run of your container once a week (usually during the weekend). The retention policies of the backup are 180 days and a maximum of 3 versions per file. For more information about DSS backup retention policies, see DSS Understanding Data Science Storage Container Backup Retention.
• For automatic backups, which are performed once a day, use the policy BACKUP_DAILY. This will perform an incremental backup run of your container every day (usually during night). The retention policies of the backup are 180 days and a maximum of 3 versions per file. For more information about DSS backup retention policies, see DSS Understanding Data Science Storage Container Backup Retention.

For containers which exclusively store static, read-only content, like simulation results, output data from instruments like genomic sequencers, microscopes, NMR spectroscopes, etc, we also provide special ARCHIVE policies. These policies help you to be compliant with the DFG rules of good scientific practice, which require you to retain scientific primary data for 10 years. This is implemented by assigning the backed up data of your container a special retention policy, that keeps backups for 10 years and also keeps two copies of the backed up data in two different locations. Please note the restrictions that apply when using this data protection policy: DSS Understanding Data Science Storage Container Backup Retention. 

• For automatic incremental archives of static data, which are performed once a week, use the policy ARCHIVE_WEEKLY. This will perform an incremental archive run of your container once a week (usually during weekend).
• For automatic incremental archives of static data, which are performed once a day, use the policy ARCHIVE_DAILY. This will perform an incremental archive run of your container once a day (usually during night).
  
  

For information on how to restore backed up files, please check this Knowledge Base article.

Understanding Data Science Storage Container Sharing with External Users

In addition to sharing container data with LRZ/TUM/LMU users, you can also share your container data with arbitrary external users, using the 3rd party Globus Sharing service. We have integrated Globus Sharing via their API into the DSSWeb Self-Service portal for your convenience, so you can easily manage Globus Sharing permissions via DSSWeb. However, in order to use Globus Sharing on a particular container, we have to do some special preparations, which are not necessary if you don't need Globus Sharing. Therefore, you have to explicitly enable Globus Sharing on your container. This can either be done at container creation time or anytime afterwards. You can also disable Globus Sharing at any time. After Globus Sharing has been enabled on your container, you can invite arbitrary external users to access your data via Globus by just specifying their Globus User ID or E-Mail address. They will then receive an invitation Mail from Globus that provides the necessary instructions in order to access the shared data.

For details on how Globus Sharing and the integration into DSS and DSSWeb works, please check this Knowledge Base article.

Understanding the Asynchronous Task Model

As we already described earlier, DSSWeb takes your high level requests (like create a new container) and automatically executes the necessary steps to fulfil your request. Every time you tell DSSWeb to create or change something, it must talk to one or more external systems, like LRZ SIM, LRZ's Backup and Archive System, the underlying filesystems of DSS and so on and ask them to execute one or more tasks. So under the covers, DSSWeb is a fairly complex, highly distributed system which depends on multiple systems working seamlessly together. However, in such highly distributed systems, we have to accept that things may fail from time to time. Fortunately, these failures often have a temporary and very seldom a permanent nature. For example, think of the situation in which one of the subsystems is not available for whatever reason. This situation is usually fixed very fast and then everything works just fine again.

In order to hide all this complexity and not to bother you with temporary errors, thereby giving you the best possible user experience, we work with an asynchronous task model. That means, whenever you tell DSSWeb to create or change something, it will just store the necessary information in its database and make sure that an asynchronous task is started in the background. So while DSSWeb may still be busy, fulfilling your request, you can already go on and do other suff and then check back later to see if your request was carried out successfully.

You can check the status of an object (like a container), by looking at it's Status and Active Task fields. Typically when you first create an object it goes to a CREATE_PENDING state, meaning the object is currently being setup. When an object reaches a "steady state", meaning it was successfully created, its status is usually ACTIVE. When you change an object, it will go to a CHANGE_PENDING state and as soon as the changes have been carried out it will go back to ACTIVE again. The same semantics often are also true for deletion of objects, meaning the object will got to a DELETE_PENDING state and only after it has been successfully cleaned up on the subsystems, it is removed from the DSSWeb database.

In the GUI, the status is usually represented via Icons. You can get a textual representation of the state by simply put your mouse pointer on such a status icon.

Understanding Task Failures

Usually, when an asynchronous task hits a failure in a subsystem, the only thing you should notice is that the task may take longer than normal to execute (there is no special state indicating a problem for an object, it will simply stay in the PENDING state). The implemented semantics are that whenever a task encounters a failure, it will retry after some minutes until the task can finally be finished. DSSWeb monitors the execution times and number of retries of tasks and in case the retry count of a task gets unusually high, it will notify LRZ personal to take a look. (We currently also think about automatically raising an Incident at the LRZ Servicedesk for you, so that you get feedback on the status of the fixing procedures.)

Understanding our Software Development Process

Our software development process is loosely based on an agile approach and is highly driven by the demands of our customers. So, if you are missing a particular feature or would like to see something particular to be implemented in DSSWeb, please don't hesitate and reach out to us by opening a request on the LRZ Servicedesk so we can discuss your requirements and record it in our product backlog.

Managing Data Curators

When a LRZ project receives the right to use DSS, the project will be automatically imported to DSSWeb and all Master Users of the LRZ project will be designated as initial data curators for the project in DSSWeb and receive a welcome mail from DSSWeb. However, from there on the roles of Master User and Data Curator are not linked anymore. This means that the data curators can manage the list of data curators for their particular project in DSSWeb independent from the Master Users in SIM. On the other side, changes of the Master User in SIM will not be reflected in DSSWeb. For every data project, there can be zero or one primary data curators. If a data curator has been assigned the primary role, whenever a user is removed from the list of invited users or the user is deleted in the LRZ SIM user directory, DSS will change ownership of the files owned by this user to the primary data curator. If no primary data curator is assigned for a project, the data curator to which file ownership will be changed to is arbitrarily chosen from the list of available data curators.

Showing Data Curators

In the following, we show you how to get an overview of the current data curators of your project.

(dsscli) datacurator list --projectname pr74qo 
+-----+--------------+----------+-----------+-----------+------------+------------+
|  ID | Data Project | Username | Firstname | Lastname  | Is primary | Created    |
+-----+--------------+----------+-----------+-----------+------------+------------+
|   3 | pr74qo       | a2822xx  | Some      | Name      | True       | 2019-11-04 |
|   4 | pr74qo       | di39xxx  | Some      | Othername | None       | 2019-11-04 |
+-----+--------------+----------+-----------+-----------+------------+------------+

(dsscli) datacurator show 3
+--------------+------------+
| Field        | Value      |
+--------------+------------+
| ID           | 3          |
| Data Project | pr74qo     |
| Username     | a2822xx    |
| Firstname    | Some       |
| Lastname     | Name       |
| Is primary   | True       |
| Created      | 2019-11-04 |
+--------------+------------+

Adding new Data Curators to a project

In the following, we show how you can add additional data curators to your project. When you add a user as data curator, the user will receive an automatically generated welcome mail from DSSWeb that informs him about the granted right and points him to this documentation.

(dsscli) datacurator create --projectname pr74qo --username a2822xx
+--------------+------------------+
| Field        | Value            |
+--------------+------------------+
| ID           | 627              |
| Data Project | pr74qo           |
| Username     | a2822xx          |
| Firstname    | LRZ-SIM          |
| Lastname     | Funktionskennung |
| Created      | 2019-11-06       |
+--------------+------------------+

Updating a Data Curator

In the following, we show you how you can change the primary data curator for a project.

(dsscli) datacurator update --primary 3
+--------------+------------+
| Field        | Value      |
+--------------+------------+
| ID           | 3          |
| Data Project | pr74qo     |
| Username     | a2822xx    |
| Firstname    | Name       |
| Lastname     | Somename   |
| Is primary   | True       |
| Created      | 2019-11-04 |
+--------------+------------+

(rdsscli) datacurator update --notprimary 3
+--------------+------------+
| Field        | Value      |
+--------------+------------+
| ID           | 3          |
| Data Project | pr74qo     |
| Username     | a2822xx    |
| Firstname    | Name       |
| Lastname     | Somename   |
| Is primary   | False      |
| Created      | 2019-11-04 |
+--------------+------------+

Removing a Data Curator from a project

In the following, we show how you can remove a data curator from a project. When you remove a user as data curator of a project, he will only use the data curator right for this project. Any container access invitations of this user will not be touched. If you also want to revoke access to the data of the project, you must also delete the invitations of the user in the individual containers.

(dsscli) datacurator delete --projectname pr74qo --username a2822xx

Managing Containers

In the following we describe how you can create a new DSS Container for your data project.

Creating Containers

(dsscli) dss container create --projectname pr74qo --poolname 'LRZ_PILOT_LWP' --size=1000 --files 50000 --idmode STRICT --description 'DSS tutorial container' --globus-sharing OFF --dpmode NONE
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| ID                   | 291                                  |
| Name                 | pr74qo-dss-0005                      |
| Description          | DSS tutorial container               |
| Status               | CREATE PENDING                       |
| Active Task          | fb571b27-467c-425f-b4d9-4dbdd4a752ef |
| Read/Write Group     | pr74qo-dss-0005                      |
| Read Only Group      | pr74qo-dss-0005-ro                   |
| Quota GB             | 1000                                 |
| Quota Files          | 50000                                |
| ID Streamline Mode   | STRICT                               |
| Data Protection Mode | NONE                                 |
| Globus Sharing       | False                                |
| Globus Endpoint ID   | None                                 |
+----------------------+--------------------------------------+

Showing Containers

(dsscli) dss container list pr74qo
+-----+---------+---------------+-----------------+----------------+
|  ID | Project | Pool          | Name            | Status         |
+-----+---------+---------------+-----------------+----------------+
| 284 | pr74qo  | LRZ_PILOT     | pr74qo-dss-0000 | ACTIVE         |
| 285 | pr74qo  | LRZ_PILOT     | pr74qo-dss-0001 | ACTIVE         |
| 287 | pr74qo  | LRZ_PILOT     | pr74qo-dss-0003 | ACTIVE         |
| 288 | pr74qo  | LRZ_PILOT     | pr74qo-dss-0004 | ACTIVE         |
| 291 | pr74qo  | LRZ_PILOT_LWP | pr74qo-dss-0005 | CREATE PENDING |
+-----+---------+---------------+-----------------+----------------+

(dsscli) dss container list pr74qo --poolname 'LRZ_PILOT_LWP'
+-----+---------+---------------+-----------------+----------------+
|  ID | Project | Pool          | Name            | Status         |
+-----+---------+---------------+-----------------+----------------+
| 291 | pr74qo  | LRZ_PILOT_LWP | pr74qo-dss-0005 | CREATE PENDING |
+-----+---------+---------------+-----------------+----------------+

(dsscli) dss container show pr74qo-dss-0001
+--------------------------+--------------------------------------+
| Field                    | Value                                |
+--------------------------+--------------------------------------+
| ID                       | 285                                  |
| Project                  | pr74qo                               |
| Pool                     | LRZ_PILOT                            |
| Name                     | pr74qo-dss-0001                      |
| Path                     | /dss/dssfs01/pr74qo/pr74qo-dss-0001  |
| Description              | Testcontainer                        |
| Status                   | ACTIVE                               |
| Active Task              | None                                 |
| Read/Write Group         | pr74qo-dss-0001                      |
| Read Only Group          | pr74qo-dss-0001-ro                   |
| Quota GB                 | 1024                                 |
| Used GB                  | 9                                    |
| Used GB Percentage       | 0.88                                 |
| Quota Files              | 100000                               |
| Used Files               | 10014                                |
| Used Files Percentage    | 10.01                                |
| ID Streamline Mode       | STRICT                               |
| Data Protection Mode     | BACKUP DAILY                         |
| Globus Sharing           | True                                 |
| Globus Endpoint ID       | 88741c2c-feee-11e9-9947-0a8c187e8c12 |
| Is Lightweight Container | False                                |
+--------------------------+--------------------------------------+

(dsscli) dss container show id 285
+--------------------------+--------------------------------------+
| Field                    | Value                                |
+--------------------------+--------------------------------------+
| ID                       | 285                                  |
| Project                  | pr74qo                               |
| Pool                     | LRZ_PILOT                            |
| Name                     | pr74qo-dss-0001                      |
| Path                     | /dss/dssfs01/pr74qo/pr74qo-dss-0001  |
| Description              | Testcontainer                        |
| Status                   | ACTIVE                               |
| Active Task              | None                                 |
| Read/Write Group         | pr74qo-dss-0001                      |
| Read Only Group          | pr74qo-dss-0001-ro                   |
| Quota GB                 | 1024                                 |
| Used GB                  | 9                                    |
| Used GB Percentage       | 0.88                                 |
| Quota Files              | 100000                               |
| Used Files               | 10014                                |
| Used Files Percentage    | 10.01                                |
| ID Streamline Mode       | STRICT                               |
| Data Protection Mode     | BACKUP DAILY                         |
| Globus Sharing           | True                                 |
| Globus Endpoint ID       | 88741c2c-feee-11e9-9947-0a8c187e8c12 |
| Is Lightweight Container | False                                |
+--------------------------+--------------------------------------+

Anker
contmodify contmodify

Modifying Containers

In the following we show how certain properties of containers can be changed. Currently you can change the following properties of an existing container:

• The maximal size of the container (Quota GB)
• The maximal number of files of the container (Quota Files)
• The description
• Whether Globus Sharing is enabled or disabled
• The data protection mode

(dsscli) dss container update pr74qo-dss-0001 --size 2000 --files 200000 --description 'Testcontainer updated' --globus-sharing OFF --dpmode BACKUP_WEEKLY
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| ID                   | 285                                  |
| Name                 | pr74qo-dss-0001                      |
| Description          | Testcontainer updated                |
| Status               | CHANGE PENDING                       |
| Active Task          | 86f81a4e-2022-4111-aa28-55481cf8fda5 |
| Read/Write Group     | pr74qo-dss-0001                      |
| Read Only Group      | pr74qo-dss-0001-ro                   |
| Quota GB             | 2000                                 |
| Quota Files          | 200000                               |
| ID Streamline Mode   | STRICT                               |
| Data Protection Mode | BACKUP WEEKLY                        |
| Globus Sharing       | False                                |
| Globus Endpoint ID   | 88741c2c-feee-11e9-9947-0a8c187e8c12 |
+----------------------+--------------------------------------+

Delete Containers

In the following we show how you can delete a DSS container.

(dsscli) dss container delete pr74qo-dss-0004


!!! ATTENTION POSSIBLE DANGEROUS OPERATION !!!

You are about to remove data container pr74qo-dss-0004.
This will DELETE ALL DATA in the container INCLUDING ALL BACKUP and ARCHIVE copies.
This operation is final and cannot be stopped or rolled back.


If you are really sure that you want to delete this container, please confirm
by typing in the container name:

Confirm => pr74qo-dss-0004
Containernames match. Initiating deletion of pr74qo-dss-0004 in:
10 ...
9 ...
8 ...
7 ...
6 ...
5 ...
4 ...
3 ...
2 ...
1 ...
You have been warned ...
Successfully initiated deletion of container pr74qo-dss-0004!

Managing Container Access Rights 

Anker
internalsharing internalsharing

As discussed, Data Science Containers are collaborative spaces. As such, you can grant virtually any user, which is known in the LRZ Identity Management, access to a data container. As the Identity Management Systems of LRZ, TUM, LMU are kind of federated, this also means that you can grant any user, managed by one of these IdMs access to your data containers. 

Granting Access Rights

In order to allow an user to access a particular data container, you must invite him or her to the container group.

(dsscli) dss invitation create --containername pr74qo-dss-0005 --username a28ditix --accessmode READ_WRITE --maxgb 100 --maxfiles 10000 --ismanager NO
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| ID          | 4600                                 |
| Container   | pr74qo-dss-0005                      |
| User        | a28ditix                             |
| Quota GB    | 100                                  |
| Quota Files | 10000                                |
| Access Mode | RW                                   |
| Is Manager  | False                                |
| Status      | CREATE PENDING                       |
| Active Task | 6a373b1b-0ec3-4242-bacd-0b70436edefa |
| Expires At  | 2019-12-04                           |
| Inviter     | a2822xx                              |
+-------------+--------------------------------------+

Showing Access Rights

(dsscli) dss invitation list pr74qo 
+----+-----------------+---------+-------------+---------------------+
| ID | Container       | User    | Access Mode | Status              |
+----+-----------------+---------+-------------+---------------------+
| 80 | pr74qo-dss-0000 | a2828xx | RW          | USER ACCEPTED       |
| 81 | pr74qo-dss-0000 | di57gix | RW          | USER ACCEPTED       |
| 82 | pr74qo-dss-0001 | a2828xx | RW          | USER ACCEPTED       |
| 83 | pr74qo-dss-0001 | di57gix | RW          | USER ACCEPT PENDING |
+----+-----------------+---------+-------------+---------------------+

(dsscli) dss invitation list --containername pr74qo-dss-0000
+----+-----------------+---------+-------------+---------------------+
| ID | Container       | User    | Access Mode | Status              |
+----+-----------------+---------+-------------+---------------------+
| 80 | pr74qo-dss-0000 | a2828xx | RW          | USER ACCEPTED       |
| 81 | pr74qo-dss-0000 | di57gix | RW          | USER ACCEPTED       |
+----+-----------------+---------+-------------+---------------------+

(dsscli) dss invitation list --username a2828xx
+----+-----------------+---------+-------------+---------------------+
| ID | Container       | User    | Access Mode | Status              |
+----+-----------------+---------+-------------+---------------------+
| 80 | pr74qo-dss-0000 | a2828xx | RW          | USER ACCEPTED       |
| 82 | pr74qo-dss-0001 | a2828xx | RW          | USER ACCEPTED       |
+----+-----------------+---------+-------------+---------------------+

(dsscli) dss invitation show pr74qo-dss-0001 a2822xx
+-----------------------+----------------------------------+
| Field                 | Value                            |
+-----------------------+----------------------------------+
| ID                    | 4548                             |
| Container             | pr74qo-dss-0001                  |
| User                  | a2822xx                          |
| Firstname             | Some                             |
| Lastname              | Usersname                        |
| Quota GB              | None                             |
| Used GB               | 9                                |
| Used GB Percentage    | 0.44999999999999996              |
| Quota Files           | None                             |
| Used Files            | 10011                            |
| Used Files Percentage | 5.0055000000000005               |
| Access Mode           | RW                               |
| Is Manager            | False                            |
| Status                | USER ACCEPTED                    |
| Active Task           | None                             |
| Expires At            | None                             |
| Inviter               | AutoCuratorInviter               |
| Created               | 2019-11-04                       |
| Last Updated          | 2019-11-06T08:15:14.130042+01:00 |
+-----------------------+----------------------------------+

(dsscli) dss invitation show id 4548
+-----------------------+----------------------------------+
| Field                 | Value                            |
+-----------------------+----------------------------------+
| ID                    | 4548                             |
| Container             | pr74qo-dss-0001                  |
| User                  | a2822xx                          |
| Firstname             | Some                             |
| Lastname              | Usersname                        |
| Quota GB              | None                             |
| Used GB               | 9                                |
| Used GB Percentage    | 0.44999999999999996              |
| Quota Files           | None                             |
| Used Files            | 10011                            |
| Used Files Percentage | 5.0055000000000005               |
| Access Mode           | RW                               |
| Is Manager            | False                            |
| Status                | USER ACCEPTED                    |
| Active Task           | None                             |
| Expires At            | None                             |
| Inviter               | AutoCuratorInviter               |
| Created               | 2019-11-04                       |
| Last Updated          | 2019-11-06T08:15:14.130042+01:00 |
+-----------------------+----------------------------------+

Modify Access Rights

In the following we show how certain properties of invitations can be changed. Currently you can change the following properties of an existing invitation:

• For invitations that have not yet been accepted:
  • Prolong expiration date
  • Resend the invitation mail
  • The maximum amount of GB, the user is allowed to consume within the container.
  • The maximum number of files, the user is allowed to create within the container. 
  • If the invited user shall be a manager for this container, which will give him the possibility to manage invitations, auto group links and NFS exports for this container.
• For invitations that have already been accepted:
  • The access mode to grant. This can either be READ_WRITE or READ_ONLY. However, the later can only be used with containers that implement an ID Stremline Mode of STRICT.
  • The maximum amount of GB, the user is allowed to consume within the container.
  • The maximum number of files, the user is allowed to create within the container. 
  • If the invited user shall be a manager for this container, which will give him the possibility to manage invitations, auto group links and NFS exports for this container. (Default: no)

(dsscli) dss invitation update 83 --prolong
+-------------+---------------------+
| Field       | Value               |
+-------------+---------------------+
| ID          | 83                  |
| Container   | pr74qo-dss-0001     |
| User        | di57gix             |
| Quota GB    | None                |
| Quota Files | None                |
| Access Mode | RO                  |
| Status      | USER ACCEPT PENDING |
| Active Task | None                |
| Expires At  | 2018-01-04          |
| Inviter     | a2828yy             |
+-------------+---------------------+

(dsscli) dss invitation update 83 --resend
+-------------+---------------------+
| Field       | Value               |
+-------------+---------------------+
| ID          | 83                  |
| Container   | pr74qo-dss-0001     |
| User        | di57gix             |
| Quota GB    | None                |
| Quota Files | None                |
| Access Mode | RO                  |
| Status      | USER ACCEPT PENDING |
| Active Task | None                |
| Expires At  | 2018-01-04          |
| Inviter     | a2828yy             |
+-------------+---------------------+

(dsscli) dss invitation update 83 --accessmode READ_WRITE --maxgb 1000 --maxfiles 10000000 --ismanager YES
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| ID          | 83                                   |
| Container   | pr74qo-dss-0001                      |
| User        | di57gix                              |
| Quota GB    | 1000                                 |
| Quota Files | 10000000                             |
| Access Mode | RW                                   |
| Is Manager  | True                                 |
| Status      | CHANGE PENDING                       |
| Active Task | 84a85e4a-933f-464c-aa70-273efe32ccc6 |
| Expires At  | 2017-12-07                           |
| Inviter     | a2828yy                              |
+-------------+--------------------------------------+

Revoking Access Rights

In the following, we show how existing invitations can be revoked/deleted. You can revoke accepted as well as pending invitations. If you revoke a pending invitation, the invitation link, sent to the person you invited will be invalidated, so accepting the invitation is no longer possible. If you revoke an already accepted invitation, the user will be removed from the container access group and file ownership of each file, technically owned by the user, will be transfered to a data curator. See Understanding Container Access Revocation and User Deletion Actions for more information on the topic.

(dsscli) dss invitation delete pr74qo-dss-0001 di57gix
Successfully initiated deletion of invitation 83!

Managing DSS Container Access Rights for External Users via Globus

If you need to share data with people who do not have a user account at LRZ, LMU or TUM, you can do so, as discussed above, using the Globus Sharing feature of the Globus Data Management Portal.

Granting Globus Sharing Access Rights

In order to allow a user to access a particular data container or a particular subdirectory of a data container, you must invite him or her via Globus Sharing to access the data. In Globus terms this is called "creating a Globus Sharing ACL".