This page provides a historically structured record of updates to the AI Systems. The information is made available for transparency, allowing you to see when changes were introduced and how the system has evolved.

Note: We updated our changelog documentation to align with the LRZ Status RSS feed. As a result, entries older than 2026 may use a different formatting.

Ups, es scheint, dass Sie eine Tabelle oder ein Makro, das eine Tabelle generiert, innerhalb des Tabellenfilter-Makros platzieren müssen.

AI Systems on LRZ Service Status Ankündigungen und Vorfälle
[Störung] 2026-07-15 - AI Systems under unscheduled maintenance from July 9, 15:00 until further notice. Do., 09.07.2026 15:00 – Mo., 13.07.2026 10:00 Betroffene Services: [AI Systems] We are continuing to monitor the situation and will provide updates as soon as new information becomes available. We currently have no updates to share. We are actively working to restore the AI Systems as quickly as possible. We currently have no further updates to share, but we will keep you informed as soon as new information becomes available. Thank you for your patience and understanding. The AI Systems have entered an unplanned maintenance period in response to an security advisory. As part of this unscheduled maintenance, all currently running jobs will continue to run and, at this time, will not be terminated. Jobs that were queued but had not yet started will remain in the queue and will be scheduled automatically. We sincerely apologize for the disruption and will keep you informed as new information becomes available. In the meantime, we kindly ask for your patience and understanding.
[Behoben] [Störung] 2026-06-30 - AI Systems was undergoing unscheduled maintenance from June 29 15:00 to June 30 12:00. Mo., 29.06.2026 15:00 – Di., 30.06.2026 10:00 Betroffene Services: [AI Systems] The AI Systems have entered an unplanned maintenance in response to an urgent advisory. We expect the system to be available again on Tuesday, June 30, at 10:00. As part of the unscheduled maintenance, all running jobs were terminated and will need to be restarted once the AI Systems are back online. Jobs that were queued but had not yet started will remain in the queue and will be scheduled automatically after the AI Systems resume operation. We sincerely apologize for the disruption and will keep you informed as new information becomes available. In the meantime, we kindly ask for your patience and understanding.
[Abgeschlossen] [Wartung] 2026-06-24 - AI Systems was under maintenance from June 22 to 24, 2026. Mo., 22.06.2026 08:00 – Mi., 24.06.2026 18:00 Betroffene Services: [AI Systems] The following changes were introduced: Operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes. Upgraded the GPFS parallel file system to version 5.3.2-4 for security and reliability improvements. Upgraded the GPU software stack to NVIDIA Driver 580 and CUDA 13 for support of the latest AI software and CUDA features.
[Behoben] [Störung] 2026-05-26 18:00 - Offline due to Power Outage Mo., 25.05.2026 04:00 – Di., 26.05.2026 18:00 Betroffene Services: [AI Systems] Recovery of the AI Systems infrastructure is currently ongoing following the recent power outage. Services and system components are being brought back online gradually and in a controlled manner. Please expect partial availability during the recovery phase. While many systems may already become reachable again, diagnostics, integrity checks, and validation procedures are still ongoing to ensure stable and reliable operation. Users may therefore continue to experience temporary interruptions, reduced availability, or delayed service restoration for individual components while recovery work continues. Further updates will follow as additional systems return to operation and diagnostics are completed. previous title: “Offline due to Power Outage, 25 May 2026” Due to a power grid outage, the AI Systems (including BayernKI and MCML) were unavailable between 25 May 04:00 CET and 26 May 2026 15:00 CET. Action required: Jobs that were running at the time of the outage were abruptly terminated. Please review your job queue and history, and manually resubmit any affected jobs.
[Behoben] [Störung] Unscheduled Maintenance, 19 May 2026 Fr., 15.05.2026 09:30 – Di., 19.05.2026 11:30 Betroffene Services: [AI Systems] The AI Systems login nodes were taken offline between 15.5.2026 09:30 CET and 19.5.2026 11:30 CET. System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority. Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation. Another maintenance window will be scheduled to apply upstream patches as they become available.
[Behoben] [Störung] Unscheduled Maintenance, 7 May 2026 Do., 07.05.2026 23:00 – Fr., 08.05.2026 15:00 Betroffene Services: [AI Systems] The AI Systems login nodes were taken offline between 7.5.2026 23:00 CET and 8.5.2026 15:00 CET. System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority. Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation. Another maintenance window will be scheduled to apply upstream patches as they become available.
[Behoben] [Störung] Unscheduled Maintenance, 30 Apr 2026 Do., 30.04.2026 09:00 – Do., 30.04.2026 17:00 Betroffene Services: [AI Systems] The AI Systems login nodes were taken offline between 30.4.2026 09:00 CET and 30.4.2026 17:00 CET. System access had to be blocked to address recently published operating system security vulnerabilities. All actively running jobs had to be terminated and will have to be rescheduled by the users. Queued jobs will resume by order of priority. Available mitigation measures have been rolled out on all system nodes. Log analysis indicates the system wasn’t exposed to publicly known attack mechanisms. Continuous assessment led to the decision to resume service operation. Another maintenance window will be scheduled to apply upstream patches as they become available.
[Abgeschlossen] [Wartung] 2026-03-18 - AI Systems was under maintenance from March 16 to 18, 2026. Mo., 16.03.2026 06:00 – Mi., 18.03.2026 18:00 Betroffene Services: [AI Systems] The following changes were introduced: Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes Networking interface & switch configurations have been homogenized and switch firmware has been updated Virtual infrastructure components have been moved to new backend
[Abgeschlossen] [Wartung] 2025-12-03 - AI Systems was under maintenance from December 1 to 3, 2025. Mo., 01.12.2025 06:00 – Mi., 03.12.2025 17:00 Betroffene Services: [AI Systems] The following changes were introduced: The AI Systems have been added to status.lrz.de and any future maintenance and service disruption announcements will be made public primarily via this channel Firmware, operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes The web-based frontend, Open OnDemand, has been updated to version 3.1.16 Data Science Storage configuration parameters have been updated

2025-3 Maintenance

The AI Systems (including the BayernKI and MCML system segments) will undergo a maintenance procedure between September 8th and 11th, 2025. On these days, the system will not be available to users. Normal user operation is expected to resume during the course of Thursday, September 11th.

Changed:

Firmware and operating system components and packages of all AI Systems nodes have been been updated providing stability and security fixes
GPFS storage components have been updated and the terrabyte filesystems have been made available on the cluster

2025 Reminder

The LRZ AI Systems have to undergo a short maintenance early next week (KW 31). For this, the system will be drained over the weekend and start-up of new jobs will be delayed until after the maintenance. We expect the actual downtime to not exceed more than 10 minutes. UPDATE: The maintenance concluded within the expected timeframe.

2025-2 Maintenance

The following changes were introduced during the maintenance procedure between May 19th and 21st, 2025.

Changed:

AI Systems access management has been fundamentally redesigned. Master users of projects with AI Systems access permissions can enable individual accounts by managing a relevant group within their project. Add/remove individual accounts to/from the <projectID>-ai-c group for AI Systems access, use of general system resources, and VPN access to the Munich Scientific Network (MWN). For details see 1. Access and <MCML> 3.0 Specifics for MCML Members
Various firmware components, the operating system and packages of all AI Systems nodes as well as the Nvidia GPU drivers have been been updated to recent point releases providing stability and security fixes
The web-based frontend, Open OnDemand, has been updated to version 3.1.11

2025-1 Maintenance

Various system components have been updated during the maintenance procedure between February 17th and 19th, 2025.

Added:

An additional node has been added to the web-based frontend environment of the AI Systems, accessible via https://login.ai.lrz.de.
<MCML> A new storage system has been added, accessible to MCML users. It is be usable as temporary/scratch disk space (for details see 3.0 Specifics for MCML Members).

Changed:

The web-based frontend, Open OnDemand, has been updated to version 3.1.10
Partition names of HGX H100 systems have been updated and unified to lrz-hgx-h100-94x4 and <MCML> mcml-hgx-h100-94x4
Firmware, the operating system kernel and packages of all AI Systems nodes as well as the Nvidia GPU drivers/components and GPFS storage applications have been been updated to recent point releases providing stability and security fixes

2024 Infrastructure Power Cut

NOTICE: The AI Systems will be affected by an infrastructure power cut scheduled in November 2024. The following system partitions will become unavailable for 3 days during the specified time frame. We apologise for the inconvenience associated with that.

Calendar Week 46, 2024-11-11 - 2024-11-13

lrz-v100x2
lrz-hpe-p100x4
lrz-dgx-1-p100x8
lrz-dgx-1-v100x8
lrz-cpu (partly)
test-v100x2
lrz-hgx-a100-80x4
mcml-hgx-a100-80x4
mcml-hgx-a100-80x4-mig

2024-3 Maintenance

Various system components have been updated during the maintenance procedure between September 30th and October 2nd, 2024:

Added:

New HGX-based nodes with H100 GPUs have been prepared, but are not yet user accessible - watch out for additional announcements

Changed:

Slurm Workload Manager has been updated to release 24.05.3
Configuration options affecting job submission limits, wall time and the fair share mechanism have been optimized and harmonized to facilitate higher throughput and reduced wait times (subject to ongoing adjustments)
Jupyter Notebook / JupyterLab container images have been updated and provide a new PyTorch version
The operating system kernel and packages of all AI Systems nodes have been been updated to recent point releases providing stability and security fixes
<MCML> Previous MIG nodes have been reconfigured and reallocated for general usage (for details see 3.0 Specifics for MCML Members)

2024-2 Maintenance

Various system components have been updated during the maintenance procedure on July 1st-3rd, 2024:

Added:

Recent additions of HGX-based nodes with A100 GPUs have been finalized (lrz-hgx-a100-80x4 and mcml-hgx-a100-80x4 partitions)
Some additional CPU resources have been made available as part of the lrz-cpu partition

Changed:

The Enroot container runtime has been updated to release 3.5.0
The web-based frontend, Open OnDemand, has been updated to version 3.1.7
Jupyter Notebook / JupyterLab container images have been updated
RStudio Server container images have been updated and provide new R / RStudio Server versions
The operating system kernel and packages of all AI Systems nodes as well as the Nvidia drivers and GPFS storage applications have been been updated to recent point releases providing stability and security fixes

Removed

RStudio Server container images with R versions prior to 4.4.0 have been removed; if absolutely necessary, these can still be provided and used as custom container images

2024-1 Maintenance

Multiple system components have been updated and there are various user-facing changes that were introduced during the maintenance procedure on March 11th-14th, 2024:

Breaking:

enroot start currently cannot be used directly with a sqsh container image. Instead, it requires an existing container. The following commands show an example of how to create a container and use enroot start:
```
enroot import <container-tag>  # when importing from a registry; skip if local image file is available
enroot create --name <container-name> <image-file>  # -n; this step may have been skipped previously
enroot start <container-name>
```
Alternatively, use the Pyxis --container-image option when using srun or in the preamble of your batch script (for additional details see Removed section below).

Added:

A "Globus" button has been added to the file manager application of the web-based frontend and provides direct access to the active directory within the Globus research management portal. This allows for improved file management and data transfer capabilities (for further details see Using DSS world wide via Globus Online)

Changed:

The operating system of all AI Systems nodes has been updated to Ubuntu 22.04 LTS / Nvidia DGX OS 6; various DGX firmware components have been updated
The Nvidia drivers have been updated to version R535
The login infrastructure has been reworked and fully virtualized to provide increased stability, redundancy and future-proof flexibility
The web-based frontend, Open OnDemand, has been updated to release 3.1.1
Jupyter Notebook / JupyterLab container images have been updated and provide new PyTorch and TensorFlow versions
RStudio Server container images have been updated and provide new R / RStudio Server versions (older versions have been thinned out)

Removed:

Due to a bug in Ubuntu 22.04's fuse-overlay package, it had to be removed. This breaks the possibility to start container images directly without the need to create containers first, as had become possible in recent Kernel versions (see Enroot's documentation and Breaking section above). We are exploring and evaluating various options for a future course of action.

2023-4 Maintenance

Various system components have been updated during the maintenance procedure on December 4th-6th, 2023:

General OS updates and system firmware updates
Slurm Workload Manager has been updated to release 23.02.6
The web-based frontend, Open OnDemand, has been updated to release 3.0.3

2023-3 Maintenance

The following list of user-facing changes was introduced during the maintenance procedure between July 24th and 25th, 2023:

The primary address of the web-based frontend has been changed. Use login.ai.lrz.de for all connections to the LRZ AI Systems. All previous addresses may still be functional, but are going to be removed in the future (deprecation notice).
The available CPU options for interactive applications in the web-based frontend have been adjusted for some cases/usage combinations.
Jupyter Notebook/JupyterLab container images have been updated and provide a new PyTorch version.

2023-2 Maintenance

The following list of user-facing changes was introduced during the maintenance procedure between June 5th and 7th, 2023:

The primary address of the SSH login node has been changed. Use login.ai.lrz.de for all SSH connections to the LRZ AI Systems. The previous address is not functional anymore (see deprecation notice below).
The NVIDIA drivers have been updated to version R525 for full compatibility with the recently released CUDA 12
The software component providing the web-based frontend, Open OnDemand, has been updated to release 3.0.1

2023-1 Maintenance

Please note the following list of user-facing changes introduced during the maintenance procedure between March 13th and 15th, 2023:

The primary address of the SSH login node has been changed. Going forward, please use login.ai.lrz.de for all SSH connections to the LRZ AI Systems. The previous address is still functional, but will be removed in the future (deprecation notice).
TensorBoard has been added as new application to the available web servers of https://datalab3.srv.lrz.de

2022 Reminder

The previous LRZ AI Systems home directories, accessible from the LRZ AI Systems login nodes under /home/<lrz-account> (read-only since the latest maintenance), have been decommissioned by 2022-10-31.

2022-4 Maintenance

The resource selection for the OnDemand-based interactive apps (Jupyter Notebook, JupyterLab, RStudio Server) has been updated and unified. It does now allow for the allocation of single GPUs (in addition to combinations of CPU cores and RAM size) with all these front ends.

2022-3 Maintenance

Please note the following user-facing changes to the LRZ AI Systems, which took effect during the latest maintenance:

Most importantly, availability of storage options on the LRZ AI (and MCML) Systems changed. The previous home directories have been superseded by the default Linux Cluster home directories. The very same files and data can now be accessed in the default home directories directly after login, irrespective of using a Linux Cluster or AI Systems login node, i.e. the LRZ AI Systems and LRZ Linux Cluster now provide unified home directories.

- The previous LRZ AI Systems home directories are still accessible from the LRZ AI Systems login nodes under /home/<lrz-account> (read-only). These directories will be decommissioned by 2022-10-31, so make sure to copy your files into the new unified home directories as soon as possible!
- In addition, the full offer of Data Science Storage (DSS) systems and containers can now directly be accessed from all LRZ AI Systems login and compute nodes.
- Use the command dssusrinfo all on the login nodes to get an overview of all individually accessible DSS containers and their utilization.

For further details see Storage on the LRZ AI Systems