2024-10-23 AI Training Series - Introduction to Container Technology & Application to AI at LRZ (hdta2w24)

Contents

Since the introduction of docker back in 2013, container technology has become the industry standard for software packaging, distribution, and deployment.
Creating a container consists in bundling an application, its dependencies and runtime in a single unit that can later run independently of the underlying infrastructure. Unlike virtual machines, containers are lightweight and yield higher performances while providing greater versatility and interoperability. As containers accommodate an easy, safe, reliable, and scalable way to run applications and pipelines, they are an attractive candidate for high performance computing and artificial intelligence workloads.
With this module, we will showcase the most enticing features and niceties offered by containers. Not only will we explore their history and implementations, but we will also dive into actual and cutting edge uses with a particular emphasis on artificial intelligence tasks, reproducible biomedical pipelines, and automated workflows.
Participants will roll up their sleeves and get their hands on the compute cloud of LRZ to set containers in action. By the end of the course, participants will be able to transfer their experience and knowledge to their specific use-cases and requirements.

Three content blocks of roughly one hour each:

1 A Tour of Containers and Kickstart
    1.1 You Are Here
    1.2 Basic Concepts for Containers
    1.3 Containers vs. the World
    1.4 Kickstarter: Your First Steps With Containers

2 Under the Hood of Containers: Warm-up Lap to Flat-out
    2.1 The Bolts and Nuts of Containers
        2.1.1 namespaces, cgroups, and copy-on-write Storage
        2.1.2 Volumes, Networking, and More
        2.1.3 Configuration and Image Creation
    2.2 Warm-up lap: Make an ANN  Dream in a Container
    2.3 Containers and Security
        2.3.1 Security and User Namespace
        2.3.2 Security and Docker Socket Exploit
    2.4 Containers and HPC
    2.5 Flat-out: Containers and HPC With Enroot and Charliecloud

3 Containers on Nitromethane
    3.1 Abstract Your Containers
        3.1.1 Continuous Integration / Continuous Deployment
        3.1.2 Reproducible Scientific Pipelines
    3.2 Hardware Accelerated Containers
    3.3 Orchestration and Scaling Across a Compute Cluster
    3.4 Last Lap: Reproducible Transcriptomic Workflow With Containers and Nextflow

End: Wrap-up, take home message, last questions and book / website / podcast / events recommendations

Prerequisites

•  AI Training Series: Orientation Session (or comparable previous knowledge): access the remote work environment, navigate the work environment, edit files, pull images, spin containers...

Hands-On

• Hands-on #1: Your first steps with containers
• Hands-on #2: Make an artificial neural network dream in a container
• Hands-on #3: Container and HPC with Charliecloud 
• Hands-on #4: Enable transcriptomic reproducible workflow with containers and Nextflow

Content Level

The content level of the course is broken down as:

Language

English

Lecturers

Florent Dufour, Darshan Thummar (LRZ)

Prices and Eligibility

The course is open and free of charge for academic participants from Germany.

Registration

Please register with your official e-mail address to prove your affiliation.

Withdrawal Policy

See Withdrawal

Legal Notices

For registration for LRZ courses and workshops we use the service edoobox from Etzensperger Informatik AG (www.edoobox.com). Etzensperger Informatik AG acts as processor and we have concluded a Data Processing Agreement with them.

See Legal Notices