Docker on Login Nodes

Setup of QEMU and the VM

QEMU - Overview

QEMU is an emulator of hardware, and can thus be used for real virtualization (https://www.qemu.org/). Together with KVM, QEMU can benefit from kernel modules and VT (Virtualization Technology) hardware, which for setup but requires administrative rights. However, QEMU also possess sort of a "user mode", meaning that a normal user can build QEMU and run VMs therein without any need for administrative support. The price to pay is performance, as software-sided emulation of hardware is slower.

As a fully-sledged virtualization system, any kind of operating system could be used inside the VM, as long as that OS supports this hardware. Even other hardware could be emulated.

QEMU - Installation

The easiest way to install QEMU is user_spack.

login> module rm intel-mpi intel-mkl intel
login> module load user_spack
login> spack install qemu@3.1.1.1
 [...]
login> spack module tcl refresh qemu                     # add ~/spack/modules/... to MODULEPATH via "module use ~/spack/modules/..."

QEMU - Usage and VM Installation

In order to setup a VM harddisk,

login> qemu-img create -f qcow2 virtualdebian.img 40G

This creates a 40 GB (maximum) VM harddisk file.

To install e.g. Debian inside of the VM, download installation sources e.g. as ISO image and execute

login> qemu-system-x86_64 -hda virtualdebian.img -m 4G -net nic -net user -smp 4 -vnc :4,password=on -monitor stdio
QEMU 3.1.1.1 monitor - type 'help' for more information
(qemu) change vnc password                                                                                            # <- in order to set a >>NEW<< VNC password (for security)
Password: ***********
(qemu)

Afterwards, you can connect via a VNC viewer through a SSH forward tunnel to login:5904. Should be the VNC display :4 already occupied, please use a free one. (The VNC port is 5900 + <display number>)

If you use qemu-kvm (installed in the system, also SSH X forwarding can be used).

Now, you can install the VM OS as you like. On LXC, with free internet access, you can also install Debian via NetInstall. On SNG, we recommend to install from a install DVD.
Although you could install a full desktop system, for the purpose of Docker images deployment, a basic installation with an SSH server suffices. Docker and Charliecloud can be installed afterwards.

A ready image is available under /lrz/sys/tools/qemu/virtualdebian.img.tgz, with root and user (debian) password password. Please change it at first use!

QEMU - Access to the VM via SSH

Access to the VM via VNC is always possible. But more convenient also for exchanging data from and to the VM is SSH.

login> qemu-system-x86_64 -hda virtualdebian.img -m 4G -device virtio-net,netdev=vmnic -netdev user,id=vmnic,hostfwd=tcp:127.0.0.1:12345-:22 -smp 4 -vnc :4,password=on -monitor stdio
QEMU 3.1.1.1 monitor - type 'help' for more information
(qemu) 

Again, is the bind port (here 12345) already occupied, you will see an error like "qemu-system-x86_64: Could not set up host forwarding rule 'tcp:127.0.0.1:12345-:22'". In that case, use another port!

Now (after some grace period for letting the system boot up), you can now login from the same login node, where the VM is running, to that VM via

login> ssh -p 12345 debian@localhost
Warning: Permanently added '[localhost]:12345' (ECDSA) to the list of known hosts.               # is you change the VM, the host keys change, too. Clear the ~/.ssh/known_hosts
debian@localhost's password:                                                                     # see password above
Linux qemu-debian 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64
 [...]
debian@qemu-debian:~$

On SuperMUC-NG, a proxy-jump connection to such a VM with reverse SSH tunnel and a HTTP proxy (local on your desktop) is also possible, such that even internet access for installation and docker image download is possible.

For switching off the VM, we propose to use the OS means. On Debian, it is switching to root, and executing

debian@qemu-debian:~$ su
root@qemu-debian:~# systemctl poweroff

Docker/Charliecloud Deployment - Normal Workflow/Usage

As preparation, do the preparation steps as described above. Just start you VM in QEMU. Setup the necessary SSH connections and tunnels, etc. With Docker and Charliecloud already being installed, you can follow the steps as described under https://hpc.github.io/charliecloud/tutorial.html#seconds-to-charliecloud

debian@qemu-debian:~$ su                                                       # docker requires root rights
root@qemu-debian:# ch-build -t hello .                                         # go sure to have a valid Dockerfile in the current location
root@qemu-debian:# ch-builder2tar hello .                                      # creates a hello.tgz in current folder
root@qemu-debian:# chown debian hello.tgz                                      # maybe not necessary

You can now copy this file from outside, but still on that login node, from this VM via scp (remember to use the right port on localhost!).

Take case not to overload the VM harddisk (40 GB are not much)!

 root@qemu-debian:# docker image ls
 root@qemu-debian:# docker image rm -f <image hashes> 

Shared Folder between QEMU VM and Host

Also this is possible. Add

  variant('virtfs', default=False)

  depends_on('glib@2.40:')
  depends_on('pixman@0.21.8:')
  depends_on('libcap',when='+virtfs')
  depends_on('attr',when='+virtfs')

  def configure_args(self):
    args = []
    if '+virtfs' in self.spec:
      args.append('--enable-virtfs')
    return args

to the QEMU Spack package.py file, and install QEMU via

login> spack install qemu@3.1.1.1+virtfs

The VM can then be started via

login> qemu-system-x86_64 -hda virtualdebian.img -m 4G [...] -smp 4 -virtfs local,path=/gpfs/scratch/.../shared_folder,mount_tag=host0,security_model=passthrough,id=host0 -vnc :4,password=on -monitor stdio

(SSH port binding is then maybe not necessary.) This must then be mounted inside of the VM. The /etc/fstab can be extended with (see coincident names in QEMU CMD parameters with mount name!)

 host0 /mnt 9p trans=virtio,version=9p2000.L 0 0

Caution: This method is however somewhat fragile. (1) if the mount point is not there, you may make your VM hanging. (2) It seems not possible to create files and folders in the host folder, which is shared. You can only overwrite existing folders and files.

To overcome this inconvenience, the SSH login appears in many respect more comfortable.