Docker on Login Nodes
Setup of QEMU and the VM
QEMU - Overview
QEMU is an emulator of hardware, and can thus be used for real virtualization (https://www.qemu.org/). Together with KVM, QEMU can benefit from kernel modules and VT (Virtualization Technology) hardware, which for setup but requires administrative rights. However, QEMU also possess sort of a "user mode", meaning that a normal user can build QEMU and run VMs therein without any need for administrative support. The price to pay is performance, as software-sided emulation of hardware is slower.
As a fully-sledged virtualization system, any kind of operating system could be used inside the VM, as long as that OS supports this hardware. Even other hardware could be emulated.
QEMU - Installation
The easiest way to install QEMU is user_spack.
login> module rm intel-mpi intel-mkl intel login> module load user_spack login> spack install qemu@3.1.1.1 [...] login> spack module tcl refresh qemu # add ~/spack/modules/... to MODULEPATH via "module use ~/spack/modules/..."
QEMU - Usage and VM Installation
In order to setup a VM harddisk,
login> qemu-img create -f qcow2 virtualdebian.img 40G
This creates a 40 GB (maximum) VM harddisk file.
To install e.g. Debian inside of the VM, download installation sources e.g. as ISO image and execute
login> qemu-system-x86_64 -hda virtualdebian.img -m 4G -net nic -net user -smp 4 -vnc :4,password=on -monitor stdio QEMU 3.1.1.1 monitor - type 'help' for more information (qemu) change vnc password # <- in order to set a >>NEW<< VNC password (for security) Password: *********** (qemu)
Afterwards, you can connect via a VNC viewer through a SSH forward tunnel to login:5904. Should be the VNC display :4 already occupied, please use a free one. (The VNC port is 5900 + <display number>)
If you use qemu-kvm (installed in the system, also SSH X forwarding can be used).
Now, you can install the VM OS as you like. On LXC, with free internet access, you can also install Debian via NetInstall. On SNG, we recommend to install from a install DVD.
Although you could install a full desktop system, for the purpose of Docker images deployment, a basic installation with an SSH server suffices. Docker and Charliecloud can be installed afterwards.
A ready image is available under /lrz/sys/tools/qemu/virtualdebian.img.tgz, with root and user (debian) password password. Please change it at first use!
QEMU - Access to the VM via SSH
Access to the VM via VNC is always possible. But more convenient also for exchanging data from and to the VM is SSH.
login> qemu-system-x86_64 -hda virtualdebian.img -m 4G -device virtio-net,netdev=vmnic -netdev user,id=vmnic,hostfwd=tcp:127.0.0.1:12345-:22 -smp 4 -vnc :4,password=on -monitor stdio QEMU 3.1.1.1 monitor - type 'help' for more information (qemu)
Again, is the bind port (here 12345) already occupied, you will see an error like "qemu-system-x86_64: Could not set up host forwarding rule 'tcp:127.0.0.1:12345-:22'". In that case, use another port!
Now (after some grace period for letting the system boot up), you can now login from the same login node, where the VM is running, to that VM via
login> ssh -p 12345 debian@localhost Warning: Permanently added '[localhost]:12345' (ECDSA) to the list of known hosts. # is you change the VM, the host keys change, too. Clear the ~/.ssh/known_hosts debian@localhost's password: # see password above Linux qemu-debian 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 [...] debian@qemu-debian:~$
On SuperMUC-NG, a proxy-jump connection to such a VM with reverse SSH tunnel and a HTTP proxy (local on your desktop) is also possible, such that even internet access for installation and docker image download is possible.
For switching off the VM, we propose to use the OS means. On Debian, it is switching to root, and executing
debian@qemu-debian:~$ su root@qemu-debian:~# systemctl poweroff
Docker/Charliecloud Deployment - Normal Workflow/Usage
As preparation, do the preparation steps as described above. Just start you VM in QEMU. Setup the necessary SSH connections and tunnels, etc. With Docker and Charliecloud already being installed, you can follow the steps as described under https://hpc.github.io/charliecloud/tutorial.html#seconds-to-charliecloud
debian@qemu-debian:~$ su # docker requires root rights root@qemu-debian:# ch-build -t hello . # go sure to have a valid Dockerfile in the current location root@qemu-debian:# ch-builder2tar hello . # creates a hello.tgz in current folder root@qemu-debian:# chown debian hello.tgz # maybe not necessary
You can now copy this file from outside, but still on that login node, from this VM via scp (remember to use the right port on localhost!).
Take case not to overload the VM harddisk (40 GB are not much)!
root@qemu-debian:# docker image ls root@qemu-debian:# docker image rm -f <image hashes>
Shared Folder between QEMU VM and Host
Also this is possible. Add
variant('virtfs', default=False)
depends_on('glib@2.40:')
depends_on('pixman@0.21.8:')
depends_on('libcap',when='+virtfs')
depends_on('attr',when='+virtfs')
def configure_args(self):
args = []
if '+virtfs' in self.spec:
args.append('--enable-virtfs')
return args
to the QEMU Spack package.py file, and install QEMU via
login> spack install qemu@3.1.1.1+virtfs
The VM can then be started via
login> qemu-system-x86_64 -hda virtualdebian.img -m 4G [...] -smp 4 -virtfs local,path=/gpfs/scratch/.../shared_folder,mount_tag=host0,security_model=passthrough,id=host0 -vnc :4,password=on -monitor stdio
(SSH port binding is then maybe not necessary.) This must then be mounted inside of the VM. The /etc/fstab can be extended with (see coincident names in QEMU CMD parameters with mount name!)
host0 /mnt 9p trans=virtio,version=9p2000.L 0 0
Caution: This method is however somewhat fragile. (1) if the mount point is not there, you may make your VM hanging. (2) It seems not possible to create files and folders in the host folder, which is shared. You can only overwrite existing folders and files.
To overcome this inconvenience, the SSH login appears in many respect more comfortable.