FAQ: Using Dask and ipcluster on Linux Cluster

In order to use the load balancing features of dask (https://dask.org/) on the Linux Cluster, here is a recipe how to set it up:

Using the conda command install:
$ module load anaconda3
$ conda create -n py38 python=3.8
$ conda install ipython
$ conda install ipcluster
now you are ready to run a ipcluster program on a single node on the Linux Cluster

Use this slurm batch file:

#!/bin/bash
#SBATCH -J ipcluster
#SBATCH --get-user-env
#SBATCH --clusters=cm2_tiny
#SBATCH --partition=cm2_tiny
#SBATCH --nodes=1-1
#SBATCH --cpus-per-task=28
#SBATCH --time=08:00:00
module load slurm_setup

source ~/miniconda39/bin/activate

ipcluster start --daemonize
echo "ipcluster started"
sleep 10
echo "starting calculation"
python -c "from ipyparallel import Client; c=Client(); print(c.ids)"
ipcluster stop