Job Farming with redis

Redis (https://redis.io/) is a key value store database which runs in memory but also has disk persistency. It can reconstruct the data back in memory once the system restarts. It has bindings for many clients (https://redis.io/clients) where we focus on R, bash and python. For R there exists a library (https://github.com/bwlewis/doRedis) which allows to use foreach loops in R which get translated into a job queue in redis which can then be executed on a set of nodes (clients).

A simple example would be the following:

Startup the redis database on a login node:

$ module load redis

$ redis-server --port 8123 --protected-mode no

protected mode no means that the server listens to port 8123 and does not need authentication of the client. This is acceptable inside the LRZ infrastructure, but if you want outside connections you have to provide some type of authentication or tunneling.

Startup a set of clients via slurm which connect to the redis server