run_starclass_mpi.py command line utility

Scheduler using MPI for running the TASOC classification pipeline on a large scale multi-core computer.

The setup uses the task-pull paradigm for high-throughput computing using mpi4py. Task pull is an efficient way to perform a large number of independent tasks when there are more tasks than processors, especially when the run times vary for each task.

The basic example was inspired by https://github.com/jbornschein/mpi4py-examples/blob/master/09-task-pull.py

Example

To run the program using four processes (one master and three workers) you can execute the following command:

>>> mpiexec -n 4 python run_starclass_mpi.py

Code author: Rasmus Handberg <rasmush@phys.au.dk>