Skip to content

Distributed execution

The platform can orchestrate applications installed on a single computer. Its real strength, however, is its distributed design: a workflow can combine models and data located at different institutions, on different servers and on HPC clusters. Each model stays where its licence, data or know-how is, and only data objects travel over the network.

How it works

  • Transparent remote objects. MuPIF is built on the Pyro5 distributed object system. A remote model is used exactly like a local one, because the platform handles all network communication between the objects.
  • Nameserver. Models, job managers and data are registered in a nameserver, so a workflow finds them by name, not by address.
  • Job managers and model servers. A job manager on each computational server starts model instances on demand, limits the number of concurrently running jobs and cleans up after them.
  • Secure communication. Communication runs inside a WireGuard virtual private network (VPN) or through SSH tunnels, so partners need not open their infrastructure to the internet.
  • Monitoring. A web dashboard shows the status of the nameserver, the VPN, the job managers and the scheduler.

HPC integration

Massively parallel simulations on HPC usually run in scheduled, batch mode, and HPC centres typically do not allow permanent services. MuPIF supports this mode. An HPC model server outside the cluster prepares the inputs, submits the job (for example to PBS) over SSH, waits for the result and retrieves the outputs. Only the actual model execution consumes HPC resources.

HPC integration in a MuPIF platform

A MuPIF platform deployed in the Musicode project, with an external HPC model server submitting jobs to an HPC resource.

Run it yourself

The demo infrastructure packages the complete platform in a single container: VPN, nameserver, MuPIF DB, scheduler, monitor and web interface. Use it for training, testing or as the starting point of your own deployment. See Get started.


(since February, 2023)