Running Renku on HPC

The Renku CLI supports various backends for executing workflows. Currently, two different providers are implemented, namely cwltool and toil. If you have a specific provider you need for your infrastructure, have a look at Implementing a workflow provider for a detailed description of how to implement your own workflow provider. Alternatively, please make a feature request.

By default all workflows are executed by the cwltool provider, that exports the workflow to CWL and then uses cwltool to execute the given CWL.

The workflow backend can be changed by using the -p/--provider <PROVIDER> command line option. A backend’s default configuration can be overridden by providing the -c/--config <config.yaml> command line parameter. The following renku commands support the above mentioned workflow provider related command line options:

For example, to execute a previously created my_plan workflow with toil, one would simply run the following command:

$ renku workflow execute -p toil my_plan

Using toil as a workflow provider has the advantage that it supports running the workflows on various high-performance computing and cloud platforms.

In order to use any other provider with renku you must first install the required extras. In the case of toil this means running the install command like

$ pip install renku[toil]

Renku on Slurm

Slurm is a highly configurable open-source workload manager, which is in widespread use at government laboratories, universities and companies world wide. It is used in over half of the top 10 systems in the TOP500 listing.

As toil supports Slurm, one can easily execute the previously created renku workflows on a Slurm-managed HPC resource. One just needs to provide a simple configuration file to the provider (--config/-c):

batchSystem: slurm
disableCaching: true

The disableCaching option is necessary for Slurm; for more details see the related toil issue TOIL-1006.

Additional Slurm-specific parameters can be provided with the TOIL_SLURM_ARGS environment variable.

Taking the example above, the following command line will execute my_plan on Slurm:

$ TOIL_SLURM_ARGS="-A my_account --export=ALL" renku workflow execute -p toil -c provider.yaml my_plan

where

  • provider.yaml file contains the above mentioned two parameters for toil,

  • -A my_account specifies which account should be charged for the used resources,

  • --export=ALL specifies that all environment variables are propagated to the Slurm workers. It is often required in academic Slurm installations.

Run with Docker Containers

You can execute workflows using Docker with toil as it supports Docker containers. To do that, you need to add a docker section in the provider’s config file and define a Docker image to be used for execution:

docker:
  image: python:3.9-slim

Renku toil plugin then uses the specified image to run your workflow. For a list of available configuration options see Docker API documentation. Refer to Toil’s Docker documentation for more information on how to set up Docker for use in toil.

Note that due to the limitations in the underlying libraries, workflows that use redirection from/to stderr/stdin cannot be executed with Docker. Redirection to stdout is supported.

By default, if the config file doesn’t define a working_dir, Renku sets a default working directory and creates a volume in the container and mounts project’s directory to it to make sure that the container can access local files. However, if you define a working_dir in the config file, Renku doesn’t create this volume automatically and you must make sure that local files required to execute a workflow are accessible in the container.