Backends for running HQS Tasks

As described in Architecture, HQS Tasks is a system composed of a client part and a backend part.

The backend is the component which a client communicates with in order to submit new task executions, observe their state and finally fetch the result. Depending on the type of backend, the actual backend's logic is on a remote service and decoupled from the client. The caching mechanism allows the client – after tasks have been submitted – to detach by simply cancelling the local script; the executions still continue running in the backend, and later the client can re-attach to finally fetch the results.

Here, we briefly describe what types of backends can be used to execute tasks. Details are then found on the corresponding backend details page.

Local

The "local" backend is intended to be used when you want to run a task directly on the same machine on which the client script is running on.

This is the default backend (but in the future we might switch the default to REST, see below).

REST (Cloud)

The REST backend will execute tasks in the cloud. The client will communicate with HQS Core hosted in the HQS Cloud, and tasks are running on machines provisioned on demand.

This backend is somewhat unique in the sense that it also serves its own database in which all task executions are being collected.

Currently, this is the primary backend to be used for most users. But you still need to configure the client to use it, since the default is the local backend (we might switch this to the REST backend in the future, though).

Slurm

The Slurm backend will execute tasks on a compute cluster powered by the Slurm workload manager. The client will communicate via SSH to any "login node" that is connected to any Slurm setup.