Deployment

The deployment of a task inherently depends on the backend on which it is supposed to be executed.

Note the additional general notes below, independent of the backend.

Backend: Local

For a task to be executable locally, nothing special needs to be done other than installing itself in the current Python environment. This is recommended when actively developing a task, and usually you want to install the package in editable mode.

For example, suppose you develop your task in a project folder my_task, run the following command:

pip install -e ./my_task

Then, generate the client package and install it too - here we again recommend the editable mode:

pip install -e ./generated/python

Backend: REST

Deploying tasks to the REST backend (the HQS cloud) is currently only possible by HQS developers.

If you are an HQS developer, deployment to REST is automatically done when using the CI/CD template.

Backend: Slurm (direct)

The deployment process for this backend furthermore depends on how you're going to setup and use Python environments.

For example, if you are using conda / micromamba (like shown in the example configuration in the user documentation), install the corresponding task package there.

General Notes

Independent of the backend, please note the following.

Version Matching

The client and the task itself need to be installed in the same version.

General Requirements in Client Environment (for First Deployment)

Additionally (and maybe prior) to the above, at least once you need to install hqs-tasks-execution in the client environment using hqstage:

hqstage install hqs-tasks-execution

Note that this needs to be done in the same environment your task is going to be submitted (client script).

General Requirements in Target Environment (for First Deployment)

Additionally (and maybe prior) to the above, at least once you need to install hqs-task-execute in the target environment using hqstage there:

hqstage install hqs-task-execute

Note that this needs to be done in the same environment your task is going to be executed (i.e, for Slurm: the target on the Slurm nodes and not in the environment you use locally to submit tasks). Maybe you first need to install hqstage there, too.

Then, also at least once, install the additional requirements for the above which can be done using the provided helper program:

hqs-task-execute-install-requirements