Running Tasks

On this page, you learn how to run your first task. We assume you have read the previous pages of this documentation, where we briefly explained how HQS Tasks is supposed to work and what the role of the client and backend are, respectively.

Simple Example

For the purpose of testing the task system in general and for learning the basic concepts, example tasks have been developed, which we are going to execute here.

Furthermore, the code examples are using the REST backend (but can of course be adapted to use a different backend by simply changing the corresponding configuration section). We therefore assume that its prerequisites are met.

Last but not least, we assume you have set up a corresponding Python environment in which we can install the dependencies.

Installing the client package

To get started, we need to install the Python package hqs_tasks_execution, which is responsible for everything regarding the execution of tasks that is indepedent of a specific task or application.

Then, depending on the task we want to execute, we need the corresponding task client package. The client package for the example tasks used in this tutorial is hqs_task_example_client.

Install the packages using hqstage:

hqstage install hqs_tasks_execution hqs_task_example_client

In the next steps we will write a script, which we usually call the user script, that uses these packages to execute tasks. You can use any tooling to write your Python script, for example Jupyter.

Configuring the backend

We recommend that the code in which the HQS Tasks backend is configured is always placed at the very top of a task client script. This is because ideally, these lines can be changed in order to "move" from one backend to another, and are as such not defining what your script is doing, but where it is doing it.

Since we are using the REST backend, we just configure it with the defaults, according to its documentation:

from hqs_tasks_execution.config import global_config, BackendConfigurationREST

global_config.backend = BackendConfigurationREST()

Running a task

Most programming tutorials start with a "hello world" as a minimal example. The purpose is to learn the very basic steps to get first things done, without much "meaning".

So the first task which we will execute is hello, which takes a single string argument as the input and will return (as the output) a greeting message.

The function hello in the following code is the so-called task client function which merely serves as an interface for running this task using the HQS Tasks system.

Note that the task client functions are asynchronous (also called coroutines), i.e., they have to be called using the async/await syntax (see the asyncio package).

import asyncio
from hqs_task_example_client import hello

async def main():
    message = await hello("world")
    print(message)

asyncio.run(main())

If you are running inside a Jupyter Notebook, Jupyter will automatically provide an event loop for you. Hence, in a notebook, you can simply use the following code:

from hqs_task_example_client import hello

message = await hello("world")
print(message)

The above code will execute the task and, once it finished successfully, the output value is printed.

Concurrent Execution

Task client functions can be executed concurrently using asyncio.gather (see asyncio documentation) . This is particularly useful in combination with the REST or Slurm backend, where individual tasks may run on separate machines.

import asyncio
from hqs_task_example_client import hello



async def main():
    tasks = [hello("world"), hello("again"), hello("there")]
    messages = await asyncio.gather(*tasks)
    print(messages)

asyncio.run(main())