First Steps

First of all, in this documentation we assume you are developing (one or more) tasks in a Python package which is then deployed in some CI/CD pipeline to a target environment. This pipeline also generates a client library.

The task can then be invoked by installing the client library and invoking the according client functions. This is also explained in the HQS Tasks user documentation.

Example Project

An example project setup is given in the hqs_task_example Python package. You can install that using hqstage:

hqstage install hqs-task-example

It can then be inspected by browsing the installed package folder inside your Python environment's site packages directory (for example <ENVIRONMENT_FOLDER>/lib/python3.13/site-packages/hqs_task_example, but this varies).

Project Setup

Usually a "tasks project" is a simple Python project which makes use of the Python package hqs_task. If not yet done, install it using hqstage:

hqstage install hqs-task

Most essentially, in the current version of that package (Note: we might simplify that in the future), you need to ensure the following things:

  • Import cli from hqs_task in your package's root module's __init__.py file:

    from hqs_task import cli

    Furthermore, make sure that in the root module of your package all tasks or modules which define them are imported.

  • Register this cli function as a CLI script in your package's pyproject.toml file:

    [project.scripts]
hqs_task_example = "hqs_task_example:cli"

    The script name on the left hand side specifies the name of the CLI program which will be defined once that package is installed.

    The value on the right hand side is the module name (simply the package name if it's the package's root module) and the cli function name separated by a colon.

  • Configure your CI/CD process to build the client library and to deploy your tasks in the target environment. The steps for that are explained in the documentation. HQS developers may use the prepared CI/CD templates as explained below.

HQS-specific Setup

Warning

This section only applies to HQS-internal developers. For other developers, the steps need to be done manually or a CI/CD process needs to be setup separately. See the corresponding sections of this documentation.

For HQS Task repositories developed at HQS the task deployment and generation of the client package is made convenient by the provided CI/CD templates. A developer only has to specify a few CI/CD variables to enable automatic task deployment and client package generation.

The tasks/deploy_aws.yml template handles the deployment of tasks to AWS and will only be executed if a new version has been tagged. Note that for this to work, version tags must be protected (this can be set in the repository settings via Settings > Repository > Protected Tags).

Once the CLI script is registered as explained above, configure the CI/CD pipeline to pick this script name up as the task CLI program. In HQS' Gitlab setup with the CI/CD template, this is done by

  • setting the variable HQS_TASK_CLI to the task CLI program defined in the previous step

  • including the template files for deploying the task to a target environment; in this example tasks/deploy_aws.yml for deploying it to AWS:

    include:
  - project: templates/templates
    file:
      - tasks/deploy_aws.yml

Tip

It is recommended to verify the correctness of important CI/CD template variables before deploying tasks to AWS. When including files from the CI/CD template repository, the order of files can be important as some template variables may be overwritten by subsequent template files. For instance, with the custom container template it occurred that the CICD_CONTAINER was redefined by another step. Thus, the corresponding container/custom_cicd_container.yml template file is usually included last.

To let the pipeline handle building and deploying the client package to the HQS internal PyPi server, the HQS_CLIENT_PACKAGE variable needs to be set.

Writing a Task

A task is nothing more than a Python function which is decorated with the "task decorator" as you will see in the following example. The decorator registers the function as a task and enables it to be invoked as such from a task client script after deployment.

A simple "hello world" example may look like this:

from hqs_task import task

@task
def hello() -> None:
    print("Hello world!")

There are some restrictions on which functions can be declared as tasks:

  • The function must be defined on the root scope of a module. That means, it must not be defined inside a class or inside another function.

    A task is allowed to be defined inside a branch (conditional statement) on the root scope, which enables developers to only define tasks if some condition is met (for example optional dependencies being available).

  • The function is allowed to have none, one, or many explicitly named arguments, but no variable length arguments are supported (i.e., neither *args nor **kwargs can be specified in the signature).

  • For any arguments, default values may be supplied. As with usual Python rules, an argument with a default value may not be followed by an argument without a default value.

  • The function signature must include full type annotations: All arguments as well as the return type have to be strictly typed.

  • The types used for arguments as well as the return value are restricted to the ones listed under Types (Input / Output).

  • The function may throw exceptions (of any type), but it shall not terminate the program (intentionally) in any other abnormal way (i.e., exiting on failure).

Inputs and Outputs

Each task typically has an input and an output. Note that in the Python interface, multiple arguments are represented as a single input document (wrapped as a tuple). The return value maps to the output directly.

Technically, the input is optional, while the output is mandatory (but None is allowed to be used for an output).

For more details see Types (Input / Output).