Containerization of Tasks

In order to let the HQS Tasks backend execute a task, the task implementation needs to be deployed on some target environment.

When to Containerize and When To Not

Depending on which backend, i.e. environment type, is chosen for this, the task first needs to be containerized. Simply speaking, containerization is a way of packaging some software with the dependencies so it can be executed (almost) anywhere; most notably in the case of a Python software the Python stack is included in that container.

Not all backends supported by HQS Tasks need containers to run the software. However, the others require other means to ensure that the requirements, i.e. the Python stack and the dependant packages, are installed on the execution target environment.

At the moment of writing this, the only backend which uses containers is the REST backend, which utilizes AWS Batch for executing tasks. This AWS service is based on docker containers, hence we need to build one.

Prerequisites

You can ship several tasks, accessible by one or several task CLIs, in the same container. For the following, we assume that you already have a single task registry file which describes all tasks to be containerized, and you know what dependencies these tasks require.

In order to build a container, you need to have docker installed. Alternatives exist, but are not mentioned in this documentation.

To include your task implementation in the Docker container, you have essentially two options:

Make it available as an installable Python package via some Python package index.
Make the source code available to the build context.

We'll describe both methods below.

Choice of Base Contaienr

When building a docker container, we start with a so-called base container and add our software and its dependencies to it. To keep things simple, we start off with an official Python base container, such as python:3.13. Again, alternatives exist, but are not documented here.

Writing the Dockerfile

The so-called "Dockerfile" is the receipt for building a container; it lists several steps which are to be performed by the docker build system in order to create the docker image.

Without much detailed explanation, the following Dockerfile shall serve as an example, which can be easily adapted to other task projects.

# We build on top of the following base container.
FROM python:3.13

# Option 1: Install everything via pip.
RUN pip install hqs_task_example

# Option 2: Install from source code.
COPY ./ ./
RUN pip install ./

For option 1, we assume the Dockerfile is stored in the Python package root folder (from which it is installable). Adapt the paths if this assumption does not match your situation.

Building the Container

To now build the container, simply run the following command within the folder where you have put the Dockerfile. The name after -t is going to be the name of your docker container image, the so-called "tag".

docker build . -t hqs_task_example

Testing the Container

After building the container, you can run it with

docker run hqs_task_example