Configuration
The HQS Tasks execution client is configurable via code that is in the user script, i.e., the script you use to schedule tasks. On this page, we first describe in which ways you can apply configurations to your script and then which parameters are available to be configured.
Global vs. Scoped Configuration
All configuration parameters can be changed on a global level at any point in the script, also multiple times in the same script. The configuration is then effective for every subsequent call to any task client function or until the configuration is adapted again.
The syntax for this is as follows:
from hqs_task_execution.config import global_config
global_config.provisioning.hardware.memory = 4096
This sets the parameter provisioning.hardware.memory to 4096 for the remainder of the script.
Alternatively, most of the configuration parameters (see parameter table below) can also be applied to a part of a script inside a Python with-block. The effect of this is that after this with-block, the global parameter values are restored. This is in particular useful if you want to apply a specific parameter value only for a part of the script, maybe even for a single task, and don't want to affect subsequent task executions.
The above example translates to:
from hqs_task_execution.config import scoped_config
# For any task executed here, the "global config" is used
with scoped_config(memory_mb=8192):
# For every task executed inside this block, the parameter "provisioning.hardware.memory"
# is set to 8192. All other configuration options from the "global config" apply.
...
# For any task executed here, the "global config" is used
Note that due to the fact that we use a function call with keyword arguments,
sub-parameters like the provisioning.hardware.memory in the above example are spelled differently here,
with a so-called "shortcut" (e.g., without the provisioning.hardware prefix).
Also, for some parameters we introduce variants to be able to specify values in specific units.
In the example above this is memory_mb for specifying the memory in MiB or memory_gb for GiB.
The following rules apply:
- Parameters changed globally (or kept at the default values) are unchanged by any scoped config block. Changing global configuration from within a scoped block is technically possible, but only applied after the last scoped config closed, and therefore not recommended.
- Scoped configurations can be nested. If a parameter is changed on different levels, the most nested one is used. Of course, if that block is finished, the previous one is restored.
In other words, a scoped config block copies the current configuration and applies the changes mentioned in the function call, which then becomes the "new current" configuration, starting with the current configuration being the global one when entering a top-level scoped config block.
Configuration Parameters
In the following, we list the most commonly used configuration parameters. Parameters needed for highly specific use cases are currently not documented here. Also, some parameters are specific to the type of backend you configure; these sub-parameters are documented on the page of the corresponding backend.
Parameter Key in global_config | Shortcut for scoped_config(and variants) | Type (and variants) | Description |
|---|---|---|---|
backend | - | BackendConfiguration... | Configuration of the task execution backend (see sub-pages in the Backends section) |
cache | - | CacheConfiguration... | The type of storage used to store cache data (see Caching) |
cache.mode | cache_mode | hqs_tasks_execution.config.CacheMode (enum) | The mode in which the cache shall operate (see Caching) |
throttling.disable_submissions | disable_submissions | bool | When set to True, no execution is actually submitted; script can only succeed when everything is in cache |
provisioning.hardware.vcpu | vcpu | float | The number of virtual CPU shares allocated for executing the task |
provisioning.hardware.memory | memory_mb( memory_gb) | int( int) | The amount of memory in MiB (or GiB) allocated for executing the task |
provisioning.hardware.gpu | gpu | int | The number of GPU cards allocated for executing the task (experimental) |
provisioning.timeout | timeout( timeout_minutes / timeout_hours) | datetime.timedelta( float / float) | The amount of time the task is allowed to run for until it times out |