PluginsAzure (EE)Task RunnersRunnerBatch

Batch​Batch

Batch Batch

yaml
type: "io.kestra.plugin.ee.azure.runner.Batch"

Task runner that executes a task inside a job in Azure Batch.

This task runner is container-based so the containerImage property must be set.

To access the task's working directory, use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable. Input files and namespace files will be available in this directory.

To generate output files you can either use the outputFiles task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the {{ outputDir }} Pebble expression or the OUTPUT_DIR environment variables.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the blobStorage property. The blob storage serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this blob storage during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured blobStorage for each task run. You can access that folder using the {{ bucketPath }} Pebble expression or the BUCKET_PATH environment variable. There is two supported way to provide authentication for the blob storage:

  • connectionString and containerName properties
  • containerName, endpoint, sharedKeyAccountName and sharedKeyAccountAccessKey properties

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

Examples

Execute a Shell command.

yaml
id: new-shell
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.Batch
      account: "{{secrets.account}}"
      accessKey: "{{secrets.accessKey}}"
      endpoint: "{{secrets.endpoint}}"
      poolId: "{{vars.poolId}}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new-shell-with-file
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{inputs.file}}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.azure.ee.runner.Batch
      account: "{{secrets.account}}"
      accessKey: "{{secrets.accessKey}}"
      endpoint: "{{secrets.endpoint}}"
      poolId: "{{vars.poolId}}"
      blobStorage:
        connectionString: "{{secrets.connectionString}}"
        containerName: "{{vars.containerName}}"
    commands:
      - cp {{workingDir}}/data.txt {{workingDir}}/out.txt

Properties

accessKey

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The Batch access key.

account

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The Batch account name.

delete

  • Type: boolean
  • Dynamic: ❌
  • Required: ✔️
  • Default: true

Whether the job should be deleted upon completion.

endpoint

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The blob service endpoint.

poolId

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

Id of the pool on which to run the job.

resume

  • Type: boolean
  • Dynamic: ❌
  • Required: ✔️
  • Default: true

Whether to reconnect to the current job if it already exists.

blobStorage

completionCheckInterval

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Default: 5.000000000
  • Format: duration

Determines how often Kestra should poll the container for completion. By default, the task runner checks every 5 seconds whether the job is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick jobs and to a lower threshold (e.g. PT1M = every minute) for long-running jobs. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors.

registry

The private registry which contains the container image.

waitUntilCompletion

  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Default: 3600.000000000
  • Format: duration

The maximum duration to wait for the job completion unless the task timeout property is set which will take precedence over this property.

Azure Batch will automatically timeout the job upon reaching such duration and the task will be failed.

Outputs

Definitions

io.kestra.plugin.ee.azure.storage.blob.models.BlobStorageForBatch

Properties

containerName
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The URL of the blob container the compute node should use.

Mandatory if you want to use namespaceFiles, inputFiles or outputFiles properties.

connectionString
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Connection string of the Storage Account.

endpoint
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The blob service endpoint.

sharedKeyAccountAccessKey
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Shared Key access key for authenticating requests.

sharedKeyAccountName
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Shared Key account name for authenticating requests.

io.kestra.plugin.ee.azure.batch.models.ContainerRegistry

Properties

identityReference

The reference to the user assigned identity to use to access the Azure Container Registry instead of username and password.

password
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The password to log into the registry server.

registryServer
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The registry server URL.

If omitted, the default is "docker.io".

userName
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The user name to log into the registry server.

io.kestra.plugin.ee.azure.batch.models.ComputeNodeIdentityReference

Properties

resourceId
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The ARM resource ID of the user assigned identity.

Was this page helpful?