Orchestration on Docker¶

APIs¶

dagster_docker.DockerRunLauncher RunLauncher[source]¶

Config Schema:

image (dagster.StringSource, optional)

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional)

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional)

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource)

username (dagster.StringSource)

password (dagster.StringSource)

env_vars (List[String], optional)

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional)

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional)

Names of the networks to which to connect the launched container at creation time

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]¶

Config Schema:

image (dagster.StringSource, optional)

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional)

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional)

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource)

username (dagster.StringSource)

password (dagster.StringSource)

env_vars (List[String], optional)

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional)

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional)

Names of the networks to which to connect the launched container at creation time

retries (selector, optional)

Default Value:

{
    "enabled": {}
}

Config Schema:

enabled (strict dict, optional)

Default Value:

{}

disabled (strict dict, optional)

Default Value:

{}

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster import job
from dagster_docker import docker_executor

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.