Modules (Reusable Sub-Dataflows)

Modules let you define reusable sub-graphs of nodes in separate YAML files and compose them into larger dataflows. Modules are expanded at compile time – the runtime never sees them.

快速开始

Module file (modules/transform_module.yml):

module:
  name: transform_pipeline
  inputs: [raw_data]
  outputs: [filtered]

nodes:
  - id: doubler
    path: doubler.py
    inputs:
      data: _mod/raw_data
    outputs:
      - doubled

  - id: filter
    path: filter_even.py
    inputs:
      data: doubler/doubled
    outputs:
      - filtered

Dataflow file (dataflow.yml):

nodes:
  - id: sender
    path: sender.py
    outputs:
      - value

  - id: pipeline
    module: modules/transform_module.yml
    inputs:
      raw_data: sender/value

  - id: receiver
    path: receiver.py
    inputs:
      filtered: pipeline/filtered

After expansion, pipeline becomes two nodes: pipeline.doubler and pipeline.filter, with all wiring resolved automatically.

Module Definition File

A module file has two sections:

module: header

Unknown fields in the module header are rejected whenever Dora loads the module, including dora expand, dora build, and dora run.

nodes: list

Standard node definitions, with one special syntax: _mod/port_name references a module input port. When expanded, _mod/port_name is replaced with whatever the parent wired to that port.

module:
  name: my_module
  inputs: [camera_feed]
  outputs: [detections]

nodes:
  - id: detector
    path: detect.py
    inputs:
      image: _mod/camera_feed    # resolved to parent's wiring
    outputs:
      - detections

Module-level build

Modules can have a top-level build: command that runs before any inner node builds:

module:
  name: ml_pipeline
  inputs: [image]
  outputs: [result]

build: pip install -r requirements.txt

nodes:
  - id: model
    path: model.py
    inputs:
      image: _mod/image
    outputs:
      - result

Using Modules

Reference a module in a dataflow node using the module: field instead of path::

- id: nav_stack
  module: modules/navigation.module.yml
  inputs:
    goal_pose: localization/goal

The module node’s inputs: map wires parent outputs to module input ports. External nodes reference module outputs as <module_id>/<output_name> (e.g., nav_stack/cmd_vel).

Parameters

Pass configuration values to modules via params::

- id: fast_pipeline
  module: modules/transform_module.yml
  inputs:
    raw_data: sender/value
  params:
    speed: "2.0"
    mode: turbo

Inside the module, reference params in args: using $PARAM_<UPPERCASE_KEY>:

nodes:
  - id: processor
    path: processor.py
    args: --speed $PARAM_SPEED --mode $PARAM_MODE
    inputs:
      data: _mod/raw_data
    outputs:
      - result

Parameters are also injected as environment variables (PARAM_SPEED, PARAM_MODE) into every node inside the module.

Expansion Rules

1. Load the module YAML file and validate its header
2. Prefix all internal node IDs with {module_id}. (e.g., nav_stack.planner)
3. Replace _mod/port_name references with the actual sources from the parent’s input map
4. Rewrite internal cross-references (e.g., planner/path becomes nav_stack.planner/path)
5. Map module-declared outputs to internal node outputs, so nav_stack/cmd_vel resolves to nav_stack.controller/cmd_vel. An inner node may produce a declared output from its node-level outputs:, from an operator:/operators: block, or from a legacy custom: block. For an output produced by one operator of a multi-operator operators: node, the resolved reference keeps the operator segment that runtime nodes require: nav_stack/cmd_vel resolves to nav_stack.runtime/controller/cmd_vel
6. Replace the module node with the expanded flat nodes
7. Substitute params: values in args: fields and inject as env vars

Use dora expand to see the result:

dora expand dataflow.yml

Nested Modules

Modules can reference other modules. The expansion is recursive with a depth limit of 8 levels:

# outer_module.yml
module:
  name: outer
  inputs: [data]
  outputs: [result]

nodes:
  - id: inner
    module: inner_module.yml
    inputs:
      raw: _mod/data

  - id: postprocess
    path: postprocess.py
    inputs:
      data: inner/processed
    outputs:
      - result

After expansion, node IDs are fully qualified: outer.inner.some_node.

Only outputs declared by a nested module are visible to its parent. A parent module can wire or re-export inner/processed from the example above because processed is listed in inner_module.yml’s module.outputs; it cannot reach private outputs produced by nodes inside inner_module.yml.

Optional Inputs

Declare inputs as optional when a module should work with or without certain connections:

module:
  name: flexible_processor
  inputs: [data]
  inputs_optional: [config]
  outputs: [result]

nodes:
  - id: processor
    path: processor.py
    inputs:
      data: _mod/data
      config: _mod/config    # silently dropped if not wired
    outputs:
      - result

When the parent doesn’t wire config, the input is simply omitted from the expanded node.

Visualization

dora graph renders module boundaries as Mermaid subgraphs, making it easy to see which nodes came from which module:

dora graph dataflow.yml --open

Validation

Validate a standalone module file without a full dataflow:

dora expand --module modules/transform_module.yml

This checks:

• Valid YAML structure
• Module header is present with required name, optional inputs/outputs, and no unknown header fields
• All _mod/ references correspond to declared inputs or optional inputs
• Every declared output is produced by some inner node (counting operator:/operators: and legacy custom: outputs)
• No duplicate node IDs
• Internal wiring is consistent
• Nested module files exist, are relative paths, are acyclic, and stay within the nesting depth limit

Every check above runs recursively: a nested module file is validated in full, not just read for its declared outputs.

A declared output must have exactly one producer. If two inner nodes emit the same output name and that name is also listed in module.outputs, expansion fails rather than silently picking the first producer. module.outputs is a plain list with no output: node/port mapping syntax, so the fix is to rename the internal signal that is not being exported. This applies to dora run and dora build, not just dora expand.

Source-path resolution inside modules

A module’s inner-node path:, and its operators’ shared-library:, python:, and wasm: values, are resolved relative to the module file’s directory, not to the top-level dataflow. A module in modules/nested/ declaring python: op.py refers to modules/nested/op.py.

Exempt from the rewrite: URLs, absolute paths, and the dynamic and shell sentinels (which the daemon matches verbatim).

Two consequences worth knowing:

• A resolved path that escapes the dataflow directory is rejected. A module cannot reach a sibling project via python: ../shared/op.py, and the ../../target/debug/<bin> idiom used by the example dataflows cannot be used from inside a module.
• build: commands still run with the dataflow directory as their working directory. An artifact a build produces must therefore be referenced from where the build put it, which is not the module directory — so build: cargo build -p my-op paired with shared-library: target/debug/my-op inside a module resolves to modules/target/debug/my-op and will not be found. Use an absolute path or keep such nodes in the top-level dataflow.

安全

• Path confinement: Module file paths must resolve within the dataflow’s base directory. Absolute paths and directory traversal (../) outside the base are rejected.
• File size limit: Module files are capped at 1 MB.
• Depth limit: Recursive nesting is capped at 8 levels.
• Param key validation: Parameter keys must be alphanumeric with underscores only.

示例

See examples/module-dataflow/ for a complete working example with a sender, transform module (doubler + filter), and receiver.

dora run examples/module-dataflow/dataflow.yml