Distributed Workflows with Generic Multinode

Fuzzball supports a generic multinode implementation that allows you to run custom commands or scripts across multiple nodes without being tied to a specific parallel computing framework implementation. This is useful when you want full control over how processes are launched and coordinated across nodes, or when your application uses its own distributed communication mechanism.

Unlike the openmpi, mpich, or gasnet implementations where Fuzzball automatically wraps your command with the appropriate launcher (e.g., mpirun), the generic implementation gives you direct access to the multinode environment through a set of environment variables. You are responsible for launching and coordinating processes on remote nodes yourself.

When using the generic implementation, the following environment variables are available both as shell variables and for expansion within command arguments:

The MULTINODE_SSH_WRAPPER variable points to a wrapper that allows you to execute commands on remote nodes. Use it like: $MULTINODE_SSH_WRAPPER <hostname> <command> [args...].

By default, only the head node’s standard output and standard error are captured in the workflow logs. If you need output from other nodes to appear in the logs, set the MULTINODE_WRAPPER_FORWARD_STREAMS environment variable to 1. This forwards stdout and stderr from all worker nodes to the head node for centralized log collection.

The generic implementation supports variable expansion in command arguments. This allows you to reference multinode environment variables directly in your command line, and they will be expanded before execution.

Here is an example Fuzzfile that uses the generic implementation to run an MPI application with explicit control over how mpirun is invoked:

version: v1
jobs:
  generic-multinode:
    image:
      uri: oras://docker.io/anderbubble/openmpi-hello-world.sif@sha256:dc92ea0c541a8d9f30b4d6b78bfc57bd2fe9806689d93ece6069c6ed6519fa9a
    env:
      - PATH=/usr/lib64/openmpi/bin:/usr/local/bin:/usr/bin:/bin
    command:
      - mpirun
      - -H
      - $MULTINODE_HOSTLIST
      - --mca
      - plm_rsh_agent
      - $MULTINODE_SSH_WRAPPER
      - -np
      - $MULTINODE_TOTAL_SLOTS
      - /usr/lib64/openmpi/bin/mpi_hello_world
    resource:
      cpu:
        cores: 2
        affinity: NUMA
      memory:
        size: 1GB
    multinode:
      nodes: 2
      implementation: generic

In this example, the command field references $MULTINODE_HOSTLIST, $MULTINODE_SSH_WRAPPER, and $MULTINODE_TOTAL_SLOTS directly. These are expanded by the generic wrapper before the command is executed. This gives you full control over how the MPI launcher is configured, including the ability to pass custom options.

You can also use a script to coordinate work across nodes. In this mode, the multinode environment variables are available as standard shell environment variables within your script.

version: v1
jobs:
  generic-multinode-script:
    image:
      uri: docker://mirror.gcr.io/library/alpine:3.16
    env:
      - MULTINODE_WRAPPER_FORWARD_STREAMS=1
    script: |
      #!/bin/sh
      HOSTNAME=$(hostname)

      sleep_func() {
	      echo "$HOSTNAME is the head node"
        sleep 3
      }

      sleep_func &
      pid=$!

      IFS=','

      for HOST in ${MULTINODE_HOSTLIST_NOSLOTS}; do
        if [ "$HOSTNAME" != "$HOST" ]; then
          $MULTINODE_SSH_WRAPPER $HOST echo "$HOST is a worker node"
        fi
      done

      wait $pid
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GB
    multinode:
      nodes: 2
      implementation: generic

In this example, the script iterates over the hostlist and uses $MULTINODE_SSH_WRAPPER to run a command on each remote node. The MULTINODE_WRAPPER_FORWARD_STREAMS=1 environment variable ensures that output from the worker nodes is forwarded to the head node and captured in the workflow logs.

After running this workflow, the logs will contain output from all nodes:

generic-multinode-script-0 is the head node
generic-multinode-script-1 is a worker node

The generic implementation is a good fit when:

• You need to run a distributed framework that doesn’t match one of the supported implementations.
• You want explicit control over how processes are launched across nodes.
• Your application has its own mechanism for distributed coordination.
• You want to run custom shell scripts that orchestrate work across multiple nodes.

For standard MPI workloads, the openmpi or mpich implementations are recommended as they handle the mpirun invocation automatically. See Distributed Workflows with MPI for details.