Provisioner Definitions

A provisioner definition is a YAML file that describes how a storage provisioner operates: which driver to use, how to connect to the storage backend, and which groups have access. Provisioner definitions are applied with fuzzball volume provisioner add.

Provisioner definitions use a name-keyed format where the provisioner name is the top-level key:

my-provisioner:
  description: "Shared NFS storage for development"
  driver:
    type: nfs
    target: "nfs-server:/export/data"
  access: all
  ephemeral: all

The provisioner name (my-provisioner above) is set by the top-level key in the YAML file. It is also passed as the first argument to fuzzball volume provisioner add.

The driver section configures the storage backend. All driver types share the type field; remaining fields depend on the driver type.

For EFS provisioners, provide either filesystemId (BYO mode — use an existing EFS filesystem) or subnetIds (self-provisioned mode — Fuzzball creates the filesystem and mount targets automatically). Do not provide both at creation time.

subPath confines every volume on a provisioner to a subdirectory of the backing store. Two provisioners can then share one filesystem without their volumes colliding on name — useful when several organizations share a single Filestore instance or hostpath mount, since volume names are only unique within a provisioner.

team-a-storage:
  description: "Team A's volumes, scoped to their own subdirectory"
  driver:
    type: filestore
    target: "10.224.0.2:/workflowio"
    subPath: team-a
  access: all
  ephemeral: all

A volume named data on this provisioner lives at team-a/data on the share. The prefix is internal to the driver: volume names, CLI output, and workflow definitions are unaffected, so users see data exactly as they would without subPath.

Rules:

• Relative to the backing share. A leading / is rejected.
• Must not contain ...
• Supported only by the hostpath and Filestore drivers. The cloud drivers address volumes by cloud resource rather than by path, and reject the field.
• Immutable after creation. Existing volumes live under the current value, so changing it would orphan them. Create a new provisioner instead.

Omitting subPath leaves paths exactly as they were before the field existed, so adding it to your vocabulary does not change any existing provisioner.

Each of the three policy fields (access, create, ephemeral) accepts one of two forms:

Grant to all groups:

access: all

Grant to specific groups:

access:
  - engineering
  - datascience
  - research

Policies that are omitted or empty grant no access (default-deny). See Access Policies for details on the three-tier permission model.

Annotations are string key-value pairs used for provisioner selection in workflow volume definitions. When a workflow specifies annotations: on a volume, Fuzzball matches those annotations against provisioner annotations to select the right provisioner.

my-provisioner:
  driver:
    type: nfs
    target: "nfs-server:/export/fast"
  annotations:
    tier: fast
    region: us-west
  access: all

See Annotations and Selection for details on how matching works.

An NFS provisioner for a multi-node on-prem cluster with per-group access control:

shared-nfs:
  description: "Shared NFS storage for all teams"
  driver:
    type: nfs
    target: "nfs-server.prod.internal:/vol/shared"
    options: "hard,intr"
  access: all
  create:
    - data-engineers
    - mlops
  ephemeral: all
  annotations:
    tier: shared

A hostpath provisioner for a single-node deployment or a node with a shared clustered filesystem (Lustre, GPFS):

lustre-storage:
  description: "Lustre shared filesystem"
  driver:
    type: hostpath
    path: /mnt/lustre/fuzzball
  access: all
  create:
    - storage-admins
  ephemeral: all

A hostpath provisioner for node-local NVMe scratch storage. Volumes created with this provisioner exist only on the node where they were created:

local-nvme:
  description: "Local NVMe scratch storage"
  driver:
    type: hostpath
    path: /mnt/nvme0n1
    local: true
  ephemeral:
    - compute-jobs
    - batch-workers

When local: true is set, volumes are only accessible on the node where they were created. Fuzzball’s scheduler ensures jobs using these volumes are placed on the correct node.

A provisioner for an existing AWS EFS filesystem:

aws-efs:
  description: "AWS EFS shared cloud storage"
  driver:
    type: efs
    filesystemId: fs-12345678
    region: us-west-2
  access: all
  create:
    - engineering
  ephemeral: all

A provisioner where Fuzzball creates the EFS filesystem and mount targets:

aws-efs-auto:
  description: "Auto-provisioned AWS EFS"
  driver:
    type: efs
    region: us-west-2
    subnetIds:
      - subnet-abc123
      - subnet-def456
  access: all
  create:
    - engineering
  ephemeral: all

The EFS driver creates AWS EFS access points for each volume. Volume creation and deletion are control-plane operations that do not require a substrate node.

A segment is an optional string identifier assigned to a storage provisioner. Segments ensure that substrate-side image staging works correctly when provisioners serve different storage backends. Each segment maintains its own staged SIF copy on its local shared filesystem so that nodes in the segment can access it at job startup. Without segments, a SIF staged on one provisioner’s filesystem would not be reachable by nodes using a different provisioner’s filesystem.

As a secondary benefit, segments also isolate the image-pull lock per segment, allowing concurrent pulls of the same URI across segments.

Segments are required when provisioners point to different storage backends:

• Separate filesystems: Provisioners backed by different NFS servers, hostpath directories, or cloud filesystems need segments so each stages images locally
• Geographic or zonal separation: Provisioners in different availability zones or regions where cross-zone filesystem access is unavailable or impractical
• Multi-tenant isolation: Teams using separate storage provisioners benefit from independent image staging scopes

Add a segment field at the top level of the provisioner definition:

team-a-storage:
  description: "Storage for Team A"
  driver:
    type: nfs
    target: "nfs-a.internal:/export"
  segment: team-a
  access: all
  ephemeral: all

team-b-storage:
  description: "Storage for Team B"
  driver:
    type: nfs
    target: "nfs-b.internal:/export"
  segment: team-b
  access: all
  ephemeral: all

In this example, each provisioner points to a different NFS server. The segment field ensures each stages images locally on its own filesystem. Nodes serving team-a-storage stage cached SIF images on nfs-a.internal, and nodes serving team-b-storage stage them on nfs-b.internal.

A storage provisioner’s segment must match the segment configured on at least one node provisioner definition in the cluster configuration. Substrate nodes provisioned by that definition carry the segment label, and only those nodes serve the storage provisioner. If a provisioner declares a segment but no node provisioner definition in the cluster carries a matching segment, workflows that mount its volumes are rejected at submission.

The segment field is immutable after the provisioner is created. To change a provisioner’s segment, delete and re-create the provisioner.

If the segment field is omitted, the provisioner is unsegmented: it can be served by any substrate node, and all unsegmented provisioners share a single image staging scope. This is appropriate for single-provisioner deployments or when all provisioners share the same underlying filesystem.

Segment identifiers are lowercase DNS-label-style strings: lowercase letters, digits, and hyphens, starting and ending with a letter or digit, at most 63 characters. Use descriptive names that reflect your organizational structure:

• Team or project names: research, production, staging
• Geographic regions: us-west, eu-central
• Storage tiers: fast-nvme, standard-hdd

Segments do not affect volume data isolation or access policies. They do, however, constrain placement: a segmented provisioner is served only by substrate nodes carrying the matching segment label; jobs that mount its volumes run on nodes in that segment; and a single job cannot mount volumes from two different segments. Volumes created on different provisioners with the same segment share staged images but their data remains isolated on separate storage backends.

To modify an existing provisioner’s definition, use fuzzball volume provisioner edit:

$ fuzzball volume provisioner edit my-provisioner

This opens the current definition in your $EDITOR. Save and close to apply the changes. You can also export the current definition to a file, modify it, and re-apply:

$ fuzzball volume provisioner info my-provisioner -o yaml > provisioner.yaml

$ vi provisioner.yaml

$ fuzzball volume provisioner edit my-provisioner -f provisioner.yaml