Driver Types

Fuzzball V4 includes seven built-in storage drivers. Each driver handles a different type of storage backend. All drivers are built into the Fuzzball platform — no external container-based drivers need to be installed. The on-premises drivers are NFS and Hostpath; the cloud drivers are AWS EFS, Google Cloud Filestore, Azure Files, Oracle Cloud File Storage (FSS), and Oracle Cloud Managed Lustre.

You can list the drivers available on your cluster, along with their required and optional configuration fields, with:

$ fuzzball volume provisioner list-drivers

The NFS driver manages volumes as subdirectories on an NFS export. It is the primary driver for on-premises multi-node clusters where shared storage is provided by an NFS server.

shared-nfs:
  description: "Shared NFS storage"
  driver:
    type: nfs
    target: "nfs-server.internal:/exports/fuzzball"
    version: 4
    options: "hard,intr"
  access: all
  ephemeral: all

When a volume is created, the driver:

1. Mounts the NFS export to a temporary location
2. Validates available capacity on the export
3. Creates a subdirectory named after the volume
4. Sets POSIX ownership (UID/GID) on the directory
5. Unmounts the temporary mount

When a volume is published (mounted into a workflow container), the driver mounts the specific volume subdirectory to the target path:

NFS Export Root (/exports/fuzzball)
├── my-volume/        ← volume directory, mounted to container path
├── project-data/
└── scratch-wf-123/

NFSv3 and NFSv4 have different mount option syntax and behavior. If you specify mount options in the options field, ensure they are compatible with the NFS version you select.

NFS exports configured with root_squash will remap root (UID 0) operations to the nobody user. This affects volume creation and ownership assignment. If your NFS server uses root_squash, configure the export with no_root_squash for the network range where Fuzzball substrate nodes reside, or use the volume’s UID/GID settings to work within the squash rules.

The hostpath driver manages volumes as directories under a base path on the local filesystem. It supports two modes: shared (default) and node-local.

Which one applies depends on whether the base path really is one filesystem across the cluster. Fuzzball cannot tell that from the definition, so local records the answer — see When the path is not shared.

unsharedPath was a second way to say local: true and is deprecated. Existing provisioners keep working unchanged: both spellings mean node-local, and nothing rewrites them, so there is nothing you have to do right now. The field will be removed in a future release, so update definitions to local when convenient.

In shared mode (local: false or omitted), the base path is expected to be available on all nodes in the cluster — typically via a shared filesystem like Lustre, GPFS, or a network mount. Volumes can be accessed from any node.

lustre-storage:
  description: "Lustre shared filesystem"
  driver:
    type: hostpath
    path: /mnt/lustre/fuzzball
  access: all
  ephemeral: all

Shared mode only works if the expectation above holds. A base path that resolves to a different disk on each node looks identical in the definition, and Fuzzball assumes shared unless told otherwise. On a multi-node cluster the volume is then created on one node and the job that mounts it may run on another, where the directory does not exist.

Set local: true to record that the path is per-node:

node-local-store:
  description: "Per-node storage under /srv/fuzzball"
  driver:
    type: hostpath
    path: /srv/fuzzball
    local: true
  access: all
  ephemeral: all

A node-local provisioner serves both persistent and ephemeral volumes, and both behave the same way: the volume is created once, on one node, and Fuzzball places every stage that uses it on that node. The only difference is when the node is chosen — a persistent volume’s node is recorded and known before the workflow starts, while an ephemeral volume’s is decided when its own stage runs.

Both accept the full policy set: access and create govern persistent volumes, ephemeral governs the scratch case.

Ephemeral volumes on a node-local provisioner used to be created once per consuming node instead, giving each stage its own empty directory. A stage that wrote output and a later stage that read it saw different directories, so the handoff silently produced nothing. One volume on one node is what makes that work.

A volume on a node-local provisioner exists on exactly one node. That node is the volume’s holding node: the node whose disk actually contains the data. Fuzzball places a job on the volume’s holding node, and refuses at submission the job shapes that no single placement can satisfy, rather than starting work that cannot succeed.

A job counts as depending on a volume whether it mounts the volume or names it in requires or dependsOn:

The multinode and concurrent task-array rows apply to ephemeral volumes as well as persistent ones. A multinode job mounting an ephemeral volume from a node-local provisioner was accepted in earlier releases, because each node was given its own copy; it is now refused at submission. Use a provisioner backed by shared storage for those jobs.

A persistent volume’s holding node is recorded when the volume is created, imported, or first mounted, and read back at submission. You can choose it explicitly at creation time:

$ fuzzball volume create node-local-store dataset --node 10.0.0.3/7331

Without --node, Fuzzball picks a node that can serve the provisioner and has room for the requested size, skipping those that do not. If no node has room, the creation fails saying how many were considered and how much the roomiest had free. With --node there is no candidate set, so a target without room fails naming that node and its free space.

A volume on a node-local provisioner exists only on its holding node, so what happens when that node becomes unavailable depends on whether it is coming back:

• Cordoned or drained. Stages using the volume queue, and resume when the node returns to service. Fuzzball does not provision a replacement, because a new node cannot hold data written to another node’s disk.
• Deprovisioned. The data is gone with the node, so the workflow fails with an error naming it rather than waiting for a node that will not return.

On a cluster that provisions nodes on demand, a node holding a node-local volume is kept while any stage is still queued to run on it, so it may scale in later than an otherwise idle node would. Without that, the volume would be reclaimed along with the node while the stage that needs it was still waiting.

A workflow submitted when the holding node is already unavailable is refused at submission.

Fuzzball can only place a job on a persistent volume’s holding node if that node is recorded and reachable. Two cases are refused at submission with an explanatory error rather than failing mid-run:

• The volume has no recorded holding node. Volumes created before Fuzzball began recording holding nodes have no node stored, and Fuzzball cannot infer it after the fact. Run fuzzball volume provisioner scan <provisioner> while the holding node is up, and Fuzzball records the holding node for every volume it finds there that has none recorded, which makes those volumes usable again. Otherwise, recreate the volume on a provisioner backed by shared storage.

  This case is not refused on a single-node cluster: there is one node the volume can be on and one node the job can run on, so the missing record changes nothing.

• The recorded holding node is unavailable. No other node can reach the volume, so the job would wait indefinitely. This is refused on clusters of any size, including single-node clusters: if the only node is down, or has re-registered at a different address, it no longer matches the recorded holding node. Bring the node back, or use a volume on a provisioner backed by shared storage.

A scan of a node-local provisioner reaches one node and reports only that node’s volumes, so it cannot describe the provisioner as a whole. Select the node with --node. See Scanning a node-local provisioner for what this means for importing and for stale-volume removal.

The same setting covers fast scratch on a local disk, where the point is the disk rather than the data. Grant only an ephemeral policy and the provisioner hands out scratch and nothing else:

local-nvme:
  description: "Local NVMe scratch"
  driver:
    type: hostpath
    path: /mnt/nvme0n1
    local: true
  ephemeral:
    - compute-jobs

Data on a node-local provisioner lives on one node’s disk. It does not survive the loss of that node, and it cannot be reached from anywhere else. Use shared storage for data that must outlive a node.

When a volume is created, the driver:

1. Validates available capacity at the base path
2. Creates a subdirectory named after the volume
3. Sets POSIX ownership (UID/GID) on the directory

When a volume is published, the driver creates a bind mount from the volume directory to the target path:

Base Path (/mnt/fuzzball)
├── volume-a/         ← bind-mounted to container path
├── volume-b/
└── scratch-wf-456/

The EFS driver manages volumes as access points on an Amazon Elastic File System. It is designed for AWS cloud deployments where shared, elastic storage is needed.

Provide filesystemId or subnetIds, not both. Use filesystemId to attach an existing EFS filesystem (BYO mode). Use subnetIds to have Fuzzball create the EFS filesystem and mount targets automatically (self-provisioned mode).

BYO mode — use an existing EFS filesystem:

aws-efs:
  description: "AWS EFS for shared storage"
  driver:
    type: efs
    filesystemId: fs-12345678
    region: us-west-2
  access: all
  ephemeral: all

Self-provisioned mode — Fuzzball creates the EFS infrastructure:

aws-efs-auto:
  description: "Auto-provisioned AWS EFS"
  driver:
    type: efs
    region: us-west-2
    subnetIds:
      - subnet-abc123
      - subnet-def456
  access: all
  ephemeral: all

The EFS driver is unique among Fuzzball’s drivers — it is a control-plane driver, meaning volume creation and deletion happen via AWS API calls from the Fuzzball control plane, not from a substrate node. This provides faster operations and eliminates the need for a running substrate node during volume management.

When a volume is created, the driver:

1. Creates an AWS EFS access point on the filesystem
2. Sets the access point’s root directory to /<volume-name>
3. Configures POSIX identity via the access point’s PosixUser and CreationInfo (UID, GID, and directory permissions)

When a volume is published (mounted into a workflow container), the driver:

1. Mounts the access point via NFS 4.1 to the target path
2. Mount uses the access point’s root directory for isolation

EFS Filesystem (fs-12345678)
├── /volume-a/    ← access point root, mounted to container
├── /volume-b/    ← separate access point
└── /scratch-123/ ← ephemeral access point

Before creating an EFS provisioner, ensure:

• The EFS filesystem exists and is in the available state
• Mount targets are available in the subnets where Fuzzball nodes run
• Security groups allow NFS traffic (TCP port 2049) from the node subnet
• IAM permissions allow elasticfilesystem:CreateAccessPoint, elasticfilesystem:DeleteAccessPoint, and elasticfilesystem:DescribeAccessPoints

EFS storage is elastic — there are no per-access-point capacity quotas. The size field in volume definitions is accepted but not enforced at the EFS level.

The Azure Files driver manages volumes as subdirectories of an Azure Files NFS share. Each volume is a subdirectory in the share, mounted on substrate nodes over NFSv3. It is designed for Azure cloud deployments.

The Azure Storage account must have the NFS protocol enabled on the file share. SMB shares are not supported by this driver.

azure-files:
  description: "Azure Files NFS storage"
  driver:
    type: azure_files
    accountName: fuzzballstorage
    shareName: workflowio
  access: all
  ephemeral: all

Azure deployments bootstrapped before this release created two built-in provisioners named persistent and ephemeral instead of a single default. Both are backed by the same Azure file share, and neither can be selected by name in a workflow, because persistent and ephemeral are reserved values of a volume’s use field.

Upgrading creates the correct default provisioner but leaves the old pair in place, so no existing volume becomes unreachable. To finish the consolidation:

1. List the volumes still recorded on the old provisioners:

   $ fuzzball volume list --provisioner persistent
   $ fuzzball volume list --provisioner ephemeral

2. Point workflows at default (or omit use entirely to auto-select it).

3. Once no workflows reference them and any remaining ephemeral volumes have drained, remove the old provisioners:

   $ fuzzball volume provisioner remove persistent
   $ fuzzball volume provisioner remove ephemeral

Do not run fuzzball volume provisioner scan default while the old persistent provisioner still holds volume records. All three provisioners share one share-wide directory namespace, so scanning imports duplicate records under default and subsequent lookups by volume name fail with “exists on multiple provisioners”. Delete the old provisioners first.

The OCI FSS driver manages volumes on an Oracle Cloud File Storage Service file system, with parity to the AWS EFS driver. Volumes mount over NFSv3, and per-volume POSIX ownership is enforced server-side by an OCI FSS IdentitySquash policy. It supports both bring-your-own (BYO) mode and self-provisioned mode.

For BYO mode, provide both filesystemId and mountTargetId. For self-provisioned mode, omit them and provide subnetIds; the driver creates the file system and mount target from the compartment, availability domain, and subnets.

BYO mode — use an existing FSS file system and mount target:

oci-fss:
  description: "OCI File Storage"
  driver:
    type: oci_fss
    region: us-ashburn-1
    compartmentId: ocid1.compartment.oc1..aaaaaaaa
    availabilityDomain: "Uocm:US-ASHBURN-AD-1"
    filesystemId: ocid1.filesystem.oc1.iad.aaaaaaaa
    mountTargetId: ocid1.mounttarget.oc1.iad.aaaaaaaa
  access: all
  ephemeral: all

Self-provisioned mode — Fuzzball creates the FSS infrastructure:

oci-fss-auto:
  description: "Auto-provisioned OCI File Storage"
  driver:
    type: oci_fss
    region: us-ashburn-1
    compartmentId: ocid1.compartment.oc1..aaaaaaaa
    availabilityDomain: "Uocm:US-ASHBURN-AD-1"
    subnetIds:
      - ocid1.subnet.oc1.iad.aaaaaaaa
  access: all
  ephemeral: all

Each volume is an export on the FSS file system. Exports are created and deleted through OCI API calls, and volume creation also briefly mounts the new export from a substrate node to set its POSIX owner.

When a volume is created, the driver:

1. Creates an OCI FSS export on the mount target with the export path /<volume-name>
2. Mounts the export from a substrate node and sets the volume’s owner UID/GID on the export root
3. Enables POSIX identity enforcement via the export’s IdentitySquash settings (maps all file operations to the volume’s owner)

When a volume is published (mounted into a workflow container), the driver mounts the export via NFSv3 at the target path, so each volume sees only its own export directory.

When a volume is deleted, the driver removes the export via the OCI API. Data on the underlying file system is retained — the same behavior as EFS access-point deletion.

OCI FSS FileSystem (ocid1.filesystem...)
├── /volume-a/    ← export path, mounted to container
├── /volume-b/    ← separate export
└── /scratch-123/ ← ephemeral export

Before creating an OCI FSS provisioner, ensure:

For BYO mode:

• The file system exists and is in the ACTIVE state
• A mount target exists in a subnet accessible from Fuzzball substrate nodes

For self-provisioned mode:

• The compartment and availability domain are correct and have capacity
• Subnet IDs are provided for automatic mount-target creation

For both modes:

• Security lists allow NFS traffic (TCP/UDP ports 111 and 2048-2050) from the node subnet
• IAM policies allow managing file systems, exports, and mount targets (for example, via a manage file-family policy)

OCI FSS storage is elastic — there are no per-export capacity quotas. The size field in volume definitions is accepted but not enforced at the FSS level. POSIX ownership is enforced via IdentitySquash on the export, which maps all file operations to the volume’s configured UID/GID.

The OCI Managed Lustre driver backs a provisioner with a single Oracle Cloud File Storage with Lustre file system — a high-performance parallel filesystem for HPC and AI/ML workloads. Each Fuzzball volume is a subdirectory of the shared file system, owned by the volume’s POSIX uid/gid (the same model the GCP Filestore driver uses).

The driver operates in one of two modes, selected by which fields the definition sets:

• Static (bring-your-own) — filesystemId is set. The file system already exists (created outside Fuzzball, for example through the OCI console), and Fuzzball only creates and removes per-volume subdirectories inside it. The provisioner never creates, resizes, or deletes the file system itself; its only use of the OCI API is read-only, to resolve the mount coordinates (MGS address and file-system name). Operationally this is the same model as the hostpath driver pointed at a pre-mounted shared filesystem.
• Dynamic (self-provisioned) — filesystemId is omitted and subnetIds is set. Fuzzball creates the file system itself in the configured compartment, availability domain, and subnet, sized by lustreCapacityGbs / lustrePerformanceTier.

If a definition sets both filesystemId and subnetIds, filesystemId takes precedence and the provisioner operates statically — no file system is created.

In both modes the capacity is fixed once the file system exists: Fuzzball does not grow or shrink a Lustre file system after creation, whether it created the file system or not. To add capacity, expand the file system directly in OCI (console, CLI, or API — subject to OCI’s capacity step sizes). Fuzzball uses the new capacity transparently, since volumes are subdirectories of the file system and lustreCapacityGbs is only applied at creation.

Volumes mount with the native Lustre client (mount -t lustre <mgs>@tcp:/<fsname>/<volume>), so every substrate node that runs workloads using this provisioner must have the Lustre client kernel modules installed.

Substrate node prerequisite: Lustre client. Unlike NFS, the Lustre client is not part of a stock Linux install. Substrate node images must include the client packages matching the node kernel (kmod-lustre-client and lustre-client, e.g. Lustre 2.15.x from Whamcloud) and an LNet configuration that is not tied to a NIC name — cloud instances get different interface names per shape (ens3, enp0s5, …), so bind LNet by subnet instead, e.g. options lnet ip2nets="tcp0 10.0.0.*" in /etc/modprobe.d/lustre.conf.

Volume operations that fail with unknown filesystem type 'lustre' or mount.lustre: ... No such device indicate the client packages or kernel modules are missing on the node; LNetError: ... Error -100 starting up LNI tcp indicates the LNet configuration references an interface that does not exist on that node.

In dynamic (self-provisioned) mode, size and throughput come from lustreCapacityGbs / lustrePerformanceTier, defaulting to OCI’s minimum (31.2 TB at the 125 MBps/TB tier). The capacity is set once at creation — Fuzzball does not expand the file system as it fills (expand it directly in OCI if more space is needed). Self-provisioning is asynchronous: provisioner add returns immediately with the provisioner in Provisioning status, and file-system creation takes ~10–15 minutes before it transitions to Ready (or Error on failure) — check with fuzzball volume provisioner info <name>.

OCI Managed Lustre has a 31.2 TB minimum capacity across all performance tiers (~28.4 TiB usable as reported by df) — there is no smaller option. A self-provisioned file system is therefore a large, billable resource; use BYO mode to share one filesystem across provisioners where possible.

Static (BYO) mode — use an existing Managed Lustre file system; Fuzzball only manages subdirectories within it:

oci-lustre:
  description: "OCI Managed Lustre"
  driver:
    type: oci_lustre
    region: us-ashburn-1
    compartmentId: ocid1.compartment.oc1..aaaaaaaa
    availabilityDomain: "Uocm:US-ASHBURN-AD-1"
    filesystemId: ocid1.lustrefilesystem.oc1.iad.aaaaaaaa
  access: all
  ephemeral: all

Dynamic (self-provisioned) mode — Fuzzball creates the Lustre file system:

oci-lustre-auto:
  description: "Auto-provisioned OCI Managed Lustre"
  driver:
    type: oci_lustre
    region: us-ashburn-1
    compartmentId: ocid1.compartment.oc1..aaaaaaaa
    availabilityDomain: "Uocm:US-ASHBURN-AD-1"
    subnetIds:
      - ocid1.subnet.oc1.iad.aaaaaaaa
  access: all
  ephemeral: all

The Filestore driver manages volumes as subdirectories on a Google Cloud Filestore instance. It is the driver for GCP deployments, where Filestore provides the shared NFS storage that an NFS server provides on-premises.

When target names only the endpoint, the share defaults to workflowio. Write host:/shareName to select a different share.

The Filestore driver does not support node-local mode. A Filestore instance is reachable from every node in the cluster, so local: true is rejected.

gcp-filestore:
  description: "Google Cloud Filestore for shared storage"
  driver:
    type: filestore
    target: "10.224.0.2:/workflowio"
  access: all
  ephemeral: all

Using the default share name, and a DNS endpoint rather than an IP:

gcp-filestore:
  description: "Google Cloud Filestore, default share"
  driver:
    type: filestore
    target: "filestore.internal"
  access: all
  ephemeral: all

Each volume is a subdirectory of the share. Fuzzball creates the directory when the volume is created, applies the requested ownership and mode, and mounts the share over NFS on the node that runs the job. Removing a volume removes its directory.

The Filestore instance itself is created outside Fuzzball — by your deployment tooling or by hand — and the provisioner points at the endpoint it exposes. Fuzzball does not create or delete Filestore instances.

The AWS EFS and Azure Files drivers accept an optional nfsSyncMode field that controls whether NFS volumes are mounted with synchronous or asynchronous writes:

For most clusters the auto default is correct. Override it only when you have a specific durability or performance requirement.

Use this guide to select the right driver for your deployment: