Provisioner Management

Storage provisioners are managed with the fuzzball volume provisioner command. Organization owners can manage provisioners for their organization.

Fuzzball creates a default provisioner on cloud deployments so a fresh install has working storage without manual configuration. Two different bootstrap paths seed defaults, and it matters which one your deployment runs – they differ on when the provisioner is created, who it belongs to, and what its backing storage looks like.

Every cloud deployment gets one default provisioner for the initial organization, created when the Orchestrate service starts. Fuzzball selects the driver based on which cloud provisioner is enabled:

The initial-organization default reaches Ready as soon as it is created – the backing storage already exists.

Organizations running on the Fuzzball hosted service each get their own default provisioner, created when the organization is created. It is named default-<cluster-name> so that when several Orchestrate clusters are joined under a FuzzballFederate, each cluster’s per-organization default has a distinct name.

On AWS, per-organization defaults are self-provisioning: the EFS filesystem is created asynchronously in the background when the provisioner is created, so it briefly shows Provisioning before transitioning to Ready. On GCP the per-organization default is Ready immediately – volumes are subdirectories of the shared instance, which already exists.

List provisioners to confirm the default is present:

$ fuzzball volume provisioner list

Organization owners can edit, replace, or remove the default provisioner the same way as any other provisioner. Removing it means workflows that reference a volume without naming a provisioner will fail until a replacement is added.

Naming details. The per-organization default is named default-<cluster-name>. On Kubernetes deployments the cluster name is set from the Orchestrate cluster’s ingress domain (falling back to unset-cluster when neither an explicit name nor a domain is available), so the bare name default is only produced on single-binary deployments where the cluster name is empty.

A deployment whose default provisioner is scoped to a segment carries that in the name too, as default-<segment> – a compose deployment produces default-docker. The suffix makes it visible in this listing which provisioner serves which nodes.

Create a provisioner from a YAML definition file:

$ fuzzball volume provisioner add my-provisioner -f provisioner.yaml

Self-provisioned cloud provisioners create their backing infrastructure asynchronously. When a cloud driver definition (AWS EFS, Oracle Cloud FSS, Oracle Cloud Managed Lustre) omits its backing-resource IDs, provisioner add returns immediately with the provisioner in Provisioning status while the filesystem is created in the background. The provisioner is not usable until it transitions to Ready — check progress with fuzzball volume provisioner info <name>. Creation time varies by backend (OCI Managed Lustre takes ~10–15 minutes; EFS and FSS are faster). A failed creation transitions the provisioner to Error. Bring-your-own definitions (all resource IDs supplied) are validated synchronously and come up Ready within the add call.

If you omit -f, your $EDITOR opens with a template that you can fill in and save:

$ fuzzball volume provisioner add my-provisioner

You can override the editor for a single invocation:

$ EDITOR=nvim fuzzball volume provisioner add my-provisioner

You can also scan the storage backend after creation to import existing volumes. This is a two-step process — first add the provisioner, then scan:

$ fuzzball volume provisioner add my-provisioner -f provisioner.yaml

$ fuzzball volume provisioner scan my-provisioner

The --scan flag on provisioner add combines both steps into one command:

$ fuzzball volume provisioner add my-provisioner -f provisioner.yaml --scan

See Provisioner Definitions for the YAML format reference.

List all provisioners in the current organization:

$ fuzzball volume provisioner list

ID                                   | NAME              | STATUS | DETAIL | CLUSTER
80e204ec-ee2f-3485-9282-33f77a9a3318 | my-nfs            | Ready  |        | my-cluster
a1b2c3d4-e5f6-7890-abcd-ef1234567890 | local-scratch     | Ready  |        | my-cluster

Add -o json or -o yaml for machine-readable output:

$ fuzzball volume provisioner list -o json

DETAIL explains a provisioner’s status when the status alone is not enough. It is empty for a healthy provisioner and carries a short explanation for one that failed:

$ fuzzball volume provisioner list

ID                                   | NAME       | STATUS | DETAIL                                              | CLUSTER
80e204ec-ee2f-3485-9282-33f77a9a3318 | team-a-efs | Error  | Storage account limit reached. Contact your cloud   | my-cluster
                                     |            |        | administrator to raise the quota.                   |

The same explanation appears in fuzzball volume provisioner info output alongside the status.

A provisioner that self-provisions its backing storage — an EFS provisioner given subnetIds rather than a filesystemId — starts in Provisioning while that work happens in the background, then moves to Ready. If it fails, DETAIL records why.

A provisioner that is not Ready is skipped when Fuzzball selects a provisioner for a volume. If a workflow reports that no provisioner matches, check STATUS and DETAIL before assuming the definition is wrong.

View the full definition, status, and driver information of a provisioner:

$ fuzzball volume provisioner info my-nfs

The default output is YAML. Use -o json for JSON output:

$ fuzzball volume provisioner info my-nfs -o json

Modify an existing provisioner’s definition. This opens the current definition in your $EDITOR:

$ fuzzball volume provisioner edit my-nfs

You can also apply changes from a file:

$ fuzzball volume provisioner edit my-nfs -f updated-definition.yaml

Changing a provisioner’s driver type or target may make existing volumes inaccessible. Verify volume compatibility before modifying driver settings on a provisioner with active volumes.

You can create and edit storage provisioners through the Fuzzball Web UI as an alternative to the CLI.

1. Navigate to Volume Provisioners under the Infrastructure section in the sidebar.
2. Click Add Provisioner.
3. Fill in the provisioner name, driver type, and driver-specific configuration fields.
4. Use the Policy fields to configure access, create, and ephemeral permissions. Each policy has a mode selector: choose All groups to grant to everyone, or Specific groups and click Add group to add each group name in its own input field.
5. Click Save.

The Web UI form validates your configuration before submission and displays errors inline if required fields are missing or invalid.

Turning on the Local storage toggle removes the Create and Access policy fields and leaves Ephemeral. A node-local provisioner serves ephemeral volumes only, so those two policies do not apply to it — see Node-local provisioners.

1. Navigate to Volume Provisioners under the Infrastructure section in the sidebar.
2. Click on the provisioner name in the list to open the detail panel.
3. In the detail panel, click Edit to open the provisioner form.
4. Modify the fields as needed. Each policy field offers the same All groups / Specific groups selector used when creating a provisioner.
5. Click Save to apply your changes.

The Web UI provisioner form includes dedicated fields for access policies. You no longer need to edit raw YAML to configure group-based permissions.

The provisioner list page includes a detail panel with two sections:

• Details — driver type and configuration, status, and metadata such as creation and update times
• Permissions — the create, access, and ephemeral policies with their granted group lists

Click any provisioner in the list to open the detail panel on the right side of the page.

A policy that was never set reads Not permitted, because policies are default-deny — an unset policy grants nothing. On a node-local provisioner, Create and Access always read that way.

Delete a provisioner from the organization. You are prompted for confirmation unless you pass -y/--yes:

$ fuzzball volume provisioner remove my-old-provisioner

By default, the provisioner must have no associated volumes. If volumes still exist, the command returns an error with a hint to use --ignore-volumes.

Use --ignore-volumes to remove a provisioner that still has associated volumes. All volume records are soft-deleted, but physical data on the backing store is not touched:

$ fuzzball volume provisioner remove my-old-provisioner --ignore-volumes

--ignore-volumes is rejected if any of the provisioner’s volumes are currently in use by running workflows. Stop those workflows before removing the provisioner.

To skip the confirmation prompt (for example, in scripts), combine with -y:

$ fuzzball volume provisioner remove my-old-provisioner --ignore-volumes -y

Removing a provisioner — with or without --ignore-volumes — does not delete data on the storage backend. To re-associate the data with Fuzzball:

1. Create a new provisioner pointing to the same backend location:

   $ fuzzball volume provisioner add replacement-provisioner -f provisioner.yaml

2. Scan the backend to re-discover and import the volumes:

   $ fuzzball volume provisioner scan replacement-provisioner

Re-imported volumes appear as externally managed volumes. Subsequent deletes of these volumes will remove the Fuzzball record but will not send delete operations to the storage backend.

The scan command performs a two-way reconciliation between the storage backend and the Fuzzball database. It works consistently across NFS, hostpath, EFS, and OCI FSS provisioners. A node-local hostpath provisioner is the one exception: it presents different storage on each node, so a scan can only describe the node it reaches. See Scanning a node-local provisioner below.

The scan operation:

1. Discovers volumes that exist on the storage backend but are not yet tracked in Fuzzball, and imports them.
2. Identifies stale volumes — volumes tracked in Fuzzball whose backing data was not found during the scan — and removes their Fuzzball records.
3. Records the holding node for any discovered volume on a node-local provisioner that does not have one recorded. Without a recorded holding node, Fuzzball cannot place a job that uses the volume, so scanning is the recovery path for volumes created before Fuzzball started recording holding nodes.

This is useful after:

• Migration from V3
• Manual volume creation on the backend
• Removing and re-creating a provisioner
• Out-of-band deletion of storage (e.g., manual removal of NFS directories)

A node-local provisioner can be scanned, but a scan reaches one node and sees only that node’s storage, so what it can discover and remove is narrower. See Scanning a node-local provisioner.

By default, scan applies changes: it imports newly discovered volumes and removes stale volumes from the Fuzzball database.

$ fuzzball volume provisioner scan my-nfs
Discovered 3 volume(s):
  - project-data
  - shared-models
  - team-archive

Imported 2 new volume(s), 1 already existed:
  team-archive         already existed
  project-data         imported (id: a1b2c3d4-e5f6-7890-abcd-ef1234567890)
  shared-models        imported (id: b2c3d4e5-f6a7-8901-bcde-f12345678901)

Removed 1 stale volume(s) (backing data was already gone):
  old-dataset          (id: 80e204ec-ee2f-3485-9282-33f77a9a3318)

In this example, team-archive was already tracked by Fuzzball, so it is reported as already existed rather than imported. Imported volumes become available for use in workflows immediately. Stale volumes are soft-deleted from the Fuzzball database; their backing data is already gone, so no storage is touched.

Pass --dry-run to preview what scan would do without applying any changes. No volumes are imported, no stale records are removed, and no holding nodes are recorded:

$ fuzzball volume provisioner scan my-nfs --dry-run
Discovered 3 volume(s):
  - project-data
  - shared-models
  - team-archive

Would remove 2 stale volume(s):
  old-dataset          (id: a1b2c3d4-e5f6-7890-abcd-ef1234567890)
  temp-scratch         (id: f1e2d3c4-b5a6-0987-fedc-ba9876543210)

Without --dry-run, scan removes stale volume records immediately. Because a stale volume’s backing data is already gone, no storage is deleted — but the Fuzzball record is. Run a dry-run scan first to confirm the expected set of removals, especially if the backend may be only temporarily unreachable. A misconfigured provisioner — for example a wrong backend path or incorrect AWS/OCI credentials — can also make healthy volumes appear stale, because their backing data is not visible to the scan; a dry run surfaces this before any records are removed.

A hostpath provisioner with local: true presents different storage on each node, so its persistent volumes are spread across nodes instead of living in one location every node can reach. Each volume has a holding node — the node whose disk contains its data. See When the path is not shared for what that setting means.

A scan reaches one node and lists only that node’s volumes, which changes two things:

• Discovery is partial. Volumes on nodes the scan did not reach are not listed. The discovered set describes one node, not the provisioner.
• Stale removal is restricted. Fuzzball can only treat a volume as missing if it has already recorded that volume’s holding node as the scanned node. Volumes recorded on another node, and volumes with no recorded holding node, are never proposed for removal — their absence from this listing says nothing about whether their data still exists.

Scanning also records the holding node for any discovered volume that does not have one recorded, which makes volumes created before holding-node tracking usable again. Here both discovered volumes were already tracked by Fuzzball, so neither is newly imported:

$ fuzzball volume provisioner scan node-local
Discovered 2 volume(s):
  - nlvol
  - project-data

Imported 0 new volume(s), 2 already existed:
  nlvol                already existed
  project-data         already existed

Recording a holding node produces no distinct output — already existed is reported whether or not a holding node was recorded for that volume. To confirm the result, submit a workflow that uses the volume, or check the orchestrate service logs.

A volume already recorded on another node keeps that record: a scan of one node does not reassign volumes belonging to others. Where the scanned node also has a directory by that name, the scan reports it as a warning rather than acting on it – a volume name identifies one volume on one node, so two nodes holding the same name is something only you can resolve:

1 warning(s):
  - volume "dataset" is recorded on node 10.0.0.3:7331, but node 10.0.0.9:7331 also has a
    directory by that name. Only the recorded node is used; rename or remove one of them so
    the name identifies a single volume

Warnings are reported by --dry-run too, so you can find this before importing anything.

Use --node to choose which node the scan reaches:

$ fuzzball volume provisioner scan node-local --node 10.0.0.3/7331

Without --node, Fuzzball picks a node that can serve the provisioner, which on a node-local provisioner may not be the one holding the volumes you care about.

Holding nodes are recorded only when scan applies changes. --dry-run does not record them, so a volume refused at submission for having no recorded holding node stays refused until you run scan without --dry-run.

A volume becomes stale when:

• The backing storage (e.g., an NFS directory) was deleted manually outside of Fuzzball
• The storage backend was reconfigured or reformatted
• A provisioner was pointed to a different storage target
• Cloud resources were deleted via the cloud provider’s console or API

Stale volumes remain in the Fuzzball database but cannot be used in workflows because their data no longer exists. Running scan detects this condition — any volume tracked in Fuzzball but not found on the backend during the scan — and removes the orphaned database record. Fuzzball does not delete any data on the storage backend.

On a node-local provisioner, absence from a scan is not sufficient evidence that a volume is stale, because the scan only sees one node. Removal there is limited to volumes whose recorded holding node is the scanned node; see Scanning a node-local provisioner.

The scan operation connects to the storage backend (NFS mount, local filesystem, AWS API, OCI API) to list volumes. Scan failures usually indicate network, connectivity, or credential issues, not Fuzzball problems. Verify that:

• For NFS/hostpath: the backend path is accessible from the Fuzzball control plane
• For EFS: AWS credentials and IAM permissions are correct
• For OCI FSS: OCI credentials and filesystem/mount-target configuration are correct

All storage driver types support scanning; the operation queries whatever backend the driver is configured for. The most common cases are:

• NFS and hostpath — scans the configured directory path for subdirectories, each representing a volume
• EFS — queries the AWS API for access points associated with the configured filesystem
• OCI FSS — queries the OCI API for exports on the configured file system

In all cases, volume names are sanitized consistently to ensure compatibility with Fuzzball’s naming requirements.

The Web UI provides a graphical workflow for scanning and importing volumes from the Storage Provisioners page.

1. Locate the provisioner and click Scan
2. The scan runs a dry-run first. A Scan Results dialog appears with two tables:
   • Discovered volumes — volumes found on the backend but not yet tracked in Fuzzball
   • Stale volumes to remove — volumes tracked in Fuzzball but not found on the backend
3. Review the results and click Import all to apply. All discovered volumes are imported and stale records are removed in a single operation
4. A summary confirms how many volumes were imported, how many already existed, and how many stale records were removed

Import all applies both sides of the reconciliation: it imports all discovered volumes and removes all stale records. There is no per-volume selection — review the dry-run results carefully before clicking Import all, especially if some volumes may appear stale due to temporary connectivity issues.

View the driver types that Fuzzball supports and their configuration fields:

$ fuzzball volume provisioner list-drivers

This returns a description of each driver type along with its required and optional configuration fields. Use -o json or -o yaml for structured output.

You can export an existing provisioner’s definition and use it as a starting point for a new one. This is useful for creating a copy of the default provisioner and adjusting its access policies:

$ fuzzball volume provisioner info default -o yaml > new-provisioner.yaml

Edit the file to change the name and policies:

$ vi new-provisioner.yaml

Then create the new provisioner from the modified definition:

$ fuzzball volume provisioner add team-storage -f new-provisioner.yaml

After a fresh Fuzzball installation:

1. Verify your context is active:

   $ fuzzball context list

2. Check if a default provisioner was created automatically:

   $ fuzzball volume provisioner list

   Cloud deployments start with a default provisioner already created – see The default provisioner. Check fuzzball volume provisioner list before creating one manually.

3. If no provisioner exists, create one from a definition file:

   $ fuzzball volume provisioner add my-storage -f provisioner.yaml

4. Verify the provisioner is ready:

   $ fuzzball volume provisioner list

5. If you have existing data on the backend, scan and import:

   $ fuzzball volume provisioner scan my-storage

For guidance on migrating from V3 storage classes to V4 provisioners, see the V3 to V4 Storage Migration Guide and the V4 Breaking Changes page.