V3 to V4 Storage Migration

When upgrading from Fuzzball V3 to V4, the storage system is automatically migrated from the V3 three-tier model (Storage Driver, Storage Class, Storage Volume) to the V4 two-tier model (Storage Provisioner, Storage Volume). The upgrade creates V4 provisioners from your existing storage classes, links your volumes to them, and backfills volume ownership.

However, the automated migration handles database records only. Depending on how your V3 storage classes were configured, the actual directories on your storage backend may need adjustment so that V4 can locate them. This guide walks through the post-upgrade steps to get your storage fully functional.

Before upgrading to V4, generate a migration report on your V3 cluster. The report previews exactly what the automated migration will do: which storage classes will be converted to provisioners, what driver type each will use, and which volumes will be linked. This report is your reference for verifying the upgrade succeeded and for planning any post-upgrade path reconciliation.

Every cluster upgrade should generate this report before the V4 upgrade begins. The report is read-only and makes no changes to the cluster.

• fuzzball CLI configured and authenticated against the V3 cluster
• kubectl access to the fuzzball-system namespace
• jq installed
• The storage-migrate-preflight bundle from CIQ Support

Obtain the storage-migrate-preflight.zip bundle from CIQ Support. Then:

# unzip storage-migrate-preflight.zip
# cd storage-migrate-preflight
# ./storage-migrate-preflight.sh

The tool discovers your cluster, connects to the operator pod, and runs a dry-run of the migration. Example output:

storage-migrate: previewing migration for cluster 491506d4-33fd-6849-5a22-e832418a3615
storage-migrate: found 2 storage class(es)
storage-migrate: processing class "persistent" (id=e8778dde-..., persistent=true)
  org e700253c-...: 2 volume(s) in class "persistent"
    -> will create provisioner "persistent" (driver=HOSTPATH)
    -> volume "migration-test-vol-1" will be linked to provisioner "persistent"
    -> volume "migration-test-vol-2" will be linked to provisioner "persistent"
storage-migrate: processing class "ephemeral" (id=7f33ca42-..., persistent=false)
  no unmigrated volumes for class "ephemeral"
storage-migrate: completed successfully — 2 class(es) processed

What each line means:

• “will create provisioner” — a V4 provisioner will be created for this storage class in this organization, with the specified driver type
• “will be linked to provisioner” — this existing volume will be associated with the new provisioner
• “no unmigrated volumes” — the storage class exists but has no volumes that need migration

Save the JSON report for reference during the post-upgrade steps:

# ./storage-migrate-preflight.sh --json | jq . > preflight-report.json

Review the report and confirm:

1. The driver type mapping is correct for each storage class (HOSTPATH, NFS, or EFS)
2. The expected volumes will be linked to the right provisioners
3. No unexpected warnings about corrupted class definitions or orphaned volumes

If the preflight tool reports a driver type that doesn’t match your storage backend (for example, HOSTPATH when you use NFS), contact CIQ Support before proceeding with the upgrade. The driver type can be corrected post-upgrade, but it is easier to address beforehand.

Once the report looks correct, proceed with the V4 upgrade. See Updating Fuzzball for the upgrade procedure.

The remainder of this guide covers steps to complete after the V4 upgrade has finished. The operator runs storage-migrate automatically during reconciliation, which converts your V3 storage classes to V4 provisioners as previewed in the preflight report.

Before starting the post-upgrade steps:

• Fuzzball V4 upgrade is complete and the operator has reconciled successfully
• fuzzball CLI is installed and authenticated against the upgraded cluster
• kubectl access to the fuzzball-system namespace
• Filesystem access to the storage backend (NFS mount point, host node, or AWS console)
• Pre-upgrade migration report has been generated and reviewed (see above)

If you upgraded without generating a preflight report, the automatic migration still ran during the upgrade. You can check the operator logs for equivalent output:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

Proceed with Step 1 below using the log output in place of the preflight report.

After the V4 upgrade completes, confirm that provisioners were created from your V3 storage classes.

List all provisioners:

# fuzzball volume provisioner list

You should see one provisioner per V3 storage class per organization. For example, if you had a persistent storage class and two organizations, you will see two provisioners both named persistent, one for each organization.

Compare against the preflight-report.json you saved before the upgrade — the provisioner count and names should match the dry-run output.

Inspect a specific provisioner:

# fuzzball volume provisioner info persistent

List volumes linked to a provisioner:

# fuzzball volume list --provisioner persistent

If provisioner list returns no results, the automatic migration may not have run. Check the operator logs:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

If you see errors, refer to the troubleshooting section at the end of this guide.

V3 storage classes used template-based directory naming that placed volumes inside subdirectories. For example, a V3 persistent class on a typical on-premises deployment stored volumes at:

/mnt/fuzzball-sharedfs/persistent/<group-UUID>/

V4 provisioners expect volumes as flat directories directly under the driver’s base path:

/mnt/fuzzball-sharedfs/<volume-name>/

If your V3 storage classes used a basePath, subDir, or share parameter (most default configurations do), then the directories on your storage backend are nested one level deeper than V4 expects. You need to create symbolic links to bridge this gap.

If your V3 volumes are already at the top level of the storage backend with no nesting, skip to Step 4.

AWS EFS users: EFS requires additional reconciliation beyond symlinks. The V3 EFS CSI driver created access points without the Name tag that V4 expects, and stored data at nested paths (/persistent/<id>) rather than V4’s flat layout (/<id>). Both new access points and filesystem symlinks are needed. Contact CIQ Support for the EFS-specific migration procedure, or refer to the internal runbook for the complete step-by-step instructions.

SSH into a host that can access the storage backend (the NFS server, or any node that mounts the shared filesystem) and list the top-level directories:

# ls -la /mnt/fuzzball-sharedfs/

If you see directories like persistent/ or ephemeral/ that contain your actual volume data as subdirectories, path reconciliation is needed.

Never copy or move customer data. All path reconciliation uses symbolic links, which are instant, zero-copy, and fully reversible. Removing a symlink does not affect the original data.

This is the most common scenario. V3 default classes create a persistent/ and/or ephemeral/ subdirectory under the shared filesystem, with per-group or per-workflow UUID directories inside.

On the machine that hosts or mounts your shared filesystem, run the discover-and-link script for each V3 sub-path that contains volumes:

# ./discover-and-link.sh /mnt/fuzzball-sharedfs persistent

Scanning V3 volumes in: /mnt/fuzzball-sharedfs/persistent
  LINK: /mnt/fuzzball-sharedfs/e700253c-... -> /mnt/fuzzball-sharedfs/persistent/e700253c-.../
  LINK: /mnt/fuzzball-sharedfs/a1b2c3d4-... -> /mnt/fuzzball-sharedfs/persistent/a1b2c3d4-.../

Done. Created 2 symlink(s), skipped 0.

If you also have an ephemeral/ sub-path with data you want to preserve:

# ./discover-and-link.sh /mnt/fuzzball-sharedfs ephemeral

Ephemeral volumes are transient — they are created per-workflow and normally cleaned up after the workflow completes. In most cases, you can safely skip the ephemeral/ directory. Only run the script for ephemeral/ if you have data there that was not cleaned up and you need to preserve it.

#!/bin/bash
# discover-and-link.sh — Creates symlinks to reconcile V3 storage paths for V4.
#
# Usage:
#   ./discover-and-link.sh <storage-root> <v3-sub-path>
#
# Examples:
#   ./discover-and-link.sh /mnt/fuzzball-sharedfs persistent
#   ./discover-and-link.sh /srv/nfs/fuzzball persistent

set -euo pipefail

usage() {
    echo "Usage: $0 <storage-root> <v3-sub-path>"
    echo ""
    echo "Arguments:"
    echo "  storage-root   Absolute path to the provisioner base directory"
    echo "                 (e.g., /mnt/fuzzball-sharedfs)"
    echo "  v3-sub-path    V3 subdirectory containing volume directories"
    echo "                 (e.g., persistent or ephemeral)"
    exit 1
}

if [ $# -ne 2 ]; then
    usage
fi

STORAGE_ROOT="$1"
V3_SUBPATH="$2"
V3_DIR="${STORAGE_ROOT}/${V3_SUBPATH}"

if [ ! -d "$STORAGE_ROOT" ]; then
    echo "ERROR: Storage root does not exist: $STORAGE_ROOT" >&2
    exit 1
fi

if [ ! -d "$V3_DIR" ]; then
    echo "ERROR: V3 directory not found: $V3_DIR" >&2
    echo "This may mean the storage class did not use a '$V3_SUBPATH' sub-path," >&2
    echo "or the storage is mounted at a different location." >&2
    exit 1
fi

created=0
skipped=0

echo "Scanning V3 volumes in: $V3_DIR"

for vol_dir in "$V3_DIR"/*/; do
    [ -d "$vol_dir" ] || continue

    vol_name=$(basename "$vol_dir")
    link_path="${STORAGE_ROOT}/${vol_name}"

    if [ -e "$link_path" ] || [ -L "$link_path" ]; then
        echo "  SKIP: $link_path already exists"
        skipped=$((skipped + 1))
        continue
    fi

    echo "  LINK: $link_path -> $vol_dir"
    ln -s "$vol_dir" "$link_path"
    created=$((created + 1))
done

echo ""
echo "Done. Created $created symlink(s), skipped $skipped."
if [ "$created" -gt 0 ]; then
    echo "Verify with: ls -la $STORAGE_ROOT"
fi

After creating symlinks, confirm they point to the correct locations:

# ls -la /mnt/fuzzball-sharedfs/

You should see symlinks alongside the original V3 directories:

drwxr-xr-x  4 root root 4096 May 26 10:00 persistent/
drwxr-xr-x  2 root root 4096 May 26 10:00 ephemeral/
lrwxrwxrwx  1 root root   52 May 26 10:05 e700253c-... -> /mnt/fuzzball-sharedfs/persistent/e700253c-.../
lrwxrwxrwx  1 root root   52 May 26 10:05 a1b2c3d4-... -> /mnt/fuzzball-sharedfs/persistent/a1b2c3d4-.../

The default storage path is /mnt/fuzzball-sharedfs. If your cluster uses a different path, adjust accordingly. You can check the provisioner’s driver configuration to find the correct base path:

# fuzzball volume provisioner info persistent -o yaml

Look for the path (hostpath driver) or target (NFS driver) field in the driver configuration.

GCP and Azure V3 configurations typically used a subDir parameter (e.g., subDir=persistent) under a shared NFS export. The same symlink approach applies — mount the NFS export and run the script against the appropriate sub-path.

OCI provisions separate NFS exports for /persistent and /ephemeral. Mount each export independently and verify whether the volume directories are nested. If they are, run the symlink script against each mount point.

The automatic storage-migrate process links all V3 volumes that existed in the database. However, if you have volume directories on the storage backend that were not tracked in V3 (for example, directories created manually or by external tools), you can use the scan command to discover and import them.

The scan command discovers volumes by listing directories at the provisioner’s base path. It only discovers real directories, not symbolic links. If you created symlinks in Step 3, those symlinks make already-migrated volumes mountable but will not appear in scan results. This is expected — those volumes are already in the V4 database from the automatic migration.

If you need to discover volumes inside a V3 sub-path (like persistent/) that are not already in the V4 database, edit the provisioner to point directly at the sub-path instead of using symlinks:

# fuzzball volume provisioner edit <provisioner-name>

Change path: /mnt/fuzzball-sharedfs to path: /mnt/fuzzball-sharedfs/persistent, then run the scan.

Preview what the scan will find:

# fuzzball volume provisioner scan persistent --dry-run

The scan now performs two-way reconciliation: it discovers new volumes and identifies stale volumes (tracked in Fuzzball but not found on the backend). Symlinked directories are followed during the scan, so volumes created via symlinks in Step 3 will be discovered correctly. Always review the --dry-run output before running without --dry-run.

If the discovered volumes look correct, run the scan without --dry-run to import new volumes and remove stale records:

# fuzzball volume provisioner scan persistent

Imported volumes are marked as external — Fuzzball will not delete their backing data when the volume is removed. This is a safety feature for data that was not originally created by V4.

Verify the imported volumes:

# fuzzball volume list --provisioner persistent

Submit a simple workflow that mounts one of your migrated volumes to confirm data is accessible:

version: v4
volumes:
  test-vol:
    use: persistent
    name: <your-volume-name>
jobs:
  verify:
    image: ubuntu:22.04
    mounts:
      /data: test-vol
    script: |
      echo "Listing migrated volume contents:"
      ls -la /data/

# fuzzball workflow start verify-migration.yaml

Monitor the workflow and confirm the volume contents are visible:

# fuzzball workflow events <workflow-id>

If the workflow runs successfully and your data is listed, the migration is complete.

The automatic storage-migrate did not run or encountered an error. Check the operator logs:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system | grep storage-migrate

If the migration failed, it can be re-run safely (it is idempotent). Contact CIQ support for assistance with manual re-runs.

The V4 driver cannot find the volume directory at the expected path. Verify:

1. The provisioner’s driver base path is correct:

   # fuzzball volume provisioner info <provisioner-name> -o yaml

2. The volume directory (or symlink) exists at <base-path>/<volume-name>/:

   # ls -la /mnt/fuzzball-sharedfs/<volume-name>

3. If the path is missing, create a symlink to the actual data location (see Step 3).

If you ran the scan before creating symlinks, it may have discovered the V3 directory structure names (like persistent or ephemeral) as volume names. To fix this:

1. Remove the incorrectly imported volumes (this only removes the database record, not the data):

   # fuzzball volume delete <provisioner> <wrong-name> --preserve-data

2. Create the correct symlinks (Step 3).

3. Re-run the scan:

   # fuzzball volume provisioner scan <provisioner>

The automatic migration auto-detects the driver type from the V3 class configuration. If it chose incorrectly (for example, hostpath instead of NFS), edit the provisioner to correct it:

# fuzzball volume provisioner edit <provisioner-name>

This opens the provisioner definition in your editor. Update the driver section with the correct type and configuration, then save.

This is rare but can occur with highly restricted NFS configurations. As an alternative, you can edit the provisioner’s driver configuration to point directly at the V3 sub-path:

# fuzzball volume provisioner edit <provisioner-name>

Change the path (hostpath) or target (NFS) to include the V3 sub-path. For example, change path: /mnt/fuzzball-sharedfs to path: /mnt/fuzzball-sharedfs/persistent.