Updating Fuzzball

Updating Fuzzball in an on-prem environment consists of updating the Substrate installation running on compute nodes, the Orchestrate cluster, and ensuring that client CLIs are up-to-date.

Fuzzball Substrate is installed on each compute node from the Depot yum/dnf repository. When you update Orchestrate you should also update the substrate packages on every compute node — in particular the fuzzball-substrate-orchestrate extension — pinned to the same version as Orchestrate.

Do not simply run a bare dnf update. That pulls the newest available packages, which can overshoot your Orchestrate version (for example installing a 4.0.x extension against a 3.4.x Orchestrate) and conflicts with the recommendation to avoid skipping major or minor releases. The fuzzball-substrate-orchestrate version must match your Orchestrate version.

Set VERSION to match your Orchestrate version (without the leading v) and install the pinned packages, exactly as you did when installing Substrate on the compute node:

# VERSION="4.1.0" # matches orchestrate version without the leading v

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

After the packages are updated, restart Substrate on the compute node so the new binaries take effect:

# systemctl restart fuzzball-substrate.service

You can confirm the extension version installed on the node with rpm:

# rpm -q fuzzball-substrate-orchestrate

fb version reports only the CLI and Agent versions, not the fuzzball-substrate-orchestrate extension, so version skew between Orchestrate and a compute node is not otherwise visible. The base Substrate platform is backward compatible with Orchestrate, but the fuzzball-substrate-orchestrate extension carries Orchestrate-matched functionality (for example the data-copy / S3 mover), so an older extension can silently misbehave against a newer Orchestrate. Keep the extension pinned to the Orchestrate version, and don’t forget to update Substrate whenever you update Orchestrate.

It’s easy to update your Fuzzball Orchestration cluster using the same K8s Custom Resource Definition (CRD) that installs Fuzzball. Simply repeat the steps that you used to install the Fuzzball Operator with a new version.

You can safely update from one patch release to another non-sequentially. (e.g Updating from v2.1.8 to v2.1.10 is supported.) But you should avoid skipping major or minor releases when updating.

First, make sure that you are logged into CIQ Depot.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Now simply update the version number and run the same commands that you originally ran to install the Fuzzball Operator the first time.

# VERSION="" # insert the updated version number

# CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

# helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=local-path"

Running this command will automatically re-deploy the Orchestrate cluster with updated images. You can monitor the progress of the update with the same command that you ran when first installing the cluster.

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

Under unusual circumstances, the update may not start automatically. For instance, if you incorrectly specify your Depot username or password the deployment will not proceed. Correcting this mistake will not cause the deployment to restart automatically. In cases like this, you have the option to manually start the redeployment process. Here is an example.

# kubectl get deployments -A
NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager         cert-manager                           3/3     3            3           26h
cert-manager         cert-manager-cainjector                1/1     1            1           26h
cert-manager         cert-manager-webhook                   1/1     1            1           26h
cert-manager         trust-manager                          1/1     1            1           26h
fuzzball-system      fuzzball-operator-controller-manager   1/1     1            1           26h
[...snip]

# kubectl rollout restart deployment/fuzzball-operator-controller-manager -n fuzzball-system
deployment.apps/fuzzball-operator-controller-manager restarted

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
[...snip]
Resources:
    + 7 created
    265 unchanged

Duration: 10s

2025-05-29T21:05:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballOrchestrate","name":"fuzzball-orchestrate","uid":"22ca168d-a7a7-46f8-a144-33b42937f50f","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"11472"}, "reason": "DeploymentSucceeded"}
2025-05-29T21:05:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballorchestrate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballOrchestrate", "FuzzballOrchestrate": {"name":"fuzzball-orchestrate"}, "namespace": "", "name": "fuzzball-orchestrate", "reconcileID": "162d9224-5d75-49a7-829b-cc6ad8381ebe"}

In the example above, we first determine the name of the deployment that we want to restart (fuzzball-operator-controller-manager). We then use the proper kubectl command to restart the deployment process. Finally, we use the appropriate command to monitor the logs as the deployment proceeds to successful completion.

The exact method that you use to update the Fuzzball CLI will depend on the way in which you originally installed it. See the CLI installation documentation for more information.

Older versions of the Fuzzball CLI are not guaranteed to work with newer versions of Orchestrate, so it’s a good practice to keep it up to date. This may require announcing updates to your users so that they can update the CLI on their personal machines too!

AWS deployments are created using the Fuzzball CLI, and can also be updated via the update command. The CLI applies the change to the underlying CloudFormation stack for you, so there is no need to create and execute a change set by hand in the AWS console.

Pass --version to select the target Fuzzball version. When the flag is omitted, the target version is the version embedded in the Fuzzball CLI itself (the CLI ships a CloudFormation template stamped with its own build version).

Before updating your Fuzzball Cluster take steps to ensure that no workflows are running and no users are accessing Orchestrate.

Fuzzball can be deployed in AWS using either Marketplace or non-Marketplace mode. Once a cluster has been deployed, you cannot switch between modes via update; destroy the stack and redeploy if you need to change modes. The update command detects the existing mode and enforces this automatically.

Ensure that you have access to 10 or more elastic IP addresses in the region where you are performing the Fuzzball update. See the requirements page for more information.

Much like the interactive deployment method, you can update Fuzzball on AWS with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of confirming the parameters. Values from the previous deployment are pre-populated, so in most cases you can accept them as-is and let the update apply the new version embedded in your CLI.

Include the --dry-run flag to see what will happen before you actually execute the command.

$ fuzzball cluster aws update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster aws update \
  --region "$REGION" \
  --stack-name "$STACK_NAME" \
  --non-interactive

Pass --version to select the target Fuzzball version; when omitted, the update applies the Fuzzball version embedded in the CLI you are running. For a full list of available update options, use the fuzzball cluster aws update --help command.

If your cluster was deployed in non-Marketplace mode, the update must re-hydrate the updated images into your account’s ECR, so you also need to supply Depot credentials (--depot-user + --depot-access-token, or the DEPOT_USER and DEPOT_ACCESS_TOKEN environment variables) or --container-images-dir for offline updates. See the non-Marketplace deployment section of the AWS deployment guide for details.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.

GCP deployments are created using the Fuzzball CLI, and can also be updated via the update command.

Much like the interactive deployment method, you can update Fuzzball on GCP with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of filling in all necessary parameters.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster gcp update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster gcp update \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --deployment-name "unique-name" \
  --version "$VERSION" \
  --non-interactive

For a full list of available update options, use the fuzzball cluster gcp update --help command.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.

Azure deployments are created using the Fuzzball CLI, and can also be updated via the update command.

Much like the interactive deployment method, you can update Fuzzball on Azure with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of filling in all necessary parameters.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster azure update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster azure update \
  --subscription "$SUBSCRIPTION_ID" \
  --location "$LOCATION" \
  --resource-group "$RESOURCE_GROUP" \
  --version "$VERSION" \
  --non-interactive

For a full list of available update options, use the fuzzball cluster azure update --help command.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.

Docker Compose deployments are updated in place with fuzzball cluster docker-compose update. The command operates on a single deployment, selected with --deployment-name (defaulting to default).

$ fuzzball cluster docker-compose update --help

Add --dry-run to any update invocation to preview the changes without applying them.

If the stack is running, update prompts before recreating the affected containers to apply the change. Pass -y to skip the prompt in scripts.

Use --version to move the deployment to a specific Fuzzball version:

$ fuzzball cluster docker-compose update --version v4.1.0

Alternatively, use --upgrade to bump the deployment to the Fuzzball image tag that matches the version of the fuzzball CLI you are running:

$ fuzzball cluster docker-compose update --upgrade

The two flags are mutually exclusive.

Older versions of the Fuzzball CLI are not guaranteed to work with newer versions of Orchestrate, so keep the CLI up to date. See the CLI installation documentation for how to update it.

The --reset-* flags rotate the deployment’s generated credentials. Each flag takes the new value as its argument:

$ fuzzball cluster docker-compose update --reset-database-password "$NEW_DB_PASSWORD"

$ fuzzball cluster docker-compose update --reset-keycloak-password "$NEW_KEYCLOAK_PASSWORD"

$ fuzzball cluster docker-compose update --reset-owner-password "$NEW_OWNER_PASSWORD"

The --reset-* flags require the stack to be running. Start it first with fuzzball cluster docker-compose up.

You can switch an existing deployment between single- and multi-node, and between the regular and GPU-enabled substrate images, without redeploying:

$ fuzzball cluster docker-compose update --gpu

• --tls-renew renews the wildcard TLS leaf certificate for the deployment.
• --sync-files refreshes the deployment’s files from the CLI’s embedded copies while preserving your existing .env. This is useful after upgrading the CLI to pick up changes to the generated docker-compose.yaml.

$ fuzzball cluster docker-compose update --sync-files

If an update does not behave as expected, see the Troubleshooting section in the Deployment guide for how to inspect the stack files directly.