Updating Fuzzball

Updating Fuzzball in an on-prem environment consists of updating the Substrate installation running on compute nodes, the Orchestrate cluster, and ensuring that client CLIs are up-to-date.

Assuming that you installed Fuzzball substrate from the Depot yum/dnf repository, updating substrate is simply a matter of running dnf update or similar.

Substrate is backward compatible with Orchestrate, so the newest version of Substrate will always run with your Orchestrate installation. But don’t forget to update Substrate when you update Orchestrate since an older Substrate installation may be incompatible with a newer version of Orchestrate.

It’s easy to update your Fuzzball Orchestration cluster using the same K8s Custom Resource Definition (CRD) that installs Fuzzball. Simply repeat the steps that you used to install the Fuzzball Operator with a new version.

You can safely update from one patch release to another non-sequentially. (e.g Updating from v2.1.8 to v2.1.10 is supported.) But you should avoid skipping major or minor releases when updating.

First, make sure that you are logged into CIQ Depot.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Now simply update the version number and run the same commands that you originally ran to install the Fuzzball Operator the first time.

# VERSION="" # insert the updated version number

# CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

# helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=local-path"

Running this command will automatically re-deploy the Orchestrate cluster with updated images. You can monitor the progress of the update with the same command that you ran when first installing the cluster.

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

Under unusual circumstances, the update may not start automatically. For instance, if you incorrectly specify your Depot username or password the deployment will not proceed. Correcting this mistake will not cause the deployment to restart automatically. In cases like this, you have the option to manually start the redeployment process. Here is an example.

# kubectl get deployments -A
NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager         cert-manager                           3/3     3            3           26h
cert-manager         cert-manager-cainjector                1/1     1            1           26h
cert-manager         cert-manager-webhook                   1/1     1            1           26h
cert-manager         trust-manager                          1/1     1            1           26h
fuzzball-system      fuzzball-operator-controller-manager   1/1     1            1           26h
[...snip]

# kubectl rollout restart deployment/fuzzball-operator-controller-manager -n fuzzball-system
deployment.apps/fuzzball-operator-controller-manager restarted

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
[...snip]
Resources:
    + 7 created
    265 unchanged

Duration: 10s

2025-05-29T21:05:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballOrchestrate","name":"fuzzball-orchestrate","uid":"22ca168d-a7a7-46f8-a144-33b42937f50f","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"11472"}, "reason": "DeploymentSucceeded"}
2025-05-29T21:05:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballorchestrate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballOrchestrate", "FuzzballOrchestrate": {"name":"fuzzball-orchestrate"}, "namespace": "", "name": "fuzzball-orchestrate", "reconcileID": "162d9224-5d75-49a7-829b-cc6ad8381ebe"}

In the example above, we first determine the name of the deployment that we want to restart (fuzzball-operator-controller-manager). We then use the proper kubectl command to restart the deployment process. Finally, we use the appropriate command to monitor the logs as the deployment proceeds to successful completion.

The exact method that you use to update the Fuzzball CLI will depend on the way in which you originally installed it. See the CLI installation documentation for more information.

Older versions of the Fuzzball CLI are not guaranteed to work with newer versions of Orchestrate, so it’s a good practice to keep it up to date. This may require announcing updates to your users so that they can update the CLI on their personal machines too!

AWS deployments are created using the Fuzzball CLI, and can also be updated via the update command. The CLI applies the change to the underlying CloudFormation stack for you, so there is no need to create and execute a change set by hand in the AWS console.

Unlike the GCP and Azure update commands, fuzzball cluster aws update has no --version flag. The target Fuzzball version is the version embedded in the Fuzzball CLI itself (the CLI ships a CloudFormation template stamped with its own build version). To update to a new Fuzzball version, first install the CLI for that version, then run the update command. See the CLI installation documentation for how to install a specific version.

Before updating your Fuzzball Cluster take steps to ensure that no workflows are running and no users are accessing Orchestrate.

Fuzzball can be deployed in AWS using either Marketplace or non-Marketplace mode. Once a cluster has been deployed, you cannot switch between modes via update; destroy the stack and redeploy if you need to change modes. The update command detects the existing mode and enforces this automatically.

Ensure that you have access to 10 or more elastic IP addresses in the region where you are performing the Fuzzball update. See the requirements page for more information.

Much like the interactive deployment method, you can update Fuzzball on AWS with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of confirming the parameters. Values from the previous deployment are pre-populated, so in most cases you can accept them as-is and let the update apply the new version embedded in your CLI.

Include the --dry-run flag to see what will happen before you actually execute the command.

$ fuzzball cluster aws update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster aws update \
  --region "$REGION" \
  --stack-name "$STACK_NAME" \
  --non-interactive

The update applies the Fuzzball version embedded in the CLI you are running, so make sure you have installed the CLI for the version you want before running the command. For a full list of available update options, use the fuzzball cluster aws update --help command.

If your cluster was deployed in non-Marketplace mode, the update must re-hydrate the updated images into your account’s ECR, so you also need to supply Depot credentials (--depot-user + --depot-access-token, or the DEPOT_USER and DEPOT_ACCESS_TOKEN environment variables) or --container-images-dir for offline updates. See the non-Marketplace deployment section of the AWS deployment guide for details.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.

GCP deployments are created using the Fuzzball CLI, and can also be updated via the update command.

Much like the interactive deployment method, you can update Fuzzball on GCP with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of filling in all necessary parameters.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster gcp update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster gcp update \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --deployment-name "unique-name" \
  --version "$VERSION" \
  --non-interactive

For a full list of available update options, use the fuzzball cluster gcp update --help command.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.

Azure deployments are created using the Fuzzball CLI, and can also be updated via the update command.

Much like the interactive deployment method, you can update Fuzzball on Azure with an interactive approach by running update without any options. The Fuzzball CLI will guide you through the process of filling in all necessary parameters.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster azure update

You can also update Fuzzball non-interactively by supplying the parameters in option/argument pairs and using the --non-interactive flag like so:

$ fuzzball cluster azure update \
  --subscription "$SUBSCRIPTION_ID" \
  --location "$LOCATION" \
  --resource-group "$RESOURCE_GROUP" \
  --version "$VERSION" \
  --non-interactive

For a full list of available update options, use the fuzzball cluster azure update --help command.

When updating to a release that includes a PostgreSQL major version upgrade (for example, upgrading to v3.3.0 which moves to PostgreSQL 16), the Fuzzball operator handles the in-place database migration automatically during the update process. No manual database migration is required.