Updating Fuzzball

Updating Fuzzball in an on-prem environment consists of updating the Substrate installation running on compute nodes, the Orchestrate cluster, and ensuring that client CLIs are up-to-date.

Assuming that you installed Fuzzball substrate from the Depot yum/dnf repository, updating substrate is simply a matter of running dnf update or similar.

Substrate is backward compatible with Orchestrate, so the newest version of Substrate will always run with your Orchestrate installation. But don’t forget to update Substrate when you update Orchestrate since an older Substrate installation may be incompatible with a newer version of Orchestrate.

It’s easy to update your Fuzzball Orchestration cluster using the same K8s Custom Resource Definition (CRD) that installs Fuzzball. Simply repeat the steps that you used to install the Fuzzball Operator with a new version.

You can safely update from one patch release to another non-sequentially. (e.g Updating from v2.1.8 to v2.1.10 is supported.) But you should avoid skipping major or minor releases when updating.

First, make sure that you are logged into CIQ Depot.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Now simply update the version number and run the same commands that you originally ran to install the Fuzzball Operator the first time.

# VERSION="" # insert the updated version number

# CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

# helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=local-path"

Running this command will automatically re-deploy the Orchestrate cluster with updated images. You can monitor the progress of the update with the same command that you ran when first installing the cluster.

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

Under unusual circumstances, the update may not start automatically. For instance, if you incorrectly specify your Depot username or password the deployment will not proceed. Correcting this mistake will not cause the deployment to restart automatically. In cases like this, you have the option to manually start the redeployment process. Here is an example.

# kubectl get deployments -A
NAMESPACE            NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
cert-manager         cert-manager                           3/3     3            3           26h
cert-manager         cert-manager-cainjector                1/1     1            1           26h
cert-manager         cert-manager-webhook                   1/1     1            1           26h
cert-manager         trust-manager                          1/1     1            1           26h
fuzzball-system      fuzzball-operator-controller-manager   1/1     1            1           26h
[...snip]

# kubectl rollout restart deployment/fuzzball-operator-controller-manager -n fuzzball-system
deployment.apps/fuzzball-operator-controller-manager restarted

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
[...snip]
Resources:
    + 7 created
    265 unchanged

Duration: 10s

2025-05-29T21:05:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballOrchestrate","name":"fuzzball-orchestrate","uid":"22ca168d-a7a7-46f8-a144-33b42937f50f","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"11472"}, "reason": "DeploymentSucceeded"}
2025-05-29T21:05:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballorchestrate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballOrchestrate", "FuzzballOrchestrate": {"name":"fuzzball-orchestrate"}, "namespace": "", "name": "fuzzball-orchestrate", "reconcileID": "162d9224-5d75-49a7-829b-cc6ad8381ebe"}

In the example above, we first determine the name of the deployment that we want to restart (fuzzball-operator-controller-manager). We then use the proper kubectl command to restart the deployment process. Finally, we use the appropriate command to monitor the logs as the deployment proceeds to successful completion.

The exact method that you use to update the Fuzzball CLI will depend on the way in which you originally installed it. See the CLI installation documentation for more information.

Older versions of the Fuzzball CLI are not guaranteed to work with newer versions of Orchestrate, so it’s a good practice to keep it up to date. This may require announcing updates to your users so that they can update the CLI on their personal machines too!

You can update your deployment in AWS by creating and applying a change set in CloudFormation. The steps to do so follow.

Before updating your Fuzzball Cluster take steps to ensure that no workflows are running and no users are accessing Orchestrate.

Ensure that you have access to 10 or more elastic IP addresses in the region where you are performing the Fuzzball update. See the requirements page for more information.

Log into your AWS account, navigate to CloudFormation, and find your stack.

From there, click “Update stack” and select “Create a change set”.

You can select the options to “Replace existing template” using an “Amazon S3 URL”. Supply the URL to the updated CloudFormation template listed in the Amazon Marketplace.

On the next screen, you can update values. Values from the previous deployment will already be populated in the new template. Be sure to update the “FuzzballVersion” value to reflect the version of the CloudFormation template that you used.

Review the changes that will be applied and acknowledge the message regarding CloudFormation’s ability to create new IAM resources.

Once you’ve gone through those screens you are ready to execute the change set. Select “Execute change set” in the upper right corner and specify your preferences if the change set provisioning fails or if you need to rollback the changes for any reason.

Pressing the “Execute change set” button on the screen will kick off the update process. You can watch as CloudFormation updates the stack resources and cleans up old resources. When finished, you will see the “UPDATE_COMPLETE” message.