Deployment

Fuzzball on AWS supports two deployment modes:

• Marketplace mode (default): Uses AWS Marketplace Elastic Container Registry (ECR) for images. This mode enables usage-based billing through AWS Marketplace metering, and it is recommended for most installations.
• Non-Marketplace mode (opt-in): Populates a customer-owned ECR with images from CIQ Depot or offline tarballs, and disables Marketplace metering. This method is required for restricted AWS partitions, air-gapped environments, or locked-down accounts.

Once deployed, a stack’s mode is immutable; updates preserve the existing mode and reject any attempt to switch between Marketplace and non-Marketplace modes. See the updating documentation for mode preservation details.

This guide covers both deployment methods. Choose the correct method for your environment.

$ fuzzball cluster aws preflight

$ fuzzball cluster aws preflight --provision-roles

$ fuzzball cluster aws deploy

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --permissions-boundary-arn "arn:aws:iam::123456789012:policy/my-boundary-policy" \
  --non-interactive

$ fuzzball cluster aws deploy \
  --depot-user <your-depot-username> \
  --depot-access-token <your-depot-access-token> \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ export DEPOT_USER=<your-depot-username>
$ export DEPOT_ACCESS_TOKEN=<your-depot-access-token>

$ fuzzball cluster aws deploy \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ ls /path/to/offline-images/
aws-pulumi-runner.tar       fuzzball-openapi.tar
fuzzball-admin.tar          fuzzball-orchestrate.tar
fuzzball-audit-archive.tar  fuzzball-substrate-bridge.tar
fuzzball-jetstream.tar      fuzzball-substrate-orchestrate.tar
fuzzball-operator.tar       fuzzball-sync.tar

$ fuzzball cluster aws deploy \
  --container-images-dir /path/to/offline-images \
  --domain "$DOMAIN" \
  --organization-admin "admin@example.com" \
  --cluster-admin-arns "arn:aws:iam::123456789012:user/fuzzballAdmin" \
  --instance-types "t3a.2xlarge,p3.2xlarge" \
  --non-interactive

$ fuzzball cluster aws deploy \
  --depot-user <user> \
  --depot-access-token <token> \
  --depot-host depot.internal.example.com \
  ...

$ fuzzball cluster aws update \
  --depot-user <user> \
  --depot-access-token <token>

Error: stack "fuzzball-20260101-120000" was deployed in non-Marketplace mode; updating it requires --depot-user and --depot-access-token (or DEPOT_USER / DEPOT_ACCESS_TOKEN env vars) so the new image versions can be hydrated into the customer ECR. Alternatively, pass --container-images-dir to load container images from local tar files (Helm charts still come from depot).

Error: stack "fuzzball-20260101-120000" was deployed against AWS account 123456789012 but the current session is account 987654321098; re-run with the AWS profile used to deploy this stack.

Error: stack "fuzzball-20260101-120000" was deployed against AWS region us-west-2 but the current session is region us-east-1; re-run with --region us-west-2.

$ fuzzball cluster aws delete
$ fuzzball cluster aws cleanup

Once deployed, the following subcommands let you manage the cluster:

Use fuzzball cluster aws <subcommand> --help for the full list of options for each command.

If there are any left-over resources from a previous deployment in your GCP project, they may interfere with your ability to deploy a fresh Fuzzball cluster. Please make sure that all resources from previous deployments have been fully and successfully deleted before initiating a new one. You can use the following command to forcibly remove old Fuzzball resources after destroying your GCP deployment if necessary:

$ fuzzball cluster gcp cleanup

To deploy Fuzzball in GCP, you must log in via Helm to the OCI registry that you set up during the Requirements section.

To create an access token and use it to log into the OCI registry via Helm, execute the following command:

$ gcloud auth configure-docker ${REGION}-docker.pkg.dev

$ gcloud auth print-access-token | helm registry login ${REGION}-docker.pkg.dev \
  --username oauth2accesstoken \
  --password-stdin

This token is short-lived (~1 hour). If a deployment takes longer than that, re-run helm registry login with a fresh token. The gcloud auth configure-docker step configures the OCI credential helper and only needs to be run once.

Once you are logged in, you are ready to deploy Fuzzball!

The simplest way to deploy is using interactive mode.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster gcp deploy

The CLI will prompt you for all required parameters including the following:

• GCP project
• Region and zone
• Fuzzball version (defaults to the CLI version if omitted)
• Domain name
• Organization owner email

For non-interactive deployments, you can use the following. (See the Requirements section for information on setting these environment variables.)

$ fuzzball cluster gcp deploy \
  --project "$PROJECT_ID" \
  --region "$REGION" \
  --zone "$ZONE" \
  --version "$VERSION" \
  --domain "$SELECTED_DOMAIN" \
  --dns-zone-name "$MANAGED_ZONE" \
  --dns-zone-project "$PROJECT_ID" \
  --keycloak-owner-email "admin@example.com" \
  --deployment-name "unique-name" \
  --instance-types=n1-standard-4,n2-standard-8,g2-standard-8 \
  --non-interactive

For a full list of the options and arguments that can be specified during deployment, use the fuzzball cluster gcp deploy --help command.

A successful deployment will create a fuzzball cli context for the cluster in ${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/fuzzball.yaml and print out a basic summary of the cluster properties including commands to set up kubectl access to the underlying Kubernetes cluster (see below). You can also obtain similar information later by running

$ fuzzball cluster gcp info --project $PROJECT_ID --region $REGION

 SUCCESS  GCP authentication verified for project 'PROJECT'
 INFO  Searching for Fuzzball deployments to view info for...
 SUCCESS  Found 1 Fuzzball deployment(s):

  1. DEPLOYMENT_NAME (Version: v3-3-0, Region: us-central1, Status: ACTIVE)

 INFO  Using deployment: DEPLOYMENT_NAME

 INFO  Deployment Information

  Deployment:     DEPLOYMENT_NAME
  Project:        PROJECT
  Region:         REGION
  Status:         ACTIVE
  Version:        v3.3.0
  Domain:         DOMAIN
  Created:        2026-04-08t09-20-19z

 INFO  Cluster URLs

  API:            https://api.DOMAIN
  UI:             https://ui.DOMAIN
  Keycloak:       https://auth.DOMAIN

 INFO  Context Configuration

  To connect the Fuzzball CLI to this deployment, run:

    fuzzball context create DEPLOYMENT_NAME \
      --api-url https://api.DOMAIN \
      --auth-url https://auth.DOMAIN

    fuzzball context use DEPLOYMENT_NAME

You can use Kubernetes through the kubectl command to monitor your cluster as it deploys and to manage the underlying pods and resources. First you need to configure your local kubectl installation to use your GCP deployment. Issue the following commands:

$ gcloud container clusters list --project=$PROJECT_ID --region=$REGION

This will give you your cluster name. Now you can use it to execute the following:

$ gcloud container clusters get-credentials <cluster-name> --project $PROJECT_ID --region $REGION

You may need to install the gke-gcloud-auth-plugin package and rerun the command above if it fails.

You now have enough information to proceed to the Initial Configuration section and log in and finish setting your cluster up.

At this point you can also run commands like the following to monitor your deployment and check the health of your cluster:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

$ kubectl get pods -n fuzzball

The fuzzball cluster gcp info or the deployment-info-wait.sh script referenced in the initial login section display a command to create the appropriate context to add and log into your cluster along with the credentials needed for the automatically provisioned users including the cluster admin user. Once you have done so, you can use the fuzzball command directly to monitor and manage many aspects of your deployment.

For instance, the list, status, info, and logs commands allow you to view information about your running deployment(s).

And the update, destroy, and cleanup commands allow you to manage your cluster directly.

Use the --help flag in the CLI to list these commands and see information about running each of them.

If there are any left-over resources from a previous deployment in your Azure subscription, they may interfere with your ability to deploy a fresh Fuzzball cluster. Please make sure that all resources from previous deployments have been fully and successfully deleted before initiating a new one. You can use the following command to forcibly remove old Fuzzball resources after destroying your Azure deployment if necessary:

$ fuzzball cluster azure cleanup

The Azure deployment uses the Azure CLI session for control-plane operations, plus your CIQ Depot credentials for pulling Fuzzball container images and Helm charts. Set up both before deploying.

$ az login

$ az account set --subscription "$SUBSCRIPTION_ID"

Capture your Depot credentials in environment variables (these are picked up automatically by the CLI):

$ export DEPOT_USER=<your depot username>

$ export DEPOT_ACCESS_TOKEN=<your depot access token>

By default the Azure deployment creates a private Azure Container Registry (ACR) in the deployment resource group and copies the Fuzzball images from Depot into it during deployment. If you would rather have AKS pull images straight from Depot at runtime, pass --use-depot. The ACR path is recommended for production deployments; --use-depot is convenient for short-lived dev clusters.

The simplest way to deploy is using interactive mode.

Include the --dry-run flag to get an idea of what will happen before you actually execute the command.

$ fuzzball cluster azure deploy

The CLI will prompt you for all required parameters including the following:

• Azure subscription (if you have multiple)
• Azure location (region)
• Resource group (defaults to fuzzball)
• Fuzzball version (defaults to the CLI version if omitted)
• Domain name
• Organization owner email
• DNS provider configuration

For non-interactive deployments, you can use the following. (See the Requirements section for information on setting these environment variables.)

$ fuzzball cluster azure deploy \
  --subscription "$SUBSCRIPTION_ID" \
  --location "$LOCATION" \
  --resource-group "$RESOURCE_GROUP" \
  --version "$VERSION" \
  --domain "$DOMAIN" \
  --keycloak-owner-email "admin@example.com" \
  --instance-types "Standard_D4s_v5,Standard_NC6s_v3" \
  --dns-provider azuredns \
  --dns-azuredns-subscription "$SUBSCRIPTION_ID" \
  --dns-azuredns-resource-group "$DNS_ZONE_RG" \
  --dns-azuredns-zone-name "$DOMAIN" \
  --non-interactive

For a full list of the options and arguments that can be specified during deployment, use the fuzzball cluster azure deploy --help command.

A successful deployment will create a fuzzball CLI context for the cluster in ${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/fuzzball.yaml and print out a basic summary of the cluster properties including commands to set up kubectl access to the underlying AKS cluster (see below). You can also obtain similar information later by running:

$ fuzzball cluster azure info --subscription $SUBSCRIPTION_ID --resource-group $RESOURCE_GROUP

If you did not pass --keycloak-password and --keycloak-default-user-password, the CLI generates them for you and prints them only once at the end of a successful deploy. Save them to a password manager before closing your terminal — they cannot be recovered later.

You can use Kubernetes through the kubectl command to monitor your cluster as it deploys and to manage the underlying pods and resources. First you need to configure your local kubectl installation to use your AKS deployment. Issue the following commands:

$ az aks list --resource-group $RESOURCE_GROUP --query "[].name" -o tsv

This will give you your AKS cluster name. Now you can use it to fetch credentials:

$ az aks get-credentials --resource-group $RESOURCE_GROUP --name <cluster-name>

If your deployment uses Azure AD admin group authentication (--azure-ad-admin-group-id), the first kubectl command may prompt you to complete an interactive Azure AD device login. Subsequent commands within the same session reuse the cached token.

You now have enough information to proceed to the Initial Configuration section and log in and finish setting your cluster up.

At this point you can also run commands like the following to monitor your deployment and check the health of your cluster:

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

$ kubectl get pods -n fuzzball

Once you have created and switched to the Fuzzball context for your cluster, you can use the fuzzball command directly to monitor and manage many aspects of your deployment. The Azure cloud subcommand provides the following operations:

• fuzzball cluster azure list — list every Fuzzball deployment in the subscription.
• fuzzball cluster azure status — show health and resource state for a specific deployment.
• fuzzball cluster azure info — print deployment details (domain, version, endpoints).
• fuzzball cluster azure logs — fetch operator and orchestrate logs from the AKS cluster.
• fuzzball cluster azure update — upgrade Fuzzball version or change configuration in place.
• fuzzball cluster azure destroy — tear down a deployment and delete its resource group.
• fuzzball cluster azure cleanup — sweep up orphaned resources from a failed or partial destroy.

Use the --help flag on any of these to see the full flag list.

Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

After fulfilling the prerequisites listed in the requirements and discovering your cluster’s domain using the domain discovery procedure, you are ready to deploy Fuzzball on CoreWeave.

The Fuzzball operator manages the deployment and lifecycle of Fuzzball on your Kubernetes cluster.

First, use your Depot credentials to authenticate with the Helm registry:

$ DEPOT_USER="your-depot-username"

$ ACCESS_KEY="your-depot-access-key"

Replace your-depot-username and your-depot-access-key with the credentials provided by CIQ.

$ helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Install the Fuzzball operator using the Helm chart from the depot:

$ VERSION="v4.0.1"

$ CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

$ STORAGE_CLASS="shared-vast"

$ helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system \
  --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=${STORAGE_CLASS}"

Check that the operator pod is running:

$ kubectl get pods -n fuzzball-system
NAME                                              READY   STATUS    RESTARTS   AGE
fuzzball-operator-controller-manager-xxxxx-xxxxx  2/2     Running   0          2m

Expected output should show the operator pod in Running state with 2/2 containers ready.

Create a FuzzballOrchestrate custom resource to deploy Fuzzball on CoreWeave. Here’s a complete example configuration:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-coreweave
  namespace: fuzzball-system
spec:
  # Image registry configuration
  image:
    username: <depot-username>
    password: <depot-password>
    exclusive: false

  # Database configuration
  database:
    create:
      enableDebugPod: false
      storage:
        class: shared-vast

  # Fuzzball version and cluster configuration
  fuzzball:
    version: v4.0.1
    cluster:
      name: fuzzball-coreweave

    # CoreWeave provisioner configuration
    orchestrator:
      provisioner:
        enabled: true
        coreweave:
          enabled: true
          storage:
            accessMode: ReadWriteMany
            class: shared-vast
            size: 100Gi  # Adjust based on workflow needs

    # Shared container image cache
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: shared-vast
        size: 10Gi  # Adjust based on caching needs

  # Ingress and networking configuration
  ingress:
    create:
      # Use your discovered CoreWeave domain
      domain: <YOUR_COREWEAVE_DOMAIN>
      proxy:
        type: LoadBalancer
        annotations:
          # Public LoadBalancer for internet access
          service.beta.kubernetes.io/coreweave-load-balancer-type: public
          # Wildcard DNS for all services
          service.beta.kubernetes.io/external-hostname: '*.<YOUR_COREWEAVE_DOMAIN>'

  # Keycloak identity management
  keycloak:
    create:
      createDatabase: true
      # Generate with: uuidgen
      realmId: <uuid-v4-realm-id>
      ownerEmail: <owner-email>
      # Change this password after first login!
      defaultUserPassword: <initial-user-password>

  # TLS certificate configuration
  tls:
    # cert-manager for certificate management
    certManager:
      create: {}
    # trust-manager for CA certificate distribution
    trustManager:
      create: {}
    # Let's Encrypt certificate issuer
    ingressIssuer:
      create:
        letsEncrypt:
          email: <letsencrypt-email>
          issuer: letsencrypt-prod

Replace the placeholder values with your configuration. Use the domain discovered in the domain discovery procedure.

For detailed explanations of these configuration options and additional settings, see the CRD reference material.

If your cluster already has cert-manager installed, you can configure Fuzzball to use it instead of deploying a new instance. See Deploying with External cert-manager for details.

Save the configuration to a file and apply it with kubectl:

$ kubectl apply -f fuzzball-orchestrate-coreweave.yaml

Watch the deployment status:

$ kubectl get fuzzballorchestrate -A -w

Wait until the status shows Ready. This typically takes several minutes for a full deployment.

Check the deployed resources:

$ kubectl get fuzzballorchestrate -A

$ kubectl get pvc -n fuzzball

$ kubectl get pods -n fuzzball

For detailed deployment status and troubleshooting, check the operator logs:

$ kubectl logs -l control-plane=fuzzball-operator-controller-manager -n fuzzball-system -f

The deployment steps above configure dynamic provisioning, where Fuzzball creates and destroys CoreWeave nodes on-demand. If you prefer to manage node pools yourself for predictable capacity or faster startup times, see Static Node Pool Provisioning for an alternative deployment approach.