Deployment

The deployment takes place across two groups of nodes: an odd number of Server node(s) that run Fuzzball Orchestrate, and one or more Compute node(s) that run Fuzzball Substrate. This guide assumes one Server node and one Compute node; where a production-ready, high availability (HA) cluster differs, special instructions are called out inline.

This guide is designed to be completed primarily through copy-and-paste instructions. Some values cannot be anticipated and are included as environment variables that you are expected to set. Be aware that you will switch from one node to another at various points, so variables need to be assigned again in the new environment.

The following variables are determined by your specific network configuration:

• IP1: The first IP address in your metallb pool (e.g. 10.0.0.99). See the appendix on installing a minimal K8s environment for details.
• NFS_SERVER_IP: The internal IP address of your NFS server (e.g. 10.0.0.3). If you serve NFS from your Server node, this is simply the internal IP address of the Server node. See the appendix on installing a minimal K8s for details.
• REALM_ID: A UUID used to configure the Keycloak instance. Create it with the uuidgen command.
• PUBLIC_INTERFACE: The literal name of the public interface — for instance, enp1s0.
• INTERNAL_INTERFACE: The literal name of the internal interface — for instance, enp8s0.

The remaining variables relate to accessing CIQ Depot and downloading artifacts from the container registry:

• DEPOT_USER: A username to access CIQ Depot, issued by your CIQ sales/support team once you subscribe to Fuzzball.
• ACCESS_KEY: A token you receive from the CIQ sales/support team after subscribing to Fuzzball.
• VERSION: The version of the Fuzzball Operator used for the deployment. You can find version information in Depot.
• CHART: The URL of the Helm Chart used to deploy the Fuzzball Operator.
• IMAGE: The URL of the image for the Fuzzball Operator.
• OWNER_EMAIL: The email address of the Fuzzball organization owner used for login.
• STORAGE_CLASS: The StorageClass used internally by K8s to support the Fuzzball cluster.

The Server node will run Fuzzball Orchestrate, which is deployed as a K8s operator. We must therefore begin by obtaining and installing the operator.

First, we need an access key to obtain artifacts from Depot. You will receive this key from the CIQ sales/support team upon subscribing to Fuzzball. Save it somewhere safe (with appropriate permission so that only you can access it).

Now you can use the Depot access key to login to the registry using Helm.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Once you have logged into the depot registry, you can install the Fuzzball operator. You can replace the VERSION below if you need a different version of Fuzzball to be installed. (You can find different Fuzzball versions by signing into Portal and looking at Fuzzball under “my products”.)

# VERSION="v4.1.0"

# CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

# STORAGE_CLASS="longhorn" # adjust this if you are using a different StorageClass

# helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=${STORAGE_CLASS}"

After executing this command, you can check that the operator is installed properly with the following commands:

# kubectl get pvc -n fuzzball-system
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
fuzzball-operator-controller-manager   Bound    pvc-dfc2f2c3-8df0-4ec8-8503-2d927b13402e   20Gi       RWO            longhorn       <unset>                 11s

This will show that the fuzzball-operator-controller-manager is Bound to a volume.

# kubectl get crd fuzzballorchestrate.deployment.ciq.com
NAME                                     CREATED AT
fuzzballorchestrate.deployment.ciq.com   2025-12-11T21:42:25Z

This command shows that a new custom resource has been defined.

# kubectl get pod -n fuzzball-system
NAME                                                    READY   STATUS    RESTARTS   AGE
fuzzball-operator-controller-manager-6dd9bd7f56-hnb9b   2/2     Running   0          81s

This will show that 2/2 pods instantiating the fuzzball-operator-controller-manager are running. (It might take a few minutes to complete, so don’t worry if there are initially 0/3 ready.)

You can also inspect the configuration of the operator with the following command:

# helm get values -n fuzzball-system fuzzball-operator
USER-SUPPLIED VALUES:
image:
  repository: depot.ciq.com/fuzzball/fuzzball-images/fuzzball-operator
  tag: v4.1.0
imagePullSecrets:
  inline:
    password: <YOUR_PASSWORD_HERE>
    registry: depot.ciq.com
    username: dgodlove@ciq.com
  name: repository-ciq-com
storageClassName: longhorn

You can rerun the helm upgrade command if you want to update your version of Fuzzball or to change selected settings.

Most Kubernetes resources created by the chart derive their name from a computed prefix. The prefix is determined as follows:

1. If fullnameOverride is set, the prefix is exactly that value.
2. Otherwise, the effective name is nameOverride if set, or the chart name (fuzzball-operator).
3. If the release name already contains the effective name, the prefix is the release name alone. Otherwise, the prefix is <release-name>-<effective-name>.

Two optional Helm values control this behavior:

• nameOverride: Substitutes an alternative chart name in step 2. For example, --set nameOverride=fbop with release name fuzzball-operator produces the prefix fuzzball-operator-fbop (release does not contain fbop), so the Deployment becomes fuzzball-operator-fbop-controller-manager.

• fullnameOverride: Skips the release name entirely and sets the prefix directly. For example, --set fullnameOverride=fbop always produces fbop-controller-manager regardless of the release name.

The table below summarizes how these values combine with the release name to form resource prefixes:

Each affected resource appends a stable suffix to this prefix:

The following resources are not affected by nameOverride or fullnameOverride:

• CRDs (fuzzballorchestrate.deployment.ciq.com, fuzzballfederate.deployment.ciq.com) — Kubernetes enforces <plural>.<group> naming and these cannot be renamed.
• ClusterRoles fuzzball-metrics-reader and fuzzball-proxy-role — fixed names generated by the kubebuilder/kube-rbac-proxy scaffolding.
• imagePullSecrets Secret — controlled by imagePullSecrets.name, not the fullname helper.

Now that the K8s Fuzzball Operator is installed on the Server node, we are ready to use it to deploy Fuzzball! The first step is to generate a suitable fuzzball.yaml file to configure the new deployment. Here is an example.

At this time, there are variables that appear in the fuzzball.yaml file that can only be set to the given value. Do not assume that any variable in the YAML file can be used to configure the Fuzzball Orchestrate CRD.

The following code block describes several commands to set the values of environment variables and one (extremely long) cat command that will use these values to create the appropriate Custom Resource Definition (CRD) file.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# IP1="" # populate with the first IP address in your metallb pool (e.g. 10.0.0.99)

# NFS_SERVER_IP="" # populate with the IP address of your NFS server

# REALM_ID=$(uuidgen --random | tr 'A-Z' 'a-z')

# OWNER_EMAIL="admin@ciq.com" # populate with the email of the owner of the Fuzzball organization

# STORAGE_CLASS="longhorn" # change this to local-path if you installed Local Storage Path

# cat > fuzzball.yaml <<EOF
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  labels:
    app.kubernetes.io/name: fuzzball-orchestrate
    app.kubernetes.io/part-of: fuzzball
  name: fuzzball-orchestrate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  ingress:
    create:
      domain: "${IP1}.nip.io"
      proxy:
        type: LoadBalancer
        annotations:
          metallb.io/allow-shared-ip: ingress-and-fuzzball
          metallb.io/loadBalancerIPs: ${IP1}
  database:
    create:
      enableDebugPod: true
      storage:
        class: ${STORAGE_CLASS}
  keycloak:
    create:
      ingress:
        hostname: auth.${IP1}.nip.io
      realmName: Fuzzball
      realmId: ${REALM_ID}
      username: keycloak
      ownerEmail: "${OWNER_EMAIL}"
      createDatabase: true
  tls:
    certManager:
      create: {} # Empty object = deploy cert-manager with defaults
    trustManager:
      create: {} # Empty object = deploy trust-manager with defaults
  fuzzball:
    substrate:
      nfs:
        destination: "/fuzzball/shared"
        server: ${NFS_SERVER_IP}
        path: "/srv/fuzzball/shared"
    audit:
      storage:
        class: ${STORAGE_CLASS}
    storage:  # example of how to specify resources for fuzzball services
      resources:
        requests:
          cpu: 250m
          memory: 256Mi
        limits:
          cpu: 1000m
          memory: 1Gi
    workflow:
      callbackService:
        type: LoadBalancer
        annotations:
          metallb.io/allow-shared-ip: ingress-and-fuzzball
          metallb.io/loadBalancerIPs: ${IP1}
      autoscaling: # example for auto-scaling workflow service
        enabled: true
        minReplicas: 1
        maxReplicas: 3
        targetCPUUtilization: 80
        targetMemoryUtilization: 80
    substrateBridge:
      log:
        storage:
          class: ${STORAGE_CLASS}
      dns:
        externalService:
          type: NodePort
    orchestrator:
      provisioner:
        enabled: false
      autoscaling:
        enabled: false
        minReplicas: 1
        maxReplicas: 3
        targetCPUUtilization: 80
        targetMemoryUtilization: 80
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
    jetstream:
      replicas: 3
      externalService:
        type: NodePort
      storage:
        class: ${STORAGE_CLASS}
        size: 10Gi
EOF

If your cluster already has cert-manager installed, you can configure Fuzzball to use it instead of deploying a new instance. See Deploying with External cert-manager for details.

Because of the environment variables you’ve set, this command will automatically populate the field spec.image.password with a CIQ Depot access key and spec.keycloak.create.realmId with a v4 UUID. It will also set the domain and hostname with the appropriate IP addresses for your setup. There are a few more customizations you may want to make before proceeding.

• It is recommended that you update spec.ingress.create.domain with a site-specific domain.
• Optionally update spec.keycloak.create.password with a unique password for the Keycloak admin user.
• Optionally define spec.keycloak.create.ingress.hostname with a site-specific domain name.

Once you have the fuzzball.yaml file created and customized to your liking, you are ready to use it to deploy Fuzzball Orchestrate.

With the Fuzzball K8s operator installed and the fuzzball.yaml configuration file in place, you are ready to deploy Fuzzball Orchestrate to the Server node.

You can do so with the following command:

# kubectl apply -f fuzzball.yaml

This command proceeds asynchronously. So it will look like it is completed even though it will still require some time for the K8s Fuzzball deployment to finish. This usually takes several minutes.

You can monitor the deployment as it occurs with the following command:

# kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1
[...snip]
Diagnostics:
  ciq:core/v1:Identity (identity):
    Waiting for keycloak to be ready: SUCCESS

  pulumi:pulumi:Stack (fuzzball-orchestrate-fuzzball-orchestrate):
    Using fuzzballCRD image username and password

Resources:
    + 9 created
    ~ 7 updated
    16 changes. 235 unchanged

Duration: 16s

2025-02-04T22:49:08Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballOrchestrate","name":"fuzzball-orchestrate","uid":"786ef498-cc8b-4d31-9729-8e0c2a6d9dcb","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"44728"}, "reason": "DeploymentSucceeded"}
2025-02-04T22:49:08Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballorchestrate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballOrchestrate", "FuzzballOrchestrate": {"name":"fuzzball-orchestrate"}, "namespace": "", "name": "fuzzball-orchestrate", "reconcileID": "ebe8f5ac-8ef7-449a-81bc-080c455aae86"}

After deployment, you can inspect the configuration that was used with the following command:

# kubectl get fuzzballorchestrate fuzzball-orchestrate -o yaml

You can edit the current configuration in-place. These changes will be immediately applied and updated by the Fuzzball operator.

# kubectl edit fuzzballorchestrate fuzzball-orchestrate

Once the deployment is complete, you can confirm that Fuzzball Orchestrate is installed correctly by inspecting that expected objects have been created. (Note that you may have more or fewer replicas of some pods.)

Look at the newly started pods:

# kubectl get pods -n fuzzball
NAME                                         READY   STATUS      RESTARTS   AGE
fuzzball-admin-0                             1/1     Running     0          8m49s
fuzzball-admin-ui-548f599b68-9j9vc           1/1     Running     0          8m29s
fuzzball-agent-848db9f89c-lmdmq              1/1     Running     0          7m19s
fuzzball-audit-8dd47456-wf2qq                1/1     Running     0          8m39s
fuzzball-auth-spicedb-77d7dd9957-jz2lr       1/1     Running     0          8m47s
fuzzball-cluster-admin-599c87b794-4txmh      1/1     Running     0          7m54s
fuzzball-jetstream-0                         1/1     Running     0          7m20s
fuzzball-jetstream-1                         1/1     Running     0          7m20s
fuzzball-jetstream-2                         1/1     Running     0          7m20s
fuzzball-load-default-catalog-sv2jg          0/1     Completed   0          3m35s
fuzzball-openapi-65c78cc7cb-wdmmw            1/1     Running     0          8m41s
fuzzball-orchestrator-779cd894d6-h9x9b       1/1     Running     0          5m17s
fuzzball-storage-6597d744b4-zz5n5            1/1     Running     0          5m15s
fuzzball-substrate-bridge-6cc6f74859-gtc2h   1/1     Running     0          6m9s
fuzzball-sync-0                              1/1     Running     0          4m20s
fuzzball-ui-7d664769bb-tkqhl                 1/1     Running     0          8m28s
fuzzball-workflow-7d8cd4cf6-8rt5t            1/1     Running     0          4m48s

Check for any pods that are not in the running state:

# kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded
No resources found

Check that ingress is configured and ready.

# kubectl get ingress -A
NAMESPACE           NAME                        CLASS    HOSTS                                ADDRESS      PORTS     AGE
fuzzball            fuzzball-admin-ui           kong     ui-admin.10.1.96.99.nip.io           10.1.96.99   80, 443   8m16s
fuzzball            fuzzball-config             kong     api.10.1.96.99.nip.io                10.1.96.99   80, 443   9m35s
fuzzball            fuzzball-openapi            kong     api.10.1.96.99.nip.io                10.1.96.99   80, 443   9m11s
fuzzball            fuzzball-substrate-bridge   kong     substrate-bridge.10.1.96.99.nip.io   10.1.96.99   80, 443   6m14s
fuzzball            fuzzball-ui                 kong     ui.10.1.96.99.nip.io                 10.1.96.99   80, 443   8m10s
fuzzball-identity   identity-keycloakx          <none>   auth.10.1.96.99.nip.io               10.1.96.99   80, 443   10m

Confirm that all persistent volume claims (PVCs) are bound.

# kubectl get pvc -A
NAMESPACE         NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
fuzzball-database data-database-postgresql-0                Bound    pvc-52d62829-32b5-41e7-90c0-11d9ddfcfccb   8Gi        RWO            longhorn       <unset>                 12m
fuzzball-system   fuzzball-operator-controller-manager      Bound    pvc-dfc2f2c3-8df0-4ec8-8503-2d927b13402e   20Gi       RWO            longhorn       <unset>                 22m
fuzzball          audit-cold-storage                        Bound    pvc-7bcd7abc-0d11-419b-b453-3a6bddccbb67   1Gi        RWO            longhorn       <unset>                 10m
fuzzball          fuzzball-substrate-logs                   Bound    pvc-e282b15b-6ab2-4b64-8a39-d6c58a382835   5Gi        RWO            longhorn       <unset>                 7m39s
fuzzball          jetstream-data-fuzzball-jetstream-0       Bound    pvc-ad6481d4-d469-4343-a8a3-9b2117e53ab4   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          jetstream-data-fuzzball-jetstream-1       Bound    pvc-4cb91882-d556-4ffc-a931-76ed45f8761a   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          jetstream-data-fuzzball-jetstream-2       Bound    pvc-a161913d-6629-4b36-b109-a83943041fba   10Gi       RWO            longhorn       <unset>                 8m47s
fuzzball          substrate-shared-config-fuzzball-sync-0   Bound    pvc-2953abd9-5e29-45d5-b731-8dcf935a76d4   5Gi        RWX            nfs-client     <unset>                 5m47s

Now that you have a running instance of Fuzzball Orchestrate it is time to connect Compute nodes to it. Compute nodes run the Fuzzball Substrate container platform to create containerized environments for Fuzzball jobs. To do this, Compute nodes require some additional configuration (firewall, NFS, etc.).

You can start by enrolling the Compute node in CIQ Depot and installing required packages.

# DEPOT_USER="" # populate with your username for CIQ Depot

# ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

# VERSION="4.1.0" # matches orchestrate version without the leading v

# dnf install -y https://depot.ciq.com/public/files/depot-client/depot/depot.x86_64.rpm

# depot register --user "${DEPOT_USER}" --token "${ACCESS_KEY}"

# depot enable fuzzball

# dnf install -y fuzzball-substrate fuzzball-substrate-orchestrate-${VERSION}-1 fuzzball-cli-${VERSION}-1

If your compute node has a GPU, be sure to install the following package as well.

# dnf install -y fuzzball-substrate-nvidia-gpu-plugin # optional

You should configure firewalld to prevent interference with intra-cluster communication. Either disable it or configure the internal interface to use the trusted zone.

# INTERNAL_INTERFACE="" # populate this with the value for your COMPUTE NODE environment (e.g. enp8s0)

# firewall-cmd --permanent --zone=trusted --add-interface=${INTERNAL_INTERFACE}

# firewall-cmd --reload

Now you can install and configure NFS on the Compute node so that Fuzzball can share configuration and logging files with the Server node.

# dnf install -y nfs-utils

You can place the following contents in the file /etc/systemd/system/fuzzball-shared.mount to create a mount unit for use with systemd. This will ensure that NFS starts properly again after a reboot.

# NFS_SERVER_IP="" # populate with the internal IP address of the Server node (e.g. 10.0.0.3)

# cat >/etc/systemd/system/fuzzball-shared.mount<<EOF
[Unit]
Description=Mount Fuzzball NFS Share
After=network-online.target remote-fs-pre.target
Wants=network-online.target
Before=remote-fs.target

[Mount]
What=${NFS_SERVER_IP}:/srv/fuzzball/shared
Where=/fuzzball/shared
Type=nfs
Options=_netdev,hard,retry=5,timeo=600,retrans=3
TimeoutSec=300

[Install]
WantedBy=remote-fs.target
EOF

# mkdir -p /etc/systemd/system/fuzzball-substrate.service.d

# cat >/etc/systemd/system/fuzzball-substrate.service.d/10-require-shared.conf<<EOF
[Unit]
Requires=fuzzball-shared.mount
After=fuzzball-shared.mount
EOF

Now create the shared mount point on the Compute node and load/enable/start the newly created mount unit.

# mkdir -p /fuzzball/shared

# systemctl daemon-reload

# systemctl enable --now fuzzball-shared.mount

# ls /fuzzball/shared/
substrate

The final command demonstrates that a configuration directory has already been placed on NFS by Fuzzball Orchestrate.

If the NFS mount fails to load or the /fuzzball/shared directory doesn’t allow you to share between the nodes, stop and troubleshoot your NFS configuration before continuing with the instructions. If the Compute node NFS configuration looks good, check the NFS configuration on the Server node.

Now that your compute node is fully configured, you are ready to configure Fuzzball Substrate to act as a Compute node agent and enable/start it.

You begin by adding the following variables to /etc/default/fuzzball-substrate.env. This lets Substrate know where it should look to find its configuration files. /fuzzball/shared in our example is the mount point of the file system shared by Orchestrate and the Substrate nodes.

# cat >/etc/default/fuzzball-substrate.env<<EOF
FB_POLL_CONFIG=/fuzzball/shared/substrate/substrate-config.yaml

# aids crash recovery
FB_MOUNTNS=true
EOF

Now you can enable/start the service like so.

# systemctl enable --now fuzzball-substrate.service

At this point you have installed Fuzzball Orchestrate on a server node and Fuzzball Substrate on a compute node! Next, follow the initial configuration guide to add your compute node to the cluster, configure Fuzzball Orchestrate to use storage, and start adding users.

The fuzzball cluster docker-compose deploy command brings up a single-host Fuzzball stack — nginx (TLS termination and subdomain routing), PostgreSQL, Keycloak, Fuzzball Orchestrate, and a Fuzzball Substrate node — suited to local development and testing. The deployment is managed entirely through the fuzzball cluster docker-compose subcommands.

By default the stack runs one substrate node co-located with Orchestrate, but this is not a hard limit. You can run additional nodes on the same host with --substrate-nodes, or attach a substrate node running on a separate host (see Adding an external substrate node).

$ fuzzball cluster docker-compose deploy

The deploy command renders the stack’s configuration files to disk and, by default, leaves the stack stopped. Pass --up to start the containers immediately after deploying, or start them later with up.

$ fuzzball cluster docker-compose deploy --up

Add --dry-run to preview the generated docker-compose.yaml and .env without writing anything to disk or starting containers.

The stack runs on Docker or, for local testing, Podman. The CLI auto-detects the container runtime, preferring Docker and falling back to Podman; pass --runtime podman or set FUZZBALL_CONTAINER_RUNTIME=podman to force Podman.

If the container images require authentication, deploy prompts for a registry user and password/token (your CIQ Depot credentials); leave the user blank to skip the registry login. You can also supply them non-interactively with --registry-user and --registry-password, and override the registry itself with --registry. Most other settings — organization name, database and Keycloak credentials, owner email — are auto-generated or defaulted and can be overridden with their corresponding flags (see deploy --help); generated secrets are recoverable later with info --secrets.

By default the stack runs a single substrate node co-located with Orchestrate. For testing or demonstrating multi-node workflows on a single host, pass --substrate-nodes N to run N substrate nodes (default 1, max 255):

$ fuzzball cluster docker-compose deploy --substrate-nodes 2

Running multiple substrate nodes on a single host can over-provision it. This is intended for development and demonstration of multi-node workflows, not for production capacity.

The stack can also accept a Fuzzball Substrate node running on a separate host, which is useful when you want compute capacity beyond the Orchestrate host. The stack exposes NATS on port 4222 so that remote substrate nodes can reach Orchestrate.

With the stack running, generate the configuration the external node needs and write it to an output directory. Pass the Orchestrate host’s IP (as reachable from the substrate host) with --orchestrate-ip; optionally add the substrate host’s own IP to the certificate SANs with --substrate-ip:

$ fuzzball cluster docker-compose generate-substrate-config ./substrate-config \
  --orchestrate-ip 10.0.0.5

This extracts Orchestrate’s CA certificates, NATS NKey, and connection settings, writing five files into the output directory: fuzzball-substrate.yaml, orchestrate.yaml, ca.crt, nginx-ca.crt, and a hosts entry. The output directory must be empty; pass --force to overwrite existing files.

The deployment must be running before you generate the config. Start it first with fuzzball cluster docker-compose up.

Then, on the substrate host, copy the generated files into your fuzzball-substrate runtime:

Finally, append the generated hosts entry to /etc/hosts so that api.<domain> and nats.<domain> resolve to the Orchestrate host:

# cat hosts | sudo tee -a /etc/hosts

Cloud deployments have an equivalent command — fuzzball cluster aws|azure|gcp|oci generate-substrate-config — for attaching external substrate nodes (for example, over a VPN) to a Kubernetes-based cluster. See --help for details.

The standard deploy brings up a CPU-only stack. On a host with GPUs, pass --gpu to use the GPU-enabled substrate image, which bundles the GPU device plugins:

$ fuzzball cluster docker-compose deploy --gpu

For more detail on GPU-enabled Docker Compose stacks, see the GPU documentation.

By default the deployment is named default. To run more than one stack on the same host, or to give a deployment a meaningful name, pass --deployment-name:

$ fuzzball cluster docker-compose deploy --deployment-name my-dev-stack

The deployment name is used as the Docker Compose project name and as the directory name under your Fuzzball CLI config home. All of the other docker-compose subcommands accept the same --deployment-name flag to select which stack they act on.

Running more than one stack on the same host at the same time is not recommended — separate stacks compete for the same host ports and resources.

The stack routes traffic to its services through subdomains of a base domain (defaulting to fb.local; override it with --domain). The ui., api., keycloak., endpoints., and nats. subdomains — along with any *.endpoints. hostname — must resolve to the host running the stack before you can reach the UI, API, or log in.

The simplest option for a single host is to add a 127.0.0.1 entry to /etc/hosts. Pass --update-etc-hosts to deploy to have the CLI add it for you (this uses sudo):

$ fuzzball cluster docker-compose deploy --update-etc-hosts

If you omit the flag, the deploy command prints the exact /etc/hosts line to add yourself, for example:

127.0.0.1 ui.fb.local api.fb.local keycloak.fb.local endpoints.fb.local nats.fb.local

If you run a local resolver such as dnsmasq, a single wildcard entry covers every subdomain — including the nested *.endpoints. hostnames — without editing /etc/hosts. For the default domain:

address=/fb.local/127.0.0.1

The deploy command generates a self-signed root CA (rootCA.pem) and a wildcard leaf certificate under the deployment’s certs/ directory. Because the CA is self-signed, browsers and CLI clients will not trust it by default, and connections to the UI and API will report a certificate warning.

To trust the certificate, either point the fuzzball CLI at the deployment’s certs/ directory or import the root CA into your operating system or browser trust store. fuzzball cluster docker-compose info prints the certs/ path and the exact command to export, for example:

$ export SSL_CERT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/certs"

The leaf certificate is valid for one year. Renew it in place without redeploying using fuzzball cluster docker-compose update --tls-renew — see Updating Fuzzball.

Once a stack has been deployed, you can start and stop it without re-rendering its files:

$ fuzzball cluster docker-compose up

$ fuzzball cluster docker-compose down

down stops and removes the stack’s containers but preserves its data volumes. To additionally remove the volumes — which destroys all data in the deployment — pass down --volumes. This prompts for confirmation; add -y to skip the prompt in scripts.

The full set of fuzzball cluster docker-compose subcommands lets you manage deployments once they exist:

Run any subcommand with --help to see its full set of options.

When something goes wrong, see Troubleshooting below for how to inspect the generated stack files and drive docker compose directly.

When a Docker Compose deployment misbehaves, it’s often useful to inspect the generated stack files or to drive docker compose directly. The CLI writes each deployment into its own directory under your Fuzzball CLI config home and runs docker compose against it on your behalf, so you can do the same thing manually.

Each deployment is stored in a subdirectory named after the deployment, under the docker-compose folder in your Fuzzball CLI config home:

${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/<deployment-name>/

If XDG_CONFIG_HOME is set in your environment, the directory is rooted there ($XDG_CONFIG_HOME/fuzzball/docker-compose/...); otherwise it falls back to $HOME/.config/fuzzball/docker-compose/.... Docker compose deployments always use this per-user config directory location.

The directory holds everything the stack needs:

• docker-compose.yaml — the generated Compose file describing every service in the stack.
• .env — the environment file with values such as IMAGE_TAG (the deployed Fuzzball version) that the Compose file substitutes in.
• certs/ — TLS material generated for the deployment.

The CLI invokes docker compose with the deployment name as the Compose project name (-p) and with the deployment directory as the working directory. To reproduce that yourself, change into the deployment directory and pass the same -p <deployment-name> flag. For the default deployment:

$ cd "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/"

List the containers and their state (equivalent to what fuzzball cluster docker-compose status inspects):

$ docker compose -p default ps

Tail the logs of a single service — for example, Orchestrate:

$ docker compose -p default logs -f fuzzball-orchestrate

Inspect the fully-rendered Compose configuration with .env values substituted in, which is helpful for confirming the image tag and other resolved settings:

$ docker compose -p default config

Always pass -p <deployment-name> so your manual docker compose commands act on the same project the CLI manages. For any deployment created with --deployment-name <name>, substitute that name for default in both the directory path and the -p flag above.