Docker Compose Stack

The fuzzball cluster docker-compose deploy command brings up a single-host Fuzzball stack — nginx (TLS termination and subdomain routing), PostgreSQL, Keycloak, Fuzzball Orchestrate, and one or more Fuzzball Substrate nodes — suited to local development, evaluation, and testing. The deployment is managed entirely through the fuzzball cluster docker-compose subcommands.

This page is the platform-agnostic reference for the Docker Compose stack. For hardware-specific walkthroughs that link back here, see Install Fuzzball on an NVIDIA DGX Spark and Install Fuzzball on an AMD Ryzen AI Max (Strix Halo).

The compose stack brings up five services on the host:

The substrate runs workflow jobs as rootless OCI containers directly on the host — no Kubernetes, no cloud, no external infrastructure. On a host with a GPU, the --gpu flag swaps the substrate to the GPU-enabled image variant, which bundles the device plugins that pass the GPU through to workflow containers.

The installation script installs the Fuzzball CLI and deploys the stack. It needs Depot credentials, which you can obtain from portal.ciq.com. Export them to skip the prompts; otherwise the script prompts you during installation.

$ export DEPOT_USER=<YOUR_USER> DEPOT_ACCESS_KEY=<YOUR_ACCESS_KEY>

Run it from an interactive terminal on the host:

$ curl -fsSL https://depot.ciq.com/dlv2/install-fb/docker-stack.sh | bash

Or download it first and run it directly:

$ curl -fsSLO https://depot.ciq.com/dlv2/install-fb/docker-stack.sh
$ bash docker-stack.sh

The script targets Ubuntu. If Docker Engine is not installed, it offers to install it from Docker’s official apt repository and adds your user to the docker group. Because a new group membership only takes effect in a fresh login session, the script bridges the current run automatically; when it finishes, run newgrp docker (or log out and back in) so docker and fuzzball work in your normal shell.

When prompted for GPU support, answer y on a host with a supported GPU to select the GPU substrate image; answer n for a CPU-only stack.

The script then hands off to fuzzball cluster docker-compose deploy with --update-etc-hosts and --up, so once it finishes you can continue at Log in with the CLI.

Re-running the script on a host that already has Fuzzball is the supported upgrade path: choose a newer CLI version at the prompt and it offers to upgrade the running stack to match. See Upgrading.

Install the Fuzzball CLI and deploy the stack by hand: download and install the CLI, then run the deploy command.

The Fuzzball images are pulled from depot.ciq.com. If your Docker daemon isn’t already authenticated to it, log in first with the Depot credentials from the CIQ sales/support team:

$ docker login depot.ciq.com

Install the fuzzball command-line tool, following the CLI installation guide — use the .deb package matching your host architecture (amd64 or arm64). Once installed, confirm the binary runs:

$ fuzzball --version

The deploy command renders the stack’s configuration files to disk and, by default, leaves the stack stopped:

$ fuzzball cluster docker-compose deploy

Pass --up to start the containers immediately after deploying, or start them later with up:

$ fuzzball cluster docker-compose deploy --up

Add --dry-run to preview the generated docker-compose.yaml and .env without writing anything to disk or starting containers.

If the container images require authentication, deploy prompts for a registry user and password/token (your CIQ Depot credentials); leave the user blank to skip the registry login. You can also supply them non-interactively with --registry-user and --registry-password, and override the registry itself with --registry. Most other settings — organization name, database and Keycloak credentials, owner email — are auto-generated or defaulted and can be overridden with their corresponding flags (see deploy --help); generated secrets are recoverable later with info --secrets.

By default the stack runs a single substrate node co-located with Orchestrate. For testing or demonstrating multi-node workflows on a single host, pass --substrate-nodes N to run N substrate nodes (default 1, max 255):

$ fuzzball cluster docker-compose deploy --substrate-nodes 2

Running multiple substrate nodes on a single host can over-provision it. This is intended for development and demonstration of multi-node workflows, not for production capacity.

The stack can also accept a Fuzzball Substrate node running on a separate host, which is useful when you want compute capacity beyond the Orchestrate host. The stack exposes NATS on port 4222 so that remote substrate nodes can reach Orchestrate.

With the stack running, generate the configuration the external node needs and write it to an output directory. Pass the Orchestrate host’s IP (as reachable from the substrate host) with --orchestrate-ip; optionally add the substrate host’s own IP to the certificate SANs with --substrate-ip:

$ fuzzball cluster docker-compose generate-substrate-config ./substrate-config --orchestrate-ip 10.0.0.5

This extracts Orchestrate’s CA certificates, NATS NKey, and connection settings, writing five files into the output directory: fuzzball-substrate.yml, orchestrate.yaml, ca.crt, nginx-ca.crt, and a hosts entry. The output directory must be empty; pass --force to overwrite existing files.

The deployment must be running before you generate the config. Start it first with fuzzball cluster docker-compose up.

Then, on the substrate host, copy the generated files into your fuzzball-substrate runtime:

Finally, append the generated hosts entry to /etc/hosts so that api.<domain> and nats.<domain> resolve to the Orchestrate host:

# cat hosts | sudo tee -a /etc/hosts

Cloud deployments have an equivalent command — fuzzball cluster aws|azure|gcp|oci generate-substrate-config — for attaching external substrate nodes (for example, over a VPN) to a Kubernetes-based cluster. See --help for details.

The deployment’s default provisioner serves volumes from /mnt/fuzzball-sharedfs, which on the Orchestrate host is a container volume shared only between the compose-managed substrate containers. A substrate node on another machine cannot reach it, no matter how its own paths are arranged.

That provisioner is scoped to the docker segment, which the compose-managed containers join and a node added by hand does not, so it is not offered to your external node. The external node therefore has no storage until you give it some. Choose one of:

• Give both hosts the same shared filesystem. Mount it on every node and add it as a provisioner with no segment, so any node can use it. This is the option that keeps volumes usable from anywhere in the cluster.

• Add a provisioner for the external host’s own disk, if the hosts genuinely have separate storage:

  $ fuzzball volume provisioner add

  Set local: true under driver: in the definition, with a path the external node can serve. Fuzzball then places the stages using a volume on the node holding it. See When the path is not shared for the restrictions this imposes – notably that a multinode job cannot mount a node-local volume at all, whether the volume is persistent or ephemeral.

A deployment created before the default provisioner was scoped to the docker segment does not gain that scoping by upgrading, so the problem above is still reachable on it. See Upgrades keep the original default provisioner.

deploy and update --sync-files rewrite docker-compose.yaml from the CLI’s embedded copy, so edits made directly to that file are lost the next time either runs.

Put local changes in docker-compose.override.yaml in the deployment directory instead. Fuzzball never writes or regenerates that file, and applies it last, so its settings win over both the base file and any Fuzzball-managed overlay:

services:
  substrate-localnode:
    volumes:
      - /srv/fuzzball-shared:/mnt

This is the supported way to add a bind mount, pin an address, or change an image without those changes being reverted.

The standard deploy brings up a CPU-only stack. On a host with GPUs, pass --gpu to use the GPU-enabled substrate image, which bundles the GPU device plugins:

$ fuzzball cluster docker-compose deploy --gpu

For more detail on GPU-enabled Docker Compose stacks — including how NVIDIA and AMD ROCm devices are requested and scheduled — see the GPU documentation.

By default the deployment is named default. To run more than one stack on the same host, or to give a deployment a meaningful name, pass --deployment-name:

$ fuzzball cluster docker-compose deploy --deployment-name my-dev-stack

The deployment name is used as the Docker Compose project name and as the directory name under your Fuzzball CLI config home. All of the other docker-compose subcommands accept the same --deployment-name flag to select which stack they act on.

Running more than one stack on the same host at the same time is not recommended — separate stacks compete for the same host ports and resources.

The stack routes traffic to its services through subdomains of a base domain. The ui., api., keycloak., endpoints., and nats. subdomains must resolve to the host running the stack before you can reach the UI, API, or log in.

There is also a wildcard requirement. Workflow service endpoints declared with type: subdomain are served at a per-endpoint hostname of the form endpoint-<id>.endpoints.<domain>. A new hostname appears for every endpoint, so they cannot be listed ahead of time, and /etc/hosts has no way to express a wildcard. Endpoints declared with the default type: path are served under the single endpoints.<domain> name and are unaffected.

Which approach you need depends on the host:

deploy defaults --domain to <host-ip>.nip.io. The nip.io zone resolves every name beneath it to the IP address embedded in the name, wildcards included, so endpoint-<id>.endpoints.10.0.0.5.nip.io resolves to 10.0.0.5 with no configuration on the host and no sudo. Names resolve both on the host itself and from other machines on the same network.

$ fuzzball cluster docker-compose deploy --up

The domain is fixed when the deployment is created — it is written into the generated TLS certificate and .env, and update cannot change it. If the host’s IP address changes, delete the deployment and deploy again. Assign the host a static or reserved address to avoid this.

Use your own domain and add a single wildcard A record pointing at the host:

*.fb.example.internal.  IN  A  10.0.0.5

Then deploy against it:

$ fuzzball cluster docker-compose deploy --domain fb.example.internal

This is the most robust option: it covers every client on the network with no per-machine setup, and the domain stays stable if the host’s address changes.

When the host has neither DNS infrastructure nor internet access, the stack can serve the wildcard zone itself:

$ fuzzball cluster docker-compose deploy --domain fb.local --setup-dns

--wildcard-dns makes Fuzzball Orchestrate answer every name under the domain with 127.0.0.1, published on 127.0.0.1:5354. The zone is authoritative only — it never forwards queries — and the published port is bound to the loopback interface, so it cannot act as a resolver for anything else on the network.

--setup-dns implies --wildcard-dns and additionally configures the host to use that zone. It requires sudo and writes two things, both removed again by delete:

1. A split-DNS entry, so that only *.<domain> is sent to the stack and every other lookup continues to use the host’s normal upstream resolver:
   • macOS — /etc/resolver/<domain>
   • Linux with systemd-resolved — /etc/systemd/resolved.conf.d/fuzzball-<deployment>.conf
2. An /etc/hosts entry for the fixed subdomains.

Both are required. The split-DNS entry is what makes the per-endpoint wildcard resolve, but on macOS it is only visible to programs that resolve names through the system library — browsers and curl, but not the statically linked fuzzball CLI. The /etc/hosts entry covers the fixed names for every client, including the CLI. After configuring DNS, deploy confirms that the fixed names resolve and warns if they do not.

On macOS, dig and nslookup query name servers directly and never consult /etc/resolver, so they report failure for wildcard names that resolve correctly in a browser. Use dscacheutil instead:

$ dscacheutil -q host -a name ui.fb.local

On Linux hosts without systemd-resolved there is no per-domain routing to configure, so deploy prints instructions for setting up dnsmasq rather than attempting the change itself. The equivalent configuration is:

server=/fb.local/127.0.0.1#5354

--update-etc-hosts adds a 127.0.0.1 entry covering the fixed subdomains, without any of the above:

$ fuzzball cluster docker-compose deploy --update-etc-hosts

Omit the flag and deploy prints the line for you to add yourself:

127.0.0.1 ui.fb.local api.fb.local keycloak.fb.local endpoints.fb.local nats.fb.local

Because hosts files cannot express wildcards, this covers everything except subdomain-style workflow endpoints. It is unnecessary with the default domain, and unnecessary with --setup-dns, which writes the same entry itself.

Before the CLI can reach the cluster you must trust the deployment’s self-signed certificate (or connect with --insecure). fuzzball cluster docker-compose info prints the service URLs, the exact command to trust the certificate, and the ready-to-paste context commands:

$ fuzzball cluster docker-compose info
Deployment:
  Name:      default
  Directory: /home/user/.config/fuzzball/docker-compose/default

Service URLs:
  UI:        https://ui.fb.local
  API:       https://api.fb.local
  Keycloak:  https://keycloak.fb.local
  Endpoints: https://endpoints.fb.local
  JetStream: nats://nats.fb.local:4222

Configuration:
  ... (deployment .env values; passwords hidden unless you pass --secrets)

TLS:
  Certs: /home/user/.config/fuzzball/docker-compose/default/certs
  To trust the deployment's wildcard certificate:
    export SSL_CERT_DIR="/home/user/.config/fuzzball/docker-compose/default/certs"

Connect:
  fuzzball context create default https://api.fb.local https://keycloak.fb.local/realms/<realm-id> fuzzball-cli
  fuzzball context login

UI:
  Open https://ui.fb.local in your browser

Run the export SSL_CERT_DIR=... line and the two fuzzball context commands exactly as printed. context login launches the device-code flow in your browser against Keycloak; sign in as the admin user admin@fb.local plus an auto-generated password (reveal it again with fuzzball cluster docker-compose info --secrets).

The deploy command generates a self-signed root CA (rootCA.pem) and a wildcard leaf certificate under the deployment’s certs/ directory. Because the CA is self-signed, browsers and CLI clients will not trust it by default, and connections to the UI and API will report a certificate warning.

To trust the certificate, either point the fuzzball CLI at the deployment’s certs/ directory or import the root CA into your operating system or browser trust store. fuzzball cluster docker-compose info prints the certs/ path and the exact command to export, for example:

$ export SSL_CERT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/certs"

The leaf certificate is valid for one year. Renew it in place without redeploying using fuzzball cluster docker-compose update --tls-renew — see Updating Fuzzball.

Once a stack has been deployed, you can start and stop it without re-rendering its files:

$ fuzzball cluster docker-compose up

$ fuzzball cluster docker-compose down

down stops and removes the stack’s containers but preserves its data volumes. To additionally remove the volumes — which destroys all data in the deployment — pass down --volumes. This prompts for confirmation; add -y to skip the prompt in scripts.

The full set of fuzzball cluster docker-compose subcommands lets you manage deployments once they exist:

Run any subcommand with --help to see its full set of options.

To bump the stack to a newer Fuzzball image tag:

$ fuzzball cluster docker-compose update --upgrade

--upgrade moves the deployment to the version your current fuzzball CLI binary was built against; use --version <tag> to pick a specific one instead. The same update command rotates credentials (--reset-owner-password, --reset-keycloak-password, --reset-database-password) and renews the wildcard TLS leaf certificate (--tls-renew). See Updating Fuzzball and fuzzball cluster docker-compose update --help for the full set.

A compose deployment scopes its default storage provisioner to the docker segment, so it is offered only to the compose-managed substrate containers that can actually reach the shared Docker volume behind it. A node added to the cluster by hand never joins that segment and is therefore never offered storage it cannot see.

Upgrading does not apply this to an existing deployment. The scoped provisioner is created under a new name, default-<segment>, and a provisioner’s identity is derived from its name, so it is a new record rather than a rename – renaming would separate the original provisioner from every volume created on it. The upgrade therefore leaves the original unsegmented default in place and adds default-docker alongside it.

Nothing that worked before the upgrade stops working: default keeps its volumes and keeps serving the compose nodes. But because it carries no segment, it is still advertised to every node in the cluster, including hand-added ones, so a workflow that selects it and runs on such a node still fails to start with a missing-directory error.

A fresh deployment is the only way to get the scoping applied automatically. If reinstalling is not practical, retire the old provisioner by hand instead:

1. Create the volumes you want to keep on default-docker, and copy their data across.

2. Remove the original:

   $ fuzzball volume provisioner remove default

Until one of those happens, treat default as usable only from the compose-managed nodes. Orchestrate logs a warning at startup whenever it finds an unsegmented default provisioner alongside a segmented one.

Both provisioners serve the same directory, so a volume name that already exists there is refused on the other with an “already exists” error rather than quietly sharing a directory. Deleting a volume through one provisioner removes the data a matching record on the other points at; a fuzzball volume provisioner scan clears the stale record.

Stop the stack while preserving its persistent volumes (postgres-data, jetstream-data, substrate-shared-fs, etc.) — useful when you want to come back to your workflows later:

$ fuzzball cluster docker-compose down

To completely remove the deployment, including its generated files and persistent state:

$ fuzzball cluster docker-compose delete

When a Docker Compose deployment misbehaves, it’s often useful to inspect the generated stack files or to drive docker compose directly. The CLI writes each deployment into its own directory under your Fuzzball CLI config home and runs docker compose against it on your behalf, so you can do the same thing manually.

pull access denied for depot.ciq.com/...

Docker isn’t authenticated to the registry. Run docker login depot.ciq.com with your Depot credentials, then re-run the deploy.

Each deployment is stored in a subdirectory named after the deployment, under the docker-compose folder in your Fuzzball CLI config home:

${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/<deployment-name>/

If XDG_CONFIG_HOME is set in your environment, the directory is rooted there ($XDG_CONFIG_HOME/fuzzball/docker-compose/...); otherwise it falls back to $HOME/.config/fuzzball/docker-compose/.... Docker compose deployments always use this per-user config directory location.

The directory holds everything the stack needs:

• docker-compose.yaml — the generated Compose file describing every service in the stack.
• .env — the environment file with values such as IMAGE_TAG (the deployed Fuzzball version) that the Compose file substitutes in.
• certs/ — TLS material generated for the deployment.

The CLI invokes docker compose with the deployment name as the Compose project name (-p) and with the deployment directory as the working directory. To reproduce that yourself, change into the deployment directory and pass the same -p <deployment-name> flag. For the default deployment:

$ cd "${XDG_CONFIG_HOME:-$HOME/.config}/fuzzball/docker-compose/default/"

List the containers and their state (equivalent to what fuzzball cluster docker-compose status inspects):

$ docker compose -p default ps

Tail the logs of a single service — for example, Orchestrate:

$ docker compose -p default logs -f fuzzball-orchestrate

Inspect the fully-rendered Compose configuration with .env values substituted in, which is helpful for confirming the image tag and other resolved settings:

$ docker compose -p default config

Always pass -p <deployment-name> so your manual docker compose commands act on the same project the CLI manages. For any deployment created with --deployment-name <name>, substitute that name for default in both the directory path and the -p flag above.

If browser logins fail but fuzzball commands keep succeeding, suspect the Keycloak realm rather than the user’s password. The two paths authenticate differently: the CLI uses a device flow against the public fuzzball-cli client and talks to Keycloak directly, while the Web UI needs the confidential fuzzball-ui client and looks it up through the organization service. A realm or client that has gone missing therefore breaks the browser while leaving the CLI working.

Fuzzball creates both clients in each realm when the organization is first bootstrapped. Neither has Keycloak service accounts enabled, so a grant_type=client_credentials request always fails against them and cannot be used to test whether a client is healthy. Use the two checks below instead.

First, confirm the realm exists. This needs no credentials:

$ curl -sk -o /dev/null -w '%{http_code}\n' \
    https://<host>/auth/realms/<realm>/.well-known/openid-configuration

200 means the realm is present; 404 means it is missing, which breaks every browser login in that organization.

If the realm is present, check that its fuzzball-ui client is too. This uses the Keycloak admin credentials from the deployment’s .env:

$ TOKEN=$(curl -sk -X POST https://<host>/auth/realms/master/protocol/openid-connect/token \
    -d grant_type=password -d client_id=admin-cli \
    -d username=<keycloak-admin> -d password=<keycloak-admin-password> \
    | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
$ curl -sk -H "Authorization: Bearer $TOKEN" \
    "https://<host>/auth/admin/realms/<realm>/clients?clientId=fuzzball-ui"

A JSON array containing a client object means it is present. An empty array ([]) means it is missing, which is the condition that breaks Web UI logins while the CLI keeps working.

Orchestrate records the same condition at startup, which is the quickest way to confirm what the two checks above found. When it cannot reach the realm or its fuzzball-ui client, it logs an error containing failed to bootstrap default organization:

$ docker compose -p default logs fuzzball-orchestrate | grep "bootstrap default organization"

If that line is present, recreate the realm and its clients rather than rotating secrets.

Keycloak realm and client settings come from Orchestrate’s configuration, not from the Compose file. A realm identifier written into docker-compose.yaml has no effect on which realm the deployment uses.

A multi-host topology — this host plus remote substrate nodes — is supported via generate-substrate-config, which exports the orchestrate’s CA, NATS NKey, and connection config for deployment to a separate machine. See --help on that command and Federation for the cross-cluster story.

Hardening for production (real TLS via cert-manager / Caddy, external managed Postgres + Keycloak, backup automation, log shipping) is covered in Cloud Deployment — most of the recommendations there apply equally to a single-host deployment when you want to move beyond local development.