Debugging

This section provides documentation for debugging and troubleshooting Fuzzball clusters. These tools are designed for administrators to diagnose issues, monitor service behavior, and perform advanced maintenance tasks.

Debug commands are useful for:

• Viewing service logs to diagnose issues
• Creating debug dumps to share with the support team

You will need to install the Fuzzball CLI (see the CLI installation documentation for details), set up a Fuzzball admin context and authenticate like so:

$ fuzzball context create default-admin-context <api_url>
Configuration for "default-admin-context" created.
Configuration for "default-admin-context" now in use.

$ FUZZBALL_ADMIN_USER="admin_username"

$ read -rs FUZZBALL_ADMIN_PASSWORD  # read interactively to set variable without echoing to screen

$ export FUZZBALL_ADMIN_USER FUZZBALL_ADMIN_PASSWORD

$ fuzzball context login
Logging into current cluster context...

The fuzzball debug logs command allows administrators to view logs from Fuzzball services running in the cluster. This is useful for debugging service issues, monitoring service behavior, and tracking down errors.

$ fuzzball debug logs <SERVICE_NAME> [flags]

The following Fuzzball services can be queried for logs:

• agent - Fuzzball agent service
• audit - Audit logging service
• auth - Authentication service
• billing - Billing service
• cluster-admin - Cluster administration service
• jetstream - JetStream message broker
• openapi - OpenAPI service
• orchestrator - Orchestrator service
• provision - Provisioning service
• schedule - Scheduling service
• storage - Storage service
• substrate-bridge - Substrate bridge service
• ui - Web UI service
• workflow - Workflow service

View the last 100 lines of logs from the workflow service:

$ fuzzball debug logs workflow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Connected to database
[fuzzball-workflow-0]: 2026-01-13T10:15:25Z INFO Workflow service ready

View logs from multiple services:

$ fuzzball debug logs workflow storage
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-storage-0]: 2026-01-13T10:15:22Z INFO Storage service initialized

Follow logs in real-time:

$ fuzzball debug logs workflow --follow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Processing workflow request

Show logs since a specific time:

$ fuzzball debug logs workflow --since 2026-01-13T10:00:00Z

Show logs from the last hour:

$ fuzzball debug logs workflow --since-duration 1h

Filter by minimum log level (errors and above):

$ fuzzball debug logs workflow --min-log-level 4

Show the last 500 lines:

$ fuzzball debug logs workflow --tail 500

Output logs in JSON format:

$ fuzzball debug logs workflow --json
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:23Z INFO Starting workflow service","level":2}
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:24Z INFO Connected to database","level":2}

• The --since and --since-duration flags cannot be used together
• When using --follow, logs will stream continuously until interrupted (Ctrl+C)
• The --json flag is useful for programmatic parsing and integration with log analysis tools
• Log levels: 0 (Unspecified), 1 (Debug), 2 (Info), 3 (Warning), 4 (Error), 5 (Fatal)

The fuzzball debug dump command creates a comprehensive dump of the Fuzzball cluster state, including configuration, logs, and diagnostic information. This is useful for gathering all relevant information when reporting issues or performing detailed troubleshooting.

$ fuzzball debug dump [flags]

Create a cluster dump in the current directory:

$ fuzzball debug dump

$ ls -lhd fuzzball-cluster-dump-*
drwxr-xr-x@ 9 user  group   288B Jan 15 12:11 fuzzball-cluster-dump-20260115-121100

$ tree -L2 fuzzball-cluster-dump-20260115-121100
fuzzball-cluster-dump-20260115-121100
├── cluster
│   ├── nodes
│   └── rbac
├── cluster_info.txt
├── database
│   ├── configmaps
│   ├── events.txt
│   ├── pods
│   ├── secrets
│   └── services
├── errors.txt
├── fuzzball
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── pods
│   ├── rbac
│   ├── secrets
│   └── services
└── fuzzball-system
    ├── configmaps
    ├── deployments
    ├── events.txt
    ├── pods
    ├── rbac
    ├── secrets
    └── services

$ du -sh fuzzball-cluster-dump-20260115-121100
 23M    fuzzball-cluster-dump-20260115-121100   # small dump for a test deployment

$ tar -czf fuzzball-cluster-dump-20260115-121100.tgz \
    fuzzball-cluster-dump-20260115-121100  # tar or zip for easy sharing

This creates a directory named fuzzball-cluster-dump-YYYYMMDD-HHMMSS containing the cluster state.

The cluster dump typically includes:

• Service logs from all Fuzzball components
• Kubernetes resource configurations
• Service health status
• Database schema information
• System configuration details
• Other diagnostic information useful for troubleshooting

• The dump process may take several minutes depending on cluster size and log volume
• Ensure sufficient disk space is available for the dump
• The dump directory is timestamped to prevent overwrites
• Review the dump contents before sharing, as it may contain sensitive information
• Use this command when reporting issues to CIQ support for comprehensive diagnostics

A job that ends with a message naming a node died because that node stopped responding, not because of anything in the workflow:

node 10.0.0.42/9100: memory errors; job stopped responding on this node

Where a fault is named, the node was carrying that condition when the job died. Where only the node is named, the node had no active condition – the job may have hung, been killed for memory, or deadlocked, and the node is reported so you know where it was running.

Start with the node rather than the workflow:

$ fuzzball node show 10.0.0.42/9100
$ fuzzball node events 10.0.0.42/9100 --latest

See Node Health Monitoring for how attribution is decided.

fuzzball debug report collects the cluster dump above and diagnostics from the compute nodes, into one directory to send to support.

Node diagnostics travel over the substrate channel the nodes already hold, so no SSH access and no inbound port on the node is required. Each node contributes its health report and the recent kernel fault records behind it – the machine checks and GPU Xid records that its reliability score is derived from.

$ fuzzball debug report [flags]

Press Ctrl+C to stop a collection in progress. Whatever has already been written stays in the destination directory, but the bundle is left unfinished: its manifest.json is not amended, and node data is only written once every node has answered, so an interrupted collection usually has no node directories at all.

--nodes and --node are mutually exclusive.

Collect the control plane and the nodes worth looking at – those that are degraded, unknown, cordoned or offline:

$ fuzzball debug report --nodes unhealthy
bundle written to fuzzball-debug-report-20260812-104233

Collect from one specific node, whatever its health:

$ fuzzball debug report --node 10.0.0.42/9100

fuzzball-debug-report-20260812-104233
├── cluster/            # the cluster dump described above
├── nodes/
│   ├── gpu-14/
│   │   ├── health.json         # signals, conditions and score inputs
│   │   └── kernel-faults.txt   # the machine check and Xid records behind them
│   └── gpu-17/
│       └── error.txt           # this node did not answer, and why
└── manifest.json

Every bundle carries a manifest.json listing each source, how many artifacts it produced, and for anything that failed, the reason:

{
  "generatedAt": "2026-08-12T10:42:33Z",
  "scrubbed": true,
  "sources": [
    { "name": "pod-logs", "artifacts": 42 },
    { "name": "nodes", "artifacts": 2 },
    { "name": "node-diagnostics", "artifacts": 0,
      "error": "1 of 2 nodes did not answer: 10.0.0.17/7334" }
  ],
  "errors": ["node-diagnostics: 1 of 2 nodes did not answer: 10.0.0.17/7334"]
}

scrubbed reports whether secret values were removed. Every Kubernetes secret in the bundle keeps its name, type and key names, but each value is replaced with its length – redacted (7 bytes) rather than the credential. That is enough to answer whether a key is present, whether it is empty, and whether a certificate is plausibly a certificate, without the bundle carrying anything worth stealing.

Scrubbing covers secret values, not everything a bundle can contain. Pod logs, events and ConfigMaps are collected verbatim, and an application that logs a token puts that token in the bundle. Review a bundle before sending it outside your organization.

Collection is best effort by design. A node that does not answer is recorded in the bundle rather than failing it, because during an incident an unreachable node is frequently the fault being investigated – and a tool that returned nothing in that case would be useless exactly when it is needed.

Check errors before sending a bundle. An empty list means everything asked for was collected. Every source is listed in sources whether or not it succeeded, so a node-diagnostics entry with no error means the nodes were collected – not that they were skipped. Note the two distinct sources: nodes is the cluster’s Kubernetes Node objects, while node-diagnostics is the per-node health snapshots gathered by --nodes. node-diagnostics always reports 0 artifacts; read its error field, not its count.

Collecting a bundle does not change what the nodes report next. Some node signals are counted as growth since the previous health report. A bundle reads those counters without resetting them, so collecting one during an incident will not hide a condition from the next health report.

When the control plane itself is unreachable, fuzzball debug report cannot run. Fuzzball ships an sos report plugin for that case, installed with the fuzzball-substrate-orchestrate package:

$ sos report -o fuzzball

It collects the same host-local sources a node would have contributed – substrate configuration, service logs, EDAC counters, GPU state and kernel fault records – and falls back to kubectl for cluster state on a control plane host. Credentials in configuration files are scrubbed automatically.

When a job is placed, the scheduler publishes a resources_allocated workflow event that shows both what the job requested and the CPU placement the scheduler computed for it:

$ fuzzball workflow events WORKFLOW
...
Resources allocated (node: 10.0.0.3/7331, requested cpu: 28, requested memory: 28 GB, projected cpus: 28 (ids: 0-6,8-14,16-22,24-30), projected numa nodes: 4 (ids: 0-3))

• requested cpu, requested memory, and requested devices echo the resource request from the workflow definition. When the request sets sockets, requested cpu is the cores value, which counts cores per socket; requested devices is the total device count across all device types.
• projected cpus and projected numa nodes are the CPUs the scheduler selected for the job and the NUMA nodes they span, each rendered as a count followed by the specific IDs (count (ids: ...)). When the job requests hardware threads or multiple sockets, the projected CPU count is larger than requested cpu because it counts every schedulable CPU the job receives. For multi-node jobs, per-node values are separated by semicolons in the same order as the comma-separated node list, and each per-node count compares against requested cpu individually.

The projected values are the scheduler’s placement decision, not a reading from the node: the node’s substrate service independently computes the enforced CPU set from the same lease. The two normally match, but the projection is not a guarantee — during upgrades, transient differences in scheduler and node resource accounting, or if placement behavior ever diverges between scheduler and node, the CPUs a job actually runs on can differ from the projection.

When available, the kernel-enforced values appear on the container_started event, read from the container process on the node when it starts. They use the same count-and-IDs format, so enforced cpus compares directly against projected cpus and enforced numa nodes against projected numa nodes:

Container started on node: 10.0.0.3/7331 (enforced cpus: 28 (ids: 0-6,8-14,16-22,24-30), enforced numa nodes: 4 (ids: 0-3))

The enforced values can be omitted when the node’s substrate version predates the feature, when the container process exits before the values can be read, or when the node cannot be reached. A container_started event without enforced fields is normal during upgrades or transient node-communication failures.

Comparing the resources_allocated and container_started events for a stage shows requested, projected, and enforced values side by side. For multi-node jobs the container_started values cover the first node only. When the enforced values are missing from the event, the same values can always be read from inside a job:

$ grep -E 'Cpus_allowed_list|Mems_allowed_list' /proc/self/status

A projected CPU count lower than requested cpu (without hardware threads or sockets in play) indicates the scheduler could not compute a full placement and should be reported.

The scheduler publishes a scheduling_blocked workflow event when a job’s allocation is waiting for node reuse because its provisioner definition’s node pool has reached the maxNodes cap. The event means the job is queued, not rejected: it proceeds once a node in the pool becomes free. This is expected when the cluster is at capacity and workflows are competing for resources.

• The definition is dynamic and its pool has reached the effective cap (min(maxNodes, 128)).
• The allocation fits within the cap, but no node is currently free for reuse.
• Two companion events are published, once per blocked allocation: provisioning_blocked, which carries the definition_id, total_nodes, and max_nodes attributes, and the generic scheduling_blocked event.
• With backfill enabled (the default), an at-cap allocation may instead hold a reservation and publish a reservation_held event.

Because the events are published once per blocked allocation, judge persistence by whether the allocation stays unscheduled, not by counting events.

1. Follow the workflow’s events and look for provisioning_blocked (its definition_id attribute names the affected definition):

   $ fuzzball workflow events WORKFLOW --follow

2. Ask the scheduler why the allocation is not scheduled. A PROVISIONING_BLOCKED reason indicates the pool cap; see Queue Observability for the full reasons table:

   $ fuzzball workflow why WORKFLOW ALLOCATION

3. Review the definition’s current maxNodes value:

   $ fuzzball node provisioner get compute-pool

• Increase maxNodes: Raise the maxNodes value on the provisioner definition and apply the updated configuration:

  definitions:
    - id: compute-pool
      provisioner: pbs
      ttl: 3600
      maxNodes: 64
      provisionerSpec:
        cpu: 8
        memory: "32GiB"
        queue: "workq"
  

  $ fuzzball cluster config set updated-config.yaml

• Wait for node reuse: Transient blocking resolves on its own as running jobs complete and free their nodes.

• Reduce workflow node requirements: If feasible, request fewer nodes so allocations fit within the available pool capacity.

• Review provisioner policies: Ensure policy expressions are not concentrating load on a single definition.

The scheduler publishes a provisioning_registration_timeout workflow event when nodes it provisioned did not register within the definition’s registration deadline. Unlike scheduling_blocked, this event is terminal: the scheduler deletes the instances the provisioning request created and fails the workflow with an error of the form only 1/3 requested nodes registered within the registration deadline.

The usual cause is a bootstrap failure that the backend never reports. The instance is created and running, but the Fuzzball Substrate on it never comes up – a cloud-init error, a failed package install, an NFS mount that does not appear, or a network path that blocks the Substrate from reaching the cluster.

The event carries the following attributes. The transaction_ attributes refer to the provisioning request: Fuzzball records each request as a provisioning transaction internally, and that is the name that appears in event attributes and log messages.

A registered_nodes value greater than zero is a partial bootstrap failure: some instances came up and others did not, which usually points at a per-instance problem (a bad zone, exhausted capacity, a per-node mount) rather than a broken definition.

1. Read the event and its attributes from the failed workflow:

   $ fuzzball workflow events WORKFLOW

2. Check whether the definition’s deadline is too short for this workload. Large images, GPU instances, and slow regions can all need longer than the 15m default. The deadline lives in the central configuration and is not part of fuzzball node provisioner get output. This command prints the entire configuration as it was last set and requires cluster-admin credentials; find your definition under definitions:. A definition with no registrationDeadline key is on the 15m default:

   $ fuzzball cluster config get

3. Reproduce the failure with a single-node workflow if you need a live instance to inspect. The instances from the expired request are deleted by the time the workflow fails, so inspect the backend while the reproduction is still running.

4. Check the Orchestrate logs for the instance cleanup that followed the timeout. A failed to delete provisioner instances for expired registration transaction message means the cleanup did not complete:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=200

• Raise the deadline: If the nodes do eventually register, just later than the deadline allows, raise registrationDeadline on the definition and apply the updated configuration:

  definitions:
    - id: gpu-pool
      provisioner: aws
      ttl: 7200
      registrationDeadline: 30m
      provisionerSpec:
        instanceType: p5.48xlarge
  

  $ fuzzball cluster config set updated-config.yaml

• Fix the bootstrap failure: If the nodes never register no matter how long the deadline, raising it only delays the failure. Investigate the image, the cloud-init or package configuration, and the network path from the node to the cluster.

• Account for backend queue time: On Slurm and PBS the deadline starts at sbatch or qsub submission, so time the batch job spends queued in the backend counts against the deadline. See Workflow Stuck in PENDING in the Slurm and PBS troubleshooting guide, which covers both backends in identical sections.

If instance cleanup fails repeatedly, the scheduler stops retrying after three consecutive failures and fails the workflow anyway rather than leaving the workflow hung. Each instance it could not delete is recorded in the Orchestrate logs as failed to delete instance during transaction cleanup, and those instances may need to be cleaned up manually in the backend.

For collecting some Fuzzball debugging information, administrators may need to interact with the Kubernetes cluster directly using kubectl. In this section we assume that you have the kubectl tool installed following the general installation instructions or when you configured your local Kubernetes cluster and configured to access your Kubernetes cluster.

A full tutorial about Kubernetes and kubectl are beyond the scope of this documentation. Below are some commands useful for debugging.

Direct access to the Kubernetes cluster can be potentially dangerous. Proceed with caution.

Inspect the resource consumption of your pods if you suspect that pods are approaching their limits, are overloaded, or use significantly fewer resources than they have been allocated:

$ kubectl top pods --all-namespaces --sort-by=memory --sum
NAMESPACE           NAME                                                    CPU(cores)   MEMORY(bytes)
fuzzball-identity   identity-keycloakx-2                                    3m           5081Mi
fuzzball-identity   identity-keycloakx-1                                    4m           5057Mi
fuzzball-identity   identity-keycloakx-0                                    3m           4990Mi
ingress             kong-kong-7bbf7cc6d7-6wq6b                              7m           470Mi
...
fuzzball            fuzzball-admin-ui-74f84b6757-m2fwt                      1m           5Mi
fuzzball            fuzzball-admin-0                                        1m           0Mi
fuzzball            postgres-debug                                          0m           0Mi
                                                                            ________     ________
                                                                            2243m        24839Mi

To ensure that your Kubernetes nodes are not running out of resources you can check with the following:

$ kubectl top nodes
NAME     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
node1    922m         11%      15603Mi         50%
node2    1077m        13%      18847Mi         61%
node3    667m         8%       15156Mi         49%

In this example the three node Kubernetes cluster is operating below capacity.

You can check that your fuzzball pods are all in a Running state and not frequently restarting like so:

$ kubectl get -n fuzzball pods
NAME                                         READY   STATUS      RESTARTS   AGE
fuzzball-admin-0                             1/1     Running     0          64m
fuzzball-admin-ui-74f84b6757-m2fwt           1/1     Running     0          64m
fuzzball-agent-7467f6484b-kjwv6              1/1     Running     0          61m
fuzzball-agent-7467f6484b-lkdmf              1/1     Running     0          62m
fuzzball-agent-7467f6484b-qxwfq              1/1     Running     0          62m
fuzzball-audit-68f75847bb-5blmp              1/1     Running     0          64m
...

Any states other than Running or Completed may signal problems, as are pods with fewer than expected Ready replicas.

More detailed information about a particular pod can be obtained with:

$ kubectl describe -n fuzzball pod/fuzzball-agent-7467f6484b-kjwv6
Name:             fuzzball-agent-7467f6484b-kjwv6
Namespace:        fuzzball
Priority:         0
Service Account:  fuzzball-agent
Node:             ip-10-0-123-57.us-west-2.compute.internal/10.0.123.57
Start Time:       Fri, 16 Jan 2026 19:56:49 -0500
Labels:           app.kubernetes.io/instance=fuzzball-agent
                  app.kubernetes.io/managed-by=pulumi
...

A stateful set of 3 or more pods form a Jetstream cluster used for message passing between Fuzzball services. A buildup of messages for certain topics like the workflow-engine may indicate that Orchestrate is overloaded and processing events like state-changes of workflow stages too slowly. This can be checked with the following command:

$ kubectl exec -n fuzzball fuzzball-jetstream-0 -- /app/nats stream ls
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                   Streams                                                   │
├────────────────────────────────────────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name                                   │ Description │ Created             │ Messages │ Size │ Last Message │
├────────────────────────────────────────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ job-metrics-stream                     │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m2s        │
│ provisioner-aws-delete-placement-group │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ never        │
│ provisioner-core                       │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 50m14s       │
│ substrate-activity-stream              │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m0s        │
│ substrate-conn-stream                  │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 50m12s       │
│ substrate-lease-stream                 │             │ 2026-01-06 17:00:56 │ 0        │ 0 B  │ 56m0s        │
│ substrate-resource-discovery-stream    │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 56m36s       │
│ substrate-resource-stream              │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 55m27s       │
│ substrate-service-stream               │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 1d5h52m43s   │
│ workflow-execution                     │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 56m0s        │
│ leader                                 │             │ 2026-01-17 00:26:15 │ 1        │ 39 B │ 4h14m37s     │
│ provisioner-aws-pricing-ticker         │             │ 2026-01-06 16:49:54 │ 1        │ 60 B │ 10d9h16m27s  │
╰────────────────────────────────────────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯