Debugging

This section provides documentation for debugging and troubleshooting Fuzzball clusters. These tools are designed for administrators to diagnose issues, monitor service behavior, and perform advanced maintenance tasks.

When to Use Debug Commands

Debug commands are useful for:

Viewing service logs to diagnose issues
Creating debug dumps to share with the support team

Prerequisites

You will need to install the Fuzzball CLI (see the CLI installation documentation for details), set up a Fuzzball admin context and authenticate like so:

$ fuzzball admin context create default-admin-context <api_url>
Configuration for "default-admin-context" created.
Configuration for "default-admin-context" now in use.

$ FUZZBALL_ADMIN_USER="admin_username"

$ read -rs FUZZBALL_ADMIN_PASSWORD  # read interactively to set variable without echoing to screen

$ export FUZZBALL_ADMIN_USER FUZZBALL_ADMIN_PASSWORD

$ fuzzball admin context login
Logging into current cluster context...

Service Logs

The fuzzball admin debug logs command allows administrators to view logs from Fuzzball services running in the cluster. This is useful for debugging service issues, monitoring service behavior, and tracking down errors.

Basic Usage

$ fuzzball admin debug logs <SERVICE_NAME> [flags]

Available Services

The following Fuzzball services can be queried for logs:

agent - Fuzzball agent service
audit - Audit logging service
auth - Authentication service
billing - Billing service
cluster-admin - Cluster administration service
jetstream - JetStream message broker
openapi - OpenAPI service
orchestrator - Orchestrator service
provision - Provisioning service
schedule - Scheduling service
storage - Storage service
substrate-bridge - Substrate bridge service
ui - Web UI service
workflow - Workflow service

Command Flags

Flag	Short	Description	Default
`--since`	`-s`	Show logs since a specific time (RFC3339 or YYYY-MM-DD format)	-
`--since-duration`	`-d`	Show logs since a specific duration (e.g., ‘1h’, ‘30m’)	-
`--tail`	`-t`	Number of lines to show from the end of the logs	100
`--follow`	`-f`	Follow the log output (stream logs)	false
`--min-log-level`	-	Minimum log level to display (1: Debug, 2: Info, 3: Warning, 4: Error, 5: Fatal)	0
`--trace-id`	-	Filter logs by trace ID	-
`--ignore-errors`	-	Ignore errors when fetching logs	false
`--json`	`-j`	Output in JSON format	false

Example Usage

View the last 100 lines of logs from the workflow service:

$ fuzzball admin debug logs workflow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Connected to database
[fuzzball-workflow-0]: 2026-01-13T10:15:25Z INFO Workflow service ready

View logs from multiple services:

$ fuzzball admin debug logs workflow storage
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-storage-0]: 2026-01-13T10:15:22Z INFO Storage service initialized

Follow logs in real-time:

$ fuzzball admin debug logs workflow --follow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Processing workflow request

Show logs since a specific time:

$ fuzzball admin debug logs workflow --since 2026-01-13T10:00:00Z

Show logs from the last hour:

$ fuzzball admin debug logs workflow --since-duration 1h

Filter by minimum log level (errors and above):

$ fuzzball admin debug logs workflow --min-log-level 4

Show the last 500 lines:

$ fuzzball admin debug logs workflow --tail 500

Output logs in JSON format:

$ fuzzball admin debug logs workflow --json
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:23Z INFO Starting workflow service","level":2}
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:24Z INFO Connected to database","level":2}

Notes

The --since and --since-duration flags cannot be used together
When using --follow, logs will stream continuously until interrupted (Ctrl+C)
The --json flag is useful for programmatic parsing and integration with log analysis tools
Log levels: 0 (Unspecified), 1 (Debug), 2 (Info), 3 (Warning), 4 (Error), 5 (Fatal)

Cluster Dump

The fuzzball admin debug dump command creates a comprehensive dump of the Fuzzball cluster state, including configuration, logs, and diagnostic information. This is useful for gathering all relevant information when reporting issues or performing detailed troubleshooting.

Basic Usage

$ fuzzball admin debug dump [flags]

Command Flags

Flag	Short	Description	Default
`--dest`	`-d`	Destination path to save the dump	Current directory

Example Usage

Create a cluster dump in the current directory:

$ fuzzball admin debug dump

$ ls -lhd fuzzball-cluster-dump-*
drwxr-xr-x@ 9 user  group   288B Jan 15 12:11 fuzzball-cluster-dump-20260115-121100

$ tree -L2 fuzzball-cluster-dump-20260115-121100
fuzzball-cluster-dump-20260115-121100
├── cluster
│   ├── nodes
│   └── rbac
├── cluster_info.txt
├── database
│   ├── configmaps
│   ├── events.txt
│   ├── pods
│   ├── secrets
│   └── services
├── errors.txt
├── fuzzball
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── pods
│   ├── rbac
│   ├── secrets
│   └── services
└── fuzzball-system
    ├── configmaps
    ├── deployments
    ├── events.txt
    ├── pods
    ├── rbac
    ├── secrets
    └── services

$ du -sh fuzzball-cluster-dump-20260115-121100
 23M    fuzzball-cluster-dump-20260115-121100   # small dump for a test deployment

$ tar -czf fuzzball-cluster-dump-20260115-121100.tgz \
    fuzzball-cluster-dump-20260115-121100  # tar or zip for easy sharing

This creates a directory named fuzzball-cluster-dump-YYYYMMDD-HHMMSS containing the cluster state.

Dump Contents

The cluster dump typically includes:

Service logs from all Fuzzball components
Kubernetes resource configurations
Service health status
Database schema information
System configuration details
Other diagnostic information useful for troubleshooting

Notes

The dump process may take several minutes depending on cluster size and log volume
Ensure sufficient disk space is available for the dump
The dump directory is timestamped to prevent overwrites
Review the dump contents before sharing, as it may contain sensitive information
Use this command when reporting issues to CIQ support for comprehensive diagnostics

Advanced Debugging

Kubernetes-based Orchestrate deployments

For collecting some Fuzzball debugging information, administrators may need to interact with the Kubernetes cluster directly using kubectl. In this section we assume that you have the kubectl tool installed following the general installation instructions or when you configured your local Kubernetes cluster and configured to access your Kubernetes cluster.

A full tutorial about Kubernetes and kubectl are beyond the scope of this documentation. Below are some commands useful for debugging.

Direct access to the Kubernetes cluster can be potentially dangerous. Proceed with caution.

Resource Utilization

Inspect the resource consumption of your pods if you suspect that pods are approaching their limits, are overloaded, or use significantly fewer resources than they have been allocated:

$ kubectl top pods --all-namespaces --sort-by=memory --sum
NAMESPACE           NAME                                                    CPU(cores)   MEMORY(bytes)
identity            identity-keycloakx-2                                    3m           5081Mi
identity            identity-keycloakx-1                                    4m           5057Mi
identity            identity-keycloakx-0                                    3m           4990Mi
ingress             kong-kong-7bbf7cc6d7-6wq6b                              7m           470Mi
...
fuzzball            fuzzball-admin-ui-74f84b6757-m2fwt                      1m           5Mi
fuzzball            fuzzball-admin-0                                        1m           0Mi
fuzzball            postgres-debug                                          0m           0Mi
                                                                            ________     ________
                                                                            2243m        24839Mi

To ensure that your Kubernetes nodes are not running out of resources you can check with the following:

$ kubectl top nodes
NAME     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
node1    922m         11%      15603Mi         50%
node2    1077m        13%      18847Mi         61%
node3    667m         8%       15156Mi         49%

In this example the three node Kubernetes cluster is operating below capacity.

Kubernetes Pod Health

You can check that your fuzzball pods are all in a Running state and not frequently restarting like so:

$ kubectl get -n fuzzball pods
NAME                                         READY   STATUS      RESTARTS   AGE
fuzzball-admin-0                             1/1     Running     0          64m
fuzzball-admin-ui-74f84b6757-m2fwt           1/1     Running     0          64m
fuzzball-agent-7467f6484b-kjwv6              1/1     Running     0          61m
fuzzball-agent-7467f6484b-lkdmf              1/1     Running     0          62m
fuzzball-agent-7467f6484b-qxwfq              1/1     Running     0          62m
fuzzball-audit-68f75847bb-5blmp              1/1     Running     0          64m
...

Any states other than Running or Completed may signal problems, as are pods with fewer than expected Ready replicas.

More detailed information about a particular pod can be obtained with:

$ kubectl describe -n fuzzball pod/fuzzball-agent-7467f6484b-kjwv6
Name:             fuzzball-agent-7467f6484b-kjwv6
Namespace:        fuzzball
Priority:         0
Service Account:  fuzzball-agent
Node:             ip-10-0-123-57.us-west-2.compute.internal/10.0.123.57
Start Time:       Fri, 16 Jan 2026 19:56:49 -0500
Labels:           app.kubernetes.io/instance=fuzzball-agent
                  app.kubernetes.io/managed-by=pulumi
...

Jetstream Messages

A stateful set of 3 or more pods form a Jetstream cluster used for message passing between Fuzzball services. A buildup of messages for certain topics like the workflow-engine may indicate that Orchestrate is overloaded and processing events like state-changes of workflow stages too slowly. This can be checked with the following command:

$ kubectl exec -n fuzzball fuzzball-jetstream-0 -- /app/nats stream ls
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                   Streams                                                   │
├────────────────────────────────────────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name                                   │ Description │ Created             │ Messages │ Size │ Last Message │
├────────────────────────────────────────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ job-metrics-stream                     │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m2s        │
│ provisioner-aws-delete-placement-group │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ never        │
│ provisioner-core                       │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 50m14s       │
│ substrate-activity-stream              │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 56m0s        │
│ substrate-conn-stream                  │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 50m12s       │
│ substrate-lease-stream                 │             │ 2026-01-06 17:00:56 │ 0        │ 0 B  │ 56m0s        │
│ substrate-resource-discovery-stream    │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 56m36s       │
│ substrate-resource-stream              │             │ 2026-01-06 16:48:44 │ 0        │ 0 B  │ 55m27s       │
│ substrate-service-stream               │             │ 2026-01-06 17:00:57 │ 0        │ 0 B  │ 1d5h52m43s   │
│ workflow-execution                     │             │ 2026-01-06 16:49:54 │ 0        │ 0 B  │ 56m0s        │
│ leader                                 │             │ 2026-01-17 00:26:15 │ 1        │ 39 B │ 4h14m37s     │
│ provisioner-aws-pricing-ticker         │             │ 2026-01-06 16:49:54 │ 1        │ 60 B │ 10d9h16m27s  │
╰────────────────────────────────────────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯