Debugging

This section provides documentation for debugging and troubleshooting Fuzzball clusters. These tools are designed for administrators to diagnose issues, monitor service behavior, and perform advanced maintenance tasks.

When to Use Debug Commands

Debug commands are useful for:

Viewing service logs to diagnose issues
Creating debug dumps to share with the support team

Prerequisites

You will need to install the Fuzzball CLI (see the CLI installation documentation for details), set up a Fuzzball admin context and authenticate like so:

$ fuzzball admin context create default-admin-context <api_url>
Configuration for "default-admin-context" created.
Configuration for "default-admin-context" now in use.

$ FUZZBALL_ADMIN_USER="admin_username"

$ read -rs FUZZBALL_ADMIN_PASSWORD  # read interactively to set variable without echoing to screen

$ export FUZZBALL_ADMIN_USER FUZZBALL_ADMIN_PASSWORD

$ fuzzball admin context login
Logging into current cluster context...

Service Logs

The fuzzball admin debug logs command allows administrators to view logs from Fuzzball services running in the cluster. This is useful for debugging service issues, monitoring service behavior, and tracking down errors.

Basic Usage

$ fuzzball admin debug logs <SERVICE_NAME> [flags]

Available Services

The following Fuzzball services can be queried for logs:

agent - Fuzzball agent service
audit - Audit logging service
auth - Authentication service
billing - Billing service
cluster-admin - Cluster administration service
jetstream - JetStream message broker
openapi - OpenAPI service
orchestrator - Orchestrator service
provision - Provisioning service
schedule - Scheduling service
storage - Storage service
substrate-bridge - Substrate bridge service
ui - Web UI service
workflow - Workflow service

Command Flags

Flag	Short	Description	Default
`--since`	`-s`	Show logs since a specific time (RFC3339 or YYYY-MM-DD format)	-
`--since-duration`	`-d`	Show logs since a specific duration (e.g., ‘1h’, ‘30m’)	-
`--tail`	`-t`	Number of lines to show from the end of the logs	100
`--follow`	`-f`	Follow the log output (stream logs)	false
`--min-log-level`	-	Minimum log level to display (1: Debug, 2: Info, 3: Warning, 4: Error, 5: Fatal)	0
`--trace-id`	-	Filter logs by trace ID	-
`--ignore-errors`	-	Ignore errors when fetching logs	false
`--json`	`-j`	Output in JSON format	false

Example Usage

View the last 100 lines of logs from the workflow service:

$ fuzzball admin debug logs workflow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Connected to database
[fuzzball-workflow-0]: 2026-01-13T10:15:25Z INFO Workflow service ready

View logs from multiple services:

$ fuzzball admin debug logs workflow storage
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-storage-0]: 2026-01-13T10:15:22Z INFO Storage service initialized

Follow logs in real-time:

$ fuzzball admin debug logs workflow --follow
[fuzzball-workflow-0]: 2026-01-13T10:15:23Z INFO Starting workflow service
[fuzzball-workflow-0]: 2026-01-13T10:15:24Z INFO Processing workflow request

Show logs since a specific time:

$ fuzzball admin debug logs workflow --since 2026-01-13T10:00:00Z

Show logs from the last hour:

$ fuzzball admin debug logs workflow --since-duration 1h

Filter by minimum log level (errors and above):

$ fuzzball admin debug logs workflow --min-log-level 4

Show the last 500 lines:

$ fuzzball admin debug logs workflow --tail 500

Output logs in JSON format:

$ fuzzball admin debug logs workflow --json
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:23Z INFO Starting workflow service","level":2}
{"pod_name":"fuzzball-workflow-0","message":"2026-01-13T10:15:24Z INFO Connected to database","level":2}

Notes

The --since and --since-duration flags cannot be used together
When using --follow, logs will stream continuously until interrupted (Ctrl+C)
The --json flag is useful for programmatic parsing and integration with log analysis tools
Log levels: 0 (Unspecified), 1 (Debug), 2 (Info), 3 (Warning), 4 (Error), 5 (Fatal)

Cluster Dump

The fuzzball admin debug dump command creates a comprehensive dump of the Fuzzball cluster state, including configuration, logs, and diagnostic information. This is useful for gathering all relevant information when reporting issues or performing detailed troubleshooting.

Basic Usage

$ fuzzball admin debug dump [flags]

Command Flags

Flag	Short	Description	Default
`--dest`	`-d`	Destination path to save the dump	Current directory

Example Usage

Create a cluster dump in the current directory:

$ fuzzball admin debug dump

$ ls -lhd fuzzball-cluster-dump-*
drwxr-xr-x@ 9 user  group   288B Jan 15 12:11 fuzzball-cluster-dump-20260115-121100

$ tree -L2 fuzzball-cluster-dump-20260115-121100
fuzzball-cluster-dump-20260115-121100
├── cluster
│   ├── nodes
│   └── rbac
├── cluster_info.txt
├── database
│   ├── configmaps
│   ├── events.txt
│   ├── pods
│   ├── secrets
│   └── services
├── errors.txt
├── fuzzball
│   ├── configmaps
│   ├── deployments
│   ├── events.txt
│   ├── pods
│   ├── rbac
│   ├── secrets
│   └── services
└── fuzzball-system
    ├── configmaps
    ├── deployments
    ├── events.txt
    ├── pods
    ├── rbac
    ├── secrets
    └── services

$ du -sh fuzzball-cluster-dump-20260115-121100
 23M    fuzzball-cluster-dump-20260115-121100   # small dump for a test deployment

$ tar -czf fuzzball-cluster-dump-20260115-121100.tgz \
    fuzzball-cluster-dump-20260115-121100  # tar or zip for easy sharing

This creates a directory named fuzzball-cluster-dump-YYYYMMDD-HHMMSS containing the cluster state.

Dump Contents

The cluster dump typically includes:

Service logs from all Fuzzball components
Kubernetes resource configurations
Service health status
Database schema information
System configuration details
Other diagnostic information useful for troubleshooting

Notes

The dump process may take several minutes depending on cluster size and log volume
Ensure sufficient disk space is available for the dump
The dump directory is timestamped to prevent overwrites
Review the dump contents before sharing, as it may contain sensitive information
Use this command when reporting issues to CIQ support for comprehensive diagnostics