Node Management

Cluster administrators can control the scheduling behavior of compute nodes in the Fuzzball cluster. You can mark nodes as unschedulable (cordoned) or schedulable, list all nodes, retrieve node details, and monitor node health.

All node management operations require administrative privileges. Please make sure you have created and logged into your admin context like so:

$ fuzzball context create <context_name> <api_url>

$ fuzzball context login -u <admin_username> -p '<admin_password>'

List all compute nodes

$ fuzzball node list

Get details about all compute nodes

$ fuzzball node show <node-id>

Cordon a node (make unschedulable)

$ fuzzball node cordon <node-id>

Uncordon a node (make schedulable)

$ fuzzball node uncordon <node-id>

Remove the registration of a node that no longer exists

$ fuzzball node remove <node-id>

Compute nodes can be in various states:

• Node Cordon - Mark nodes as unschedulable to prevent new workflows
• Node Uncordon - Mark nodes as schedulable to allow new workflows
• Node Get - Get details on a specific compute node
• Node List - Get details on all compute nodes
• Node Drain - Take a node out of service, optionally requeueing its work
• Node Events - Read a node’s recorded health history
• Node Remove - Delete the registration of a node that no longer exists, releasing the capacity it advertised

Compute nodes report a periodic heartbeat and hardware health signals to the control plane, which turns them into typed conditions and a 0-100 reliability score. Health is tracked separately from whether a node is schedulable: out of the box almost nothing acts on it, and a node reporting faults keeps accepting work until an administrator cordons it or enables a policy.

Two exceptions, both of which apply with no configuration: the scheduler will not place new work on a node scoring below the drain threshold, and work stranded on a node that goes offline is released rather than left to time out.

Everything below is documented in Node Health Monitoring.

Seeing what a node is doing

• What nodes report – memory, machine checks, disk, thermal, GPU, and what “unavailable” means
• Reliability score – how the number is derived, and why it is explainable
• Viewing node health – CLI, API and web UI
• Node event history – 90 days of what happened to a node, and why

Acting on it

• Nodes below the drain threshold – the one thing that happens without a policy: no new work on a failing node
• Nodes that go offline – work stranded on a node that disappeared is released, not left to time out
• Automated response – observe, cordon, drain and replace, with guardrails
• Fault-triggered eviction – evacuate running work the moment a named fault appears
• Replacing cloud nodes – terminate a failing instance and let the pool rebuild

Being told about it

• Health metrics – the Prometheus series, and how to alert on absence
• Bundled monitoring stack – optional Prometheus, Alertmanager and Grafana, preconfigured
• Webhook notifications – signed CloudEvents to an endpoint of your choosing

When something has already gone wrong

• Why a job failed on a node
• Support bundle – one command covering the control plane and the compute nodes

• fuzzball workflow list - View running & completed workflows