Removing Node Registrations

A compute node appears in Fuzzball because its agent registers with the control plane and reports what hardware it has. That registration is what makes the node schedulable, and it is what the scheduler counts when deciding whether the cluster has capacity for queued work.

Removing a registration is different from cordoning or draining. A cordoned node is still a node: it stops accepting new work, but it remains in fuzzball node list and the control plane still expects it to be there. Removal deletes the registration outright, so the node and the resources it was advertising leave the scheduler’s view entirely.

Removing a registration requires an authorized Fuzzball admin context.

$ fuzzball context create <context_name> <api_url>

$ fuzzball context login -u <admin_username> -p '<admin_password>'

Normally you never need to think about this. A node whose host shuts down cleanly deregisters itself, and one that disappears without doing so is deregistered automatically once the control plane finds it unreachable.

A registration can still outlive its node when the whole cluster goes down at once – a host power cycle, or a Docker Compose deployment where the messaging service and the compute node are stopped together. The signal that would have triggered deregistration is lost along with the service that was meant to act on it, and the registration is left behind.

Until it is cleaned up, such a registration still advertises its node’s cores, memory and GPUs as free capacity, so the scheduler may try to place work on a machine that no longer exists. Cordoning does not release that capacity – only removal does.

The control plane re-checks every registration on a recurring sweep and deregisters any node it cannot reach across several consecutive attempts. A registration left behind by a restart is normally gone within a few minutes, without any operator action.

A single failed check is never enough to deregister a node, so a node that is briefly unreachable – restarting a service, or a short network interruption – keeps its registration.

Each automatic removal is recorded in the fuzzball_substrate_node_registrations_reaped_total metric. Occasional increments are expected. A rate that grows with the size of your cluster means the control plane cannot reach nodes that are in fact healthy, which is worth investigating rather than ignoring.

To remove a registration immediately rather than waiting for the sweep, use fuzzball node remove with the identifier shown in the ID column of fuzzball node list:

$ fuzzball node remove <node-id>

That identifier is the node’s address, such as 172.18.0.4/7333. Control plane logs refer to the same node by a longer key that also carries its hostname, such as node-a/172.18.0.4/7333; pass the address form from node list, not the longer key.

Example:

$ fuzzball node remove 172.18.0.4/7333
Removed the registration for node 172.18.0.4/7333

Any work the scheduler still believed was running on the node is released and requeued, rather than being left to time out one allocation at a time.

Because removal releases running work, a node the scheduler still shows work on is refused:

$ fuzzball node remove 172.18.0.4/7333
Error: node 172.18.0.4/7333 still has 2 running allocation(s); removing
its registration would evacuate them. Drain the node first, or pass force to
remove it anyway

A node with running jobs is far more likely to be alive than to be a leftover registration, so this usually means the identifier is not the one you meant. If you are certain, --force removes it and reports what that cost:

$ fuzzball node remove 172.18.0.4/7333 --force
Removed the registration for node 172.18.0.4/7333
Evacuating 2 allocation(s) that were running on it

Removal is for registrations whose node is gone. It is not a way to take a working node out of service, and it is not cheap to undo:

• A node reports its resources when they change, not on a timer, so removing a live node’s registration takes that node out of the cluster until its resources change or its agent restarts. It does not come back on its own.
• To stop new work reaching a node, use fuzzball node cordon.
• To take a node out of service and decide what happens to the work already on it, use fuzzball node drain.
• To tear down a cloud instance provisioned by Fuzzball, use fuzzball node deprovision, which releases the underlying instance as well as the registration.

In a Docker Compose deployment a node’s identity includes its container address, so recreating the compute node container on a new address registers a new node rather than updating the existing one. This is not damaging – the old registration is reaped automatically, and fuzzball node remove clears one immediately, while the recreated container registers itself on startup – but you can avoid the churn:

• Run docker compose down before bringing the stack back up when the compute node container will be recreated, rather than a bare docker compose up -d. A clean stop deregisters the node while the control plane is still running to notice it.
• Recreate only the service that changed (docker compose up -d --no-deps <service>) so the compute node keeps its address.