Node Health Monitoring

Fuzzball compute nodes report their own health to the control plane. Each node sends a periodic heartbeat carrying hardware health signals, which lets the control plane distinguish a live node from an unreachable one and record evidence of degrading hardware.

Health findings are visible through fuzzball node list, fuzzball node show, the API, the web UI and Prometheus – see Viewing Node Health and Health Metrics.

Nothing acts on them until you configure it to. Out of the box a degraded node keeps accepting work until an operator cordons it. Two opt-in mechanisms can change that: the score-driven policy engine, and fault-triggered eviction for faults that are unambiguous on sight.

Reading orchestrate service logs requires access to the Fuzzball control plane namespace. Changing the heartbeat miss threshold requires edit access to the FuzzballOrchestrate resource in the fuzzball-system namespace.

An extension running on each compute node alongside the Fuzzball Substrate collects the following:

Each node reports on a fixed interval. A machine check, hardware error record, or GPU Xid fault in the kernel ring buffer additionally triggers an out-of-band report within a few seconds; the other signals are picked up on the next scheduled report.

For the hardware signals, Fuzzball separates one that is absent from one that failed, and never reports either as passing.

A signal is absent when the hardware or tooling for it does not exist on that machine. This is expected, and Fuzzball marks the signal unavailable so you can tell what was actually checked:

• EDAC counters require ECC memory and the EDAC kernel modules.
• SMART requires smartctl from the smartmontools package.
• GPU health requires nvidia-smi, which ships with the NVIDIA driver. Nodes with no NVIDIA GPU, or with the driver absent, report GPU health as unavailable rather than as an error. AMD GPUs are inventory-only and are not health-checked.
• I/O error counts come from the SCSI ioerr_cnt attribute and are always zero for NVMe and virtio block devices.

A signal has failed when its collector ran and could not produce a reading — an unreadable EDAC tree, a SMART probe that timed out on a dying disk, a kernel ring buffer that stopped being readable. Fuzzball records a collector failure on the node in that case, so a broken sensor is visible rather than being mistaken for a clean result. The node keeps reporting every other signal normally.

Machine check counts cover only records written since the extension started on that node, and only for the periods it could read the kernel ring buffer, so treat them as a floor rather than a complete history. If it could not skip the records already in the buffer when it started, the report says so.

A fatal machine check holds its condition open until an operator deals with it. Non-fatal records raise the condition while the count is still climbing and clear once it stops, so a node is not marked indefinitely by one record from last month.

If the extension loses its read on the ring buffer it logs a warning on the node and retries with a growing backoff, up to once a minute. Machine check detection is paused until it succeeds, and the signal is reported as a collector failure for the whole of that time rather than as passing.

On nodes with an NVIDIA GPU, Fuzzball collects per-device health and raises three conditions:

Health comes from two places, because no single source has all of it. Everything except Xid comes from nvidia-smi; Xid faults are only ever written to the kernel ring buffer, so they are read from there and matched to a device by PCI bus address. An Xid whose bus address matches no known device is still counted and still raises the condition – the fault happened regardless of whether Fuzzball could name the card.

Xid numbers identify the fault class, and Fuzzball grades the condition by number rather than treating every Xid alike. Xids that mean the card is faulty – ECC faults, NVLink errors, row-remapping records, and a GPU that has fallen off the bus (48, 63, 64, 74, 79, 92, 94, 95) – are critical. Every other Xid is a warning, because many are caused by the job’s own GPU kernel rather than the hardware. Grading those critical would let one user’s bug cordon a healthy node – and, wherever fault eviction is configured, evict work off it.

The grade is set by the worst Xid a device has logged, not the most recent one. A card that reports a double-bit ECC error and then an application exception stays critical: the hardware does not recover because a job crashed on it afterwards.

The condition message carries the most recent Xid number for each affected device, so an operator can look up what it means.

Whether the condition stays raised depends on what kind of Xid it was. A hardware Xid holds the condition open until the card is replaced – a faulty GPU does not recover because the driver went quiet. Any other Xid is raised while the count is still climbing and clears once it stops, so a job that crashed last week does not keep marking the node.

The count itself accumulates for as long as the extension runs, so read it as a total for the current run rather than as a live fault indicator.

A GPU that falls off the bus takes nvidia-smi with it, so the collector fails at the same moment the fault appears. Fuzzball reports both: the Xid condition for the fault, and a collector failure recording that GPU health can no longer be read. The Xid is graded on its number, so a hardware fault is critical even though there is no device list left to attribute it to.

Only thermal and power-brake throttling raise a condition. The driver also reports clocks held down because the GPU is idle or capped by an application clock setting, which are normal operation and would otherwise raise a condition on every healthy node.

A GPU that does not expose ECC counters – ECC disabled, or hardware that never had it – reports ECC as unavailable rather than as zero errors, so an unchecked card is never mistaken for a clean one.

Each compute node reports every 30 seconds by default. When a node misses the configured number of consecutive reports, three by default, the control plane records the node as no longer reporting and logs a warning naming the node and the time of its last report.

Heartbeat tracking does not depend on detecting a network disconnect. A node whose hardware has hung, or whose substrate extension has stopped while the host is still reachable, stops sending heartbeats and is recorded as no longer reporting on the same timer as any other silent node.

Set the number of consecutive reports a node can miss in the cluster’s FuzzballOrchestrate custom resource:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  nodeHealth:
    heartbeatMissThreshold: 3

See Node Health Configuration in the CRD reference for the full field description.

The threshold is a count, not a duration, and each node is measured against its own reporting interval. Nodes that report at different intervals therefore need no per-node tuning. With the defaults, a node is recorded as no longer reporting roughly 90 seconds after its last report.

A lower threshold detects a failed node sooner, at the cost of more false positives on a congested or high-latency network. A higher threshold reduces false positives but delays detection.

The reporting cadence is set per node, in the substrate orchestrate extension configuration rather than on the FuzzballOrchestrate resource. This applies to nodes whose extension configuration you manage directly; nodes provisioned by the Fuzzball operator use the defaults below and do not expose these settings.

binary: fuzzball-substrate-orchestrate
enabled: true
config:
  health:
    disabled: false
    report-interval-seconds: 30

Turning health reporting off also removes the node from missed-heartbeat detection. A node that has never sent a health report is never recorded as no longer reporting, because it has missed no heartbeat it promised — the same applies to a node running an extension too old to report health. Nodes that reported and then had reporting disabled are still detected.

The control plane puts a floor under the window it derives from these values, and a ceiling on the interval it derives it from. It waits at least 15 seconds after a node’s last report before recording the node as no longer reporting, however short the interval. It also treats any reported interval longer than 24 hours as 24 hours, so a node configured to report very rarely is still recorded as silent after the threshold multiplied by 24 hours — three days at the default threshold.

Alongside the individual conditions, Fuzzball derives a single 0-100 reliability score for each node. It starts at 100 and deducts points for each active condition and for job failures attributed to the node, so one number summarises how much fault evidence has accumulated. Every score records the contributions behind it, so it can be explained rather than just read.

Penalties and thresholds are set in the central configuration, cluster-wide or per node provisioner definition — see Node Health Scoring.

By default nothing acts on the score: a low score is reported so an operator can decide. Whether automation acts on it is set by the policy mode – see Automated Response.

The score is not a running tally that only counts down. It is recalculated from scratch on every health report: 100, minus the penalty for each condition currently active, minus the deduction for attributed job failures. That means the two things that pull it down recover in quite different ways.

A condition clearing restores its points immediately. Once a node stops reporting a condition it is no longer in the sum, so the score returns on the next report – within one reporting interval, 30 seconds by default. There is no cooldown and no decay. A node that scored 60 for a read-only device is back at 100 one report after the device is writable again.

Whether a condition clears at all is the node’s judgement, not a timeout. Some are deliberately sticky, because the hardware has not recovered just because the workload moved on: an uncorrectable memory error stays critical until the node is serviced. Others clear on their own – a climbing GPU Xid count clears once it stops climbing.

Attributed job failures fade instead of clearing, because a failure that has already happened cannot be undone. Their deduction decays on a half-life, failureTallyHalfLifeHours, which defaults to 168 hours – one week. Three attributed failures at the default failureTallyPenalty of 5 cost 15 points; with no further failures that is roughly 7 points a week later, 4 the week after, and rounds to nothing after about five weeks.

The decay is measured from the node’s most recent failure, not from each failure individually. A single new failure returns the whole accumulated tally to full weight, so a node failing once a week never works its history off. This is deliberate: a node that keeps failing intermittently is the case the score exists to surface.

The failure count itself is never reset – only its contribution to the score decays. A node can therefore show a long failure history in fuzzball node events while scoring 100.

Uncordoning a node does not reset its score either. Returning hardware to service is a judgement that the fault does not matter enough to keep the capacity idle, not a claim the fault is gone, so a Ready node may sit at 60 indefinitely. See Returning a node to service.

Node status, health state and reliability score appear in fuzzball node list:

$ fuzzball node list

ID        CLUSTER    HOSTNAME    STATUS    HEALTH      SCORE   JOBS
node-01   default    compute-1   Ready     Healthy     100     3
node-02   default    compute-2   Ready     Degraded    55      1
node-03   default    compute-3   Ready     Unknown     -       0

A - in the SCORE column means the node has not been scored. A node is registered as Unknown with no score and stays that way until its first health report, so newly added nodes show - until they report. Unknown health means Fuzzball cannot fully see the node – not that its hardware is faulty. This happens when:

• The node has not reported yet.
• Its telemetry has gone stale, or its heartbeat has been missed.
• A collector ran and failed. Fuzzball reports the node Unknown rather than Healthy, because one signal is unreadable and calling that healthy overstates what the control plane knows.

A node carrying real fault evidence still reports Degraded, even if a collector has also failed. A signal Fuzzball cannot read never hides one it can.

The list view cannot distinguish an unscored node from one that genuinely scored zero, so both render as -. Use fuzzball node show for the difference: an unscored node reports Score: not yet scored. A node genuinely at zero has accumulated 100 points of penalties and reads as Degraded.

fuzzball node show adds the active conditions, the score breakdown, and any job failures attributed to the node:

$ fuzzball node show node-02

Status:    Ready
Health:    Degraded (last reported 2026-08-05T14:05:00Z)
Score:     55/100 (penalties: cluster)
  -40  SmartFailing
  -5   attributed job failures
Conditions:
  Disk Degraded [Critical] SMART health check failing on /dev/sda
    first seen 2026-08-05T13:41:02Z, 12 observations
Attributed job failures: 1 (last 2026-08-05T14:02:11Z)
  Disk Degraded: 1

The (penalties: cluster) note records which configuration scored the node. Two nodes’ scores are only comparable when it matches. A node that has not reported yet shows Health: Unknown with no timestamp and Score: not yet scored.

The same fields are available over the API and in the web UI nodes table. For scripting, use fuzzball node show NODE --output json, which includes the full condition list and score explanation.

The web UI dashboard summarises the same data per cluster, so a degraded node is visible without opening the nodes table. Each cluster row reports the worst state present – All healthy, 2 degraded, worst 40, or 1 unknown – and a fleet-wide tile shows how many nodes are healthy in total. A cluster Fuzzball cannot currently reach reports no health at all rather than the last answer it gave, because that answer is no longer known to be true.

In a federated deployment, health is reported for nodes in every attached cluster: the aggregated listing shows the same health state and score as the cluster that owns the node.

Node health is exported to Prometheus for dashboards and alerting:

A node that has not been scored yet publishes no node_reliability_score series, so a missing score and a score of zero stay distinguishable. Alert on absence separately rather than treating a missing series as healthy. Its health state and condition series are still published, reporting the node as unknown, so a node that never starts reporting is still visible. When a node is decommissioned all of its series are removed.

Condition severity is the gauge’s value rather than a label, so raising, escalating and clearing a condition all move the same series – as a label, a cleared condition would strand its old severity reading 1 forever and the alert would never resolve. Alert on fuzzball_scheduler_node_condition != 0 for any raised condition, or >= 3 for critical ones only.

Condition labels are lower-cased names of the condition types listed under Condition penalties – memory_errors, disk_degraded, heartbeat_missed and so on.

Every condition type gets a series once a node reports, including the ones that carry no score penalty. Two of those, heartbeat_missed and collector_failed, are worth alerting on directly: neither moves the score, so an alert is the only way to hear about them.

telemetry_stale is part of the condition vocabulary but nothing raises it, so its series stays at 0 and an alert written against it will never fire. Use heartbeat_missed for a node that has stopped reporting.

The three gpu_* conditions are raised on nodes with an NVIDIA GPU – see GPU Health.

The control plane also publishes metrics for the storage that holds each node’s resource report – see Node Report Storage.

Fuzzball can deploy Prometheus, Alertmanager and Grafana for you, preconfigured against the metrics above. It is off by default: a site that already runs monitoring should scrape the same endpoints instead of taking a second copy it then has to patch.

Enable it on the FuzzballOrchestrate resource:

spec:
  monitoring:
    enabled: true
    retentionDays: 15
    storageSize: 20Gi
    alerting:
      webhookURL: https://alerts.example.com/hook

Alertmanager comes from the Prometheus chart rather than being deployed separately, so there is one alert pipeline rather than two that can disagree.

Installed whether or not a receiver is configured. Without one, alerts still collect in the Alertmanager UI rather than evaluating into nothing.

The two score thresholds match the policy engine’s defaults, so an alert and an automated cordon agree about when a node is in trouble. If you change cordonBelow or drainBelow, change these to match – otherwise the alerts describe a policy you are no longer running.

FuzzballNodeHealthMetricsAbsent is the one that catches a broken pipeline. Every other rule goes quiet when the control plane publishes nothing, so without it an unscraped cluster looks exactly like a healthy one.

Every expression aggregates with max by (node_id). Only the scheduler replica holding leadership publishes node series, and it drops them when it loses leadership, so during a failover two replicas can briefly serve the same node – an expression that summed would double count.

Grafana is deployed with Prometheus as its default datasource and a Fuzzball Node Reliability dashboard provisioned: nodes by health state, the lowest score in the fleet, score history per node, active conditions, and job failures attributed to each node.

The exception: the scheduler will not place queued work on a node whose score has fallen below drainBelow (45 by default), whatever the policy mode. Work already running there is left alone – moving it is a drain, which stays opt-in.

Health is advisory everywhere else, and deliberately so. This is the one case where it is not, because the evidence is already in hand and the work has not started: starting a job on hardware the scheduler has concluded is failing helps nobody.

Two nodes are not gated:

• A degraded node above the threshold. Degraded starts at 95 by default, so gating on the health state would exclude a node that has lost five points for a correctable error.
• A node that has never reported health. Its score reads zero because it has no score yet, and treating that as a real score would make every newly registered node unschedulable until its first report.

The threshold is resolved per provisioner definition, so a GPU pool with its own drainBelow gates on its own number.

When a node disappears – power loss, a killed agent, an instance terminated underneath Fuzzball – the work placed on it is already gone. Fuzzball releases it immediately rather than waiting for each lease to time out one at a time:

• Workflows with policy.restart: on-node-fault are requeued and run again elsewhere.
• Everything else fails, with the node named.

This is not opt-in, unlike fault-triggered eviction. That opt-in exists because evacuating a working node is a judgement call. Here there is no judgement to make: the node is gone and the work died with it, so the only question is whether you are told now or after a timeout. A multinode job in particular no longer holds its remaining nodes for the length of that wait.

The failure names no fault class. A node disappearing says nothing about its hardware – a rack losing power and an agent being killed look the same from the control plane – so Fuzzball reports what it knows and no more:

node 10.0.0.42/9100 went offline; job failed. Set policy.restart to on-node-fault for work that is safe to run again from the beginning

When a job ends because its node stopped responding, the error names the node:

job stopped responding on node 10.0.0.42/9100

If that node is carrying a health condition at the time, the error names the fault as well, so a user can tell immediately that the failure was not theirs:

node 10.0.0.42/9100: memory errors; job stopped responding on this node

A fault is only named when the node is actually carrying one. A job can stop responding for reasons that have nothing to do with the hardware – it hung, it was killed for memory, the code deadlocked – and naming a fault on that evidence would send somebody investigating hardware that is fine. The node is still named either way, because knowing which machine a job died on is useful on its own.

Two conditions are never named as the cause: heartbeat_missed and telemetry_stale. Both mean Fuzzball could not see the node, which is not the same as the node being broken. collector_failed is excluded for the same reason – a sensor that stopped reporting is an absence of evidence, not evidence of a fault.

The failure is also counted against the node’s failure tallies, which feed its score.

The control plane keeps the latest resource report for every attached node in a JetStream key-value bucket named substrate-nodes. Bucket usage grows with the number of nodes and with the size of each report, and a report grows with the node’s core count, its devices and its annotations.

Size the bucket from the node count multiplied by the size of one report. A 192-core node with 8 GPUs and 10 annotations produces a report of roughly 4 KiB, so 3,000 such nodes need about 12 MiB. The bucket defaults to 64 MiB, which holds roughly 16,000 reports of that size.

A full bucket fails quietly – no error surfaces in the CLI, the API or a node’s status. JetStream stops accepting reports from nodes the bucket has not seen before, and stops accepting any report larger than the one it replaces. The cluster stops registering new nodes, and the scheduler keeps planning against the last resources it recorded for the nodes it already has. The only signals are the substrate bridge log, which reports a rejection at error level at most once every 30 seconds, and the metrics in Bucket Metrics.

Set the bucket’s size cap with nodesBucketMaxBytes in the cluster’s FuzzballOrchestrate custom resource:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-orchestrate
spec:
  fuzzball:
    substrate:
      nodesBucketMaxBytes: 1073741824

See Substrate Configuration in the CRD reference for the full field description.

When the substrate bridge restarts, it applies the new cap, resizing the existing bucket in place. Leaving the field unset changes nothing: the 64 MiB default is the size clusters ran with before the field existed.

Check fuzzball_substrate_nodes_bucket_bytes_used before lowering a cap. Any cap below what the bucket already stores produces the same failure as a full bucket. The field’s 64 MiB minimum only rejects caps below 64 MiB; it does not compare the new cap against what the bucket currently stores, so a bucket already holding more than 64 MiB can still be capped below its own contents.

The cap is a reservation, not a limit on actual use. JetStream subtracts it from the NATS server’s on-disk allowance of 10 GiB whether or not the bucket holds that much, and every other Fuzzball stream on that server draws on the same allowance.

That allowance is per server. A high-availability deployment replicates the bucket across three NATS servers, and each server reserves the full cap against its own 10 GiB. A 1 GiB cap therefore costs 1 GiB on every server, not 3 GiB on any one of them.

An oversized reservation stops the substrate bridge from starting rather than degrading quietly, so the cap is bounded at 2 GiB. The Kubernetes API server rejects a nodesBucketMaxBytes value below 64 MiB or above 2 GiB when the custom resource is applied, so a mistyped size fails at apply time. A value that reaches the orchestrate service configuration by another path is clamped into that range rather than preventing the bridge from starting. The bridge logs the cap in force at startup, and logs the configured value alongside it only when that value had to be clamped. Raise nodesBucketMaxBytes as the node count grows; do not set a large value up front.

Three series published by the substrate bridge cover the bucket:

The two gauges are sampled once a minute, regardless of how many nodes are attached. Alert when fuzzball_substrate_nodes_bucket_bytes_used exceeds 80% of fuzzball_substrate_nodes_bucket_bytes_max. That leaves room to raise the cap and restart the substrate bridge before the bucket fills.

Both gauges are absent until the first successful sample, so absence means usage is unknown rather than zero. Pair a headroom alert with an absent() check rather than treating a missing series as an empty bucket.

fuzzball_substrate_nodes_bucket_put_failures_total is the counterpart signal: the gauges warn before the bucket fills, and the counter shows that reports are already being rejected. It increases once per rejected write, and because a rejected report is left unacked and redelivered until it succeeds, a single stuck node produces a steady rise. Alert on any sustained increase – the report itself is not lost, but the node’s recorded resources stay frozen until the write succeeds.

Node health is tracked separately from whether a node is schedulable. A node reporting hardware faults stays in service and continues to accept new workflows unless a policy is configured to act, with one exception described below. Out of the box no policy acts: health reporting records evidence, and a missed heartbeat does not change a node’s status.

To take a node out of service by hand, cordon or drain it. See Node Cordoning and Uncordoning and Draining a Node.

Draining takes a node out of service. It cordons the node, so the scheduler places no new work on it, and then decides what to do about the work already running there.

$ fuzzball node drain 10.0.0.42/9100

By default the drain is passive: running jobs are left to finish. This is safe for any workload.

$ fuzzball node drain 10.0.0.42/9100 --active --reason "failing disk"

--active also requeues the jobs currently running on the node. They stop where they are and run again from the beginning on another node.

--active restarts jobs from the beginning. Only use it for work that tolerates being restarted – a job that has already written partial output, or that is not idempotent, may produce wrong results when re-run. Fuzzball does not checkpoint.

Draining does not introduce a separate node state. A drained node is Cordoned, and fuzzball node uncordon returns it to service. What the drain did – passive or active, and how many allocations it requeued – is recorded on the node event stream.

Only a ready node can be drained. A node that is Offline or Not Ready is not accepting work in the first place, and draining one fails rather than reporting a change that did not happen:

$ fuzzball node drain 10.0.0.42/9100

Error: node 10.0.0.42/9100 is NOT_READY and cannot be drained: only a READY or
already cordoned node can be taken out of service

--all drains every ready node and skips the rest, so the count it reports is what it actually took out of service rather than how many nodes it examined:

$ fuzzball node drain --all

Cordoned 12 node(s)

For scripting, --output json returns the same counts as structured data.

Fuzzball can act on the reliability score on its own. It does not by default: the policy mode is observe, which records what it would have done and changes nothing.

nodeHealth:
  mode: observe   # observe | cordon | drain | replace

Modes are set cluster-wide or per node provisioner definition, and a definition’s setting replaces the cluster’s – so a GPU pool can enforce while the rest of the cluster observes. See Node Health Scoring.

An automated drain is always passive. Automation never requeues running work, because restarting a job is a decision only the workflow that submitted it can make.

The score is derived from penalties that are a judgement call, not a fitted model. On a fixed-size cluster a false positive removes capacity that no amount of disagreement gets back. Observe mode produces a record of every action the policy would have taken, so a site can read its own event history and decide whether the recommendations were right before enabling enforcement.

Filter the node event stream for POLICY_RECOMMENDED events and compare them against what actually happened to those nodes. Each one is a distinct episode rather than a repeat – see What lands on the event stream – so counting them gives a usable false-positive rate directly.

Enforcement is bounded, because the failure that matters is not one wrong decision but a bad threshold condemning a whole pool at once.

• Unavailable ceiling. Automation will not act if doing so would take more than maxUnavailablePercent of a provisioner definition’s nodes out of service (default 20%). The count includes nodes cordoned by an operator, not just by automation: the ceiling is about how much capacity is missing, and a node is equally gone either way. When the ceiling blocks an action Fuzzball records a POLICY_HALTED event rather than logging quietly, because nodes failing faster than the policy may remove them needs an operator.
• Manual uncordon wins. After an operator uncordons a node, automation leaves it alone for cordonSuppressionMinutes (default 30). A manual uncordon is a human overruling the policy, and re-cordoning on the next health report would make the override meaningless.
• Nodes Fuzzball cannot fully see are never acted on. A node reporting HEARTBEAT_MISSED, TELEMETRY_STALE or COLLECTOR_FAILED is left alone whatever its score. Its score is the last one computed from complete data, and acting on it during a network interruption is how a healthy cluster gets drained. COLLECTOR_FAILED is the same case with a sensor rather than a network: the reading is missing, so the score behind it is incomplete. No policy action is taken and no policy event is emitted – the node is held at Unknown with the condition attached, and an operator decides from there.
• A node’s first score never triggers action. Policy evaluation starts from a node’s second report, so hardware is never removed on the strength of one reading.

On a cloud pool a failing node is not something to hold and repair – it is something to throw away. Set mode: replace and Fuzzball terminates the instance once it is drained, leaving the provisioner to supply a fresh one the next time a workflow needs the capacity.

nodeHealth:
  mode: replace
  maxReplacementsPerHour: 3

Only nodes Fuzzball provisioned are ever terminated. A node the site racked itself is drained and left cordoned with its conditions attached, exactly as drain mode would leave it, because terminating hardware somebody owns is not a repair and nothing would rebuild it.

Fuzzball also only terminates an instance it took out of service itself. If you cordon or drain a node by hand, Fuzzball leaves the instance alone and records a POLICY_HALTED event instead – a node you took out of service is one you are working on, and automation destroying it underneath you would take the evidence with it.

Eviction and replacement answer the same question differently depending on what the node is:

Replacement of an evicted instance requires mode: replace. Without it an evicted instance stays cordoned like owned hardware.

Terminating an instance discards anything only on its local disk. Node-level diagnostics survive on the node event stream, which outlives the node record, and most providers keep serial console output and instance logs independently. If your investigation needs more than that, collect a support bundle before enabling replace mode, or leave it off for the affected definition.

Declining to replace a node is not the same as keeping it. Any dynamically provisioned node is torn down a few minutes after it goes idle, whoever cordoned it and whatever the policy mode. Cordoning a cloud instance does not reserve it, so gather whatever evidence you need while work is still running on the node.

The sequence for a node that crosses drainBelow:

1. The node is drained, which stops new placements. Running work is left alone.
2. Fuzzball waits for that work to finish, re-checking on every health report. A node with running jobs records a POLICY_ACTED event naming how many allocations it is waiting on.
3. Once the node is empty, the instance is terminated and a NODE_REPLACED event is recorded.

There is no separate step that builds the replacement. A node provisioner definition has no minimum size – its nodes exist because a workflow asked for capacity – so the replacement is provisioned when the next workflow needs it. Nothing is billed while the pool is idle.

To move work off a failing node immediately rather than waiting for it to finish, configure fault-triggered eviction as well. The two compose: eviction empties the node, replacement then terminates it.

maxReplacementsPerHour (default 3) bounds how many of a definition’s nodes may be terminated in a rolling hour.

It guards against a bad machine image, instance type, or availability zone. Every replacement then comes up as broken as the node it replaced, and the pool churns instances at cloud prices with nothing to show for it. At the ceiling Fuzzball stops replacing and records a POLICY_HALTED event naming the count, which is the signal that the problem is the pool and not the nodes.

A termination the cloud provider rejects is not counted – a transient API error should not spend the hour’s budget.

The policy above acts on the score, which measures how much evidence has accumulated against a node. Some faults do not need accumulating. A single uncorrectable GPU error means the work running on that GPU is already producing garbage, however healthy the rest of the node looks.

Name those fault classes and Fuzzball evacuates the node the moment one is raised:

nodeHealth:
  evictOnFaultClasses:
    - GPU_ECC_ERRORS
    - GPU_XID_ERRORS
  maxEvictionRestarts: 3

Nothing evicts until a fault class is named. This is independent of the policy mode: eviction answers “is this hardware actively destroying work”, which is a different question from “should this node be taken out of service”, so a cluster in observe mode can still evacuate on a fatal GPU error.

When a configured fault is raised:

1. The node is cordoned. On hardware the site owns it stays cordoned until an operator uncordons it – the cordon plus the conditions on the node record are the evidence, and there is no separate maintenance state. On a Fuzzball-provisioned instance the node is replaced instead, if replace mode is enabled; see Evicted nodes on cloud instances.
2. Each running job is either moved to another node or failed, according to what it declared – see Restart tolerance. A job that declared nothing is failed rather than silently repeated.
3. A NODE_EVICTED event records the fault class and how much work moved.

Failed jobs name the node and the fault instead of reporting that the job stopped responding:

node gpu-14: GPU ECC errors; job failed and the node cordoned. Set policy.restart to on-node-fault for work that is safe to run again from the beginning

A job that was moved instead of failed reports the move, so a user is not sent looking for a rerun that is not coming:

node gpu-14: GPU ECC errors; work moved off the node and the node cordoned

maxEvictionRestarts (default 3) bounds how many times one job may be moved by node faults before it is failed instead, so work cannot circulate through a rack of failing hardware, restarting from the beginning each time.

Eviction is only as good as the fault classes you name. Start with the ones that unambiguously corrupt work in progress – GPU_ECC_ERRORS, GPU_XID_ERRORS, MACHINE_CHECK_ERRORS – and use the node event stream to check how often they fire before adding more.

Automation only ever takes nodes out of service. A node it cordons stays cordoned until someone runs fuzzball node uncordon, even if its score recovers to 100. Putting hardware back is a judgement about whether the underlying fault was fixed, and the score cannot see a replaced DIMM or a swapped cable – only that the errors stopped.

Policy events record decisions, not reports. A node whose score stays below a threshold produces one event when the policy reaches its conclusion, not one per health report, so an episode reads as a single entry however long the hardware stays bad. A new event follows when the conclusion changes – the recommendation escalates from cordon to drain, a guardrail starts or stops blocking, or the node recovers and degrades again.

NODE_REPLACED is the last event a node produces. Its record is purged along with the instance, so the event stream is the only place the reason survives.

Events are retained for 90 days and can be read back per node through the ListNodeEvents API, filtered by time and by event type and returned oldest first in pages.

The history is what makes a score legible after the fact. A node sitting at 55 tells you it is degraded now; the history tells you whether it has been there for a month or arrived this morning, and what happened each time it moved.

SCORE_CHANGED is recorded for every score change, not only the ones that cross a policy rung, so the sequence of these events is the score’s full history and a trend can be plotted directly from it. That is affordable because the score is a step function: it is derived from the node’s active conditions and a rounded failure-tally deduction, so it does not move between reports unless the evidence moves. A healthy node produces none of these at all, and a failing one produces a handful.

Read a node’s history with fuzzball node events, oldest first:

$ fuzzball node events 10.0.0.42/9100 --since 24h
TIME                 | TYPE             | DETAIL
2026-08-11T09:14:02Z | CONDITION_RAISED | 4 correctable GPU ECC error(s) since driver load
2026-08-11T09:14:02Z | SCORE_CHANGED    | score 100 -> 90
2026-08-11T11:37:41Z | CONDITION_RAISED | 1 uncorrectable GPU ECC error(s) on GPU 0
2026-08-11T11:37:41Z | SCORE_CHANGED    | score 90 -> 55
2026-08-11T11:37:41Z | POLICY_ACTED     | cordon: score 55

Filter by type with --type, repeatable, and use -o json to pull the series out for plotting:

$ fuzzball node events 10.0.0.42/9100 --type score_changed -o json

By default a read starts at the oldest retained event and pages forward. On a node with a long history that is the wrong end to look at first, so --latest returns the most recent events instead:

$ fuzzball node events 10.0.0.42/9100 --latest --page-size 20

--latest cannot be paged – it walks to the end of the history and returns no continuation token, because there is nothing after it.

The same history drives the score trend on the node’s detail page in the web UI.

Reads are scoped to a single node. Events are stored per node in the event stream and the node is the only index it has, so there is no cluster-wide event query – use the metrics endpoint for fleet-wide views.

Node events can be delivered to an HTTP endpoint as they happen, so a site’s existing on-call tooling learns about failing hardware without polling Fuzzball.

nodeEventWebhooks:
  - url: https://alerts.example.com/fuzzball
    secret: <shared secret>
    events:
      - POLICY_ACTED
      - POLICY_HALTED
      - NODE_REPLACED

Endpoints are cluster-scoped and take effect on the next configuration reload – no restart. Omit events to receive everything, though most receivers want a filter: condition and score events fire on every transition across the fleet.

Each delivery is a CloudEvents 1.0 structured-mode JSON envelope, sent as application/cloudevents+json; charset=utf-8. The data member is the node event exactly as the API renders it, so a receiver and an operator running fuzzball node show are looking at the same shape.

{
  "specversion": "1.0",
  "type": "dev.ciq.fuzzball.node.POLICY_ACTED",
  "source": "/fuzzball/orchestrate/scheduler",
  "id": "0f7c5e4a-3b21-4d8e-9c17-2a6b8e5d1f40",
  "time": "2026-08-11T17:04:22Z",
  "subject": "node-gpu-14",
  "datacontenttype": "application/json",
  "data": {
    "nodeId": "node-gpu-14",
    "type": "POLICY_ACTED",
    "occurredAt": "2026-08-11T17:04:22Z",
    "action": "ACTION_DRAIN",
    "score": 30,
    "policyMode": "drain",
    "message": "drain: score 30"
  }
}

Every request carries three headers:

The signature is HMAC-SHA256, keyed with the endpoint’s secret, over the timestamp, a literal ., and the raw request body:

signature = HMAC_SHA256(secret, timestamp + "." + body)

Verify with a constant-time comparison, and reject any timestamp more than a few minutes from the present – five is a reasonable default. The timestamp is inside the signed material rather than merely alongside it: signing the body alone would leave a captured delivery valid forever, so anything that recorded one could replay it later.

Retries reuse the X-Fuzzball-Event-Id but are signed afresh, so each attempt carries a different timestamp and signature. Deduplicate on the event id, not on the signature.

There are none, deliberately. The event stream in JetStream is the durable record and is queryable for 90 days; a webhook is a notification on top of it.

• Delivery happens off the scheduler’s path. An endpoint that is slow, down or misconfigured cannot affect scheduling.
• Failed attempts are retried with backoff up to maxAttempts, then dropped with a log line. An endpoint down long enough to exhaust those has a gap in its history whatever Fuzzball does, and retrying indefinitely would trade that gap for a backlog that never drains.
• If a receiver cannot keep up with a correlated burst – a rack losing power raises a condition on every node at once – events are dropped rather than queued without bound.

If you need a complete record, read the event stream rather than relying on webhook delivery.