Provisioner Configuration Reference

This document provides an exhaustive reference for all configuration parameters available in the Fuzzball central configuration system. The central configuration uses YAML format and supports node provisioners across multiple node provisioner backends with their specific parameters.

# Global cluster settings
nodeAnnotations:
  # Map of global annotations applied to all nodes
  # For example:
  global.annotation: "cluster-wide-value"
  environment: "production"

softwareTokens:
  # Map of software license token limits
  # For example:
  matlab: 20
  ansys: 10

scheduler:
  queueDepth: 64
  # Annotation keys that scheduler annotation matching should skip when
  # comparing workflow job annotations against provisioner definitions'
  # Resource.Annotations. Add keys that are routed by a provisioner-
  # definition `policy:` expression rather than by Resource.Annotations.
  ignoredAnnotations:
    - nodepool

nodeHealth:
  # Penalties and thresholds for the per-node reliability score.
  # Omit entirely to accept the defaults.
  cordonBelow: 65

image:
  # Controls image cache write-back behavior.
  # Omit entirely to accept the default (write-back enabled).
  cacheWriteBack: true

nodeEventWebhooks:
  # Endpoints that receive node health events as signed CloudEvents.
  - url: https://alerts.example.com/fuzzball
    secret: <shared secret>

definitions:
  # Array of node provisioner definitions
  # For example:
  - id: compute-nodes
    provisioner: static
    # and more provisioner-specific configuration ...

priceAdjustments:
  # Per-organization markups and discounts over the cluster's list rates
  # For example:
  - organization: 3f2b8c1e-9d4a-4f21-8f0e-2c7b6a1d5e93
    multiplier: 1.2

Per-organization markups and discounts applied to node provisioner prices. The effective hourly price of a definition is its list rate multiplied by the organization’s multiplier, and that one number is used everywhere: scheduler placement, workflow cost estimates, the prices the organization sees when it lists node provisioners, and its charges.

An organization with no entry pays the list rate.

Example:

priceAdjustments:
  # A 20% markup on everything, with a negotiated discount on one GPU type
  - organization: 3f2b8c1e-9d4a-4f21-8f0e-2c7b6a1d5e93
    multiplier: 1.2
    definitions:
      aws-g5.xlarge: 0.9
  # A flat 15% discount
  - organization: 8c41f0aa-2b57-4c93-9e18-6d0a4f2b7c31
    multiplier: 0.85

Keys under definitions are the definition IDs the cluster actually runs, after AWS instance type expansion. A definition declared as aws-${spec.instanceType} with instanceType: "g5.*" produces IDs such as aws-g5.xlarge, and that expanded form is what belongs here – the authored template never appears as a key.

List the real IDs with:

fuzzball node provisioner list

A key that names no configured definition is rejected when the configuration is set, so a typo fails loudly instead of quietly billing that definition at the list rate.

Because IDs are per definition rather than per instance type, a single instance type can span several of them where a cluster declares -spot or -gpu variants alongside the plain form. Each needs its own entry; the organization’s multiplier covers everything not named.

A multiplier applies to workflows that start after it is set. Charges already recorded keep the rate they were placed at, so changing a markup never rewrites an invoice that has already been issued.

Removing adjustments follows the shape of the configuration document:

• Omitting priceAdjustments entirely leaves the stored adjustments unchanged.
• Setting priceAdjustments: [] clears every adjustment.
• Listing some organizations updates those and leaves organizations the document does not mention alone.

Omitting the field is deliberately not the same as clearing it. A misspelled priceAdjustments key is silently ignored when the configuration is parsed, and if absence meant “remove”, one typo would drop every organization back to list prices with no error. Clear adjustments with an explicit empty list.

Prices are reported to each organization as that organization pays them, including the --max-cost-per-hour filter and --order cost sort on fuzzball node provisioner list.

Cluster admins can read any organization’s effective prices, and additionally see the cluster’s pre-adjustment list rate:

fuzzball node provisioner list --organization 3f2b8c1e-9d4a-4f21-8f0e-2c7b6a1d5e93

Members of an organization always see their own prices, whatever they pass to --organization, and never see the list rate behind them.

Price adjustments apply to compute. Storage, egress, and object cache are charged at their configured rates.

Global annotations applied to all cluster nodes.

Example:

nodeAnnotations:
  cluster.name: "hpc-cluster-01"
  datacenter: "us-west-2"
  environment: "production"
  cost.center: "research"

Software license token limits for concurrent usage control.

Software tokens are currently on the roadmap but not yet implemented.

Example:

softwareTokens:
  matlab: 25
  ansys: 15
  comsol: 8
  abaqus: 10

scheduler:
  # maximum number of requests in queue processed by scheduling iteration
  queueDepth: 64
  # how often the scheduler processes the queue
  interval: 60s
  # Cluster-wide expression (expr-lang) computing each allocation's scheduling
  # priority every tick; a higher value is scheduled sooner. When unset, the
  # default sums the organization, account, user, and workflow priority inputs
  # and ages each allocation by one unit per hour spent in the queue.
  priority: "organization.priority + account.priority + user.priority + workflow.priority"
  # Preemption: allow a blocked higher-priority allocation to evict a
  # preemptible, lower-priority running allocation. Disabled by default.
  # Evictions only happen when they make the blocked allocation placeable.
  preemptionEnabled: false
  # Pool utilization percentage (0-100) at or above which preemption may
  # evict; below it the blocked allocation is served by free or newly
  # provisioned capacity instead.
  preemptionThreshold: 75
  preemptionGap: 10
  minPreemptionRuntime: 60s
  # Allow evicting a set of victims for one blocked allocation in a single
  # scheduler pass (instead of at most one). Disabled by default.
  preemptionMultiVictimEnabled: false
  maxEvictionsPerTick: 8
  preemptionDrainTimeout: 30s
  # One-level (EASY-style) backfill: let lower-priority allocations fill a gap
  # behind a blocked head allocation without delaying it. Enabled by default;
  # set to false to schedule each node pool strictly in priority order — a
  # blocked allocation then stops lower-priority work on the same pool for
  # that pass.
  backfillEnabled: true
  # How long a running internal allocation (image or data fetch) is expected
  # to hold its resources, used by the backfill availability estimate in
  # place of the internal job's TTL. Estimate only: it never terminates a
  # fetch that runs longer.
  internalJobReleaseEstimate: 10m
  # Federate deployments only: when auto-routing a workflow submission,
  # discount each orchestrate cluster's score by how much of its ready queue
  # outranks the submission on that cluster's own priority scale. Disabled by
  # default; routing then picks purely by fit score and data locality. Set in
  # the FEDERATE cluster's central config.
  federationPriorityRoutingEnabled: false
  # Blocking ratio (outranking ready allocations per usable node) at which a
  # submission's routing score for a cluster halves.
  federationRoutingCongestionThreshold: 1.0
  # Annotation keys that scheduler annotation matching should skip when
  # comparing job annotations against each candidate definition's
  # Resource.Annotations. See "Scheduler annotation matching" below.
  ignoredAnnotations:
    - nodepool

A node provisioner has two scoring knobs that determine its fitness for a given workflow job:

• definition.annotations — values matched key-by-key against the job’s annotations by scheduler annotation matching, with built-in matchers per key (string equality by default; the GPU dimensions use substring or numeric-minimum matchers; see Built-in matchers below).
• definition.policy — an Expr expression evaluated against the job’s request; returns a boolean that gates eligibility.

The two knobs are independent: scheduler annotation matching does not look at definition.policy, and policy evaluation does not look at definition.annotations. Either can route on the same annotation key; a deployment is free to use one, the other, or both.

By default every annotation key on a workflow job must be matched by an entry in definition.annotations on each candidate definition; otherwise the candidate is rejected for that job. The platform’s own annotation keys — a specific enumerated set of fuzzball.io/* keys including workflow/job/account identifiers (fuzzball.io/workflow.id, fuzzball.io/job.name, …) and the provisioner-definition pinning key — are skipped automatically, as is the dynamically keyed fuzzball.io/connect/<service> namespace (the client-side connect command, which cannot be enumerated as fixed keys). Apart from that namespace this is an allowlist, not a prefix match: the fuzzball.io/ prefix is reserved for the platform, and any user-defined key placed under that prefix is not auto-exempt and will still need to be added to scheduler.ignoredAnnotations. Cluster admins should declare only deployment-specific keys (their own labels for routing, etc.) in that list.

Add an annotation key to ignoredAnnotations when routing for that key is handled by definition.policy rather than by definition.annotations. Without the entry, scheduler annotation matching would additionally require the key on every candidate definition’s annotations map, redundant with what the policy already evaluates.

For example, a deployment whose node provisioners look like

definitions:
  - id: pool-small
    provisioner: pbs
    policy: |-
      request.job_annotations["nodepool"] in ["pbs-small", "small", ""]
    # ... no definition.annotations["nodepool"] needed; the policy handles it

should set:

scheduler:
  ignoredAnnotations:
    - nodepool

so that a workflow with resource.annotations.nodepool: small reaches the policy without being rejected by scheduler annotation matching first.

Scheduler annotation matching uses string equality (ExactMatch) by default. The following GPU dimensions have built-in non-exact matchers:

Fuzzball derives a reliability score from each node’s health conditions and the job failures attributed to it — see Node Health Monitoring. The nodeHealth section tunes how that score is calculated. Omit it entirely to accept the defaults.

The score is shown by fuzzball node list, fuzzball node show, the API, the web UI and Prometheus. Whether automation acts on it is set by mode, which defaults to observe — Fuzzball records what it would have done and takes no action. Raise mode to let automation cordon, drain, or replace.

drainBelow has one effect that does not depend on mode: the scheduler will not place new work on a node scoring below it, in any mode. Work already running there is left alone. Tuning this value therefore changes scheduling even on a cluster that has never enabled a policy — see Nodes below the drain threshold.

nodeHealth:
  # Points deducted while each condition is active.
  penalties:
    MEMORY_ERRORS: 45
    MACHINE_CHECK_ERRORS: 45
    DISK_DEGRADED: 40
    THERMAL_THROTTLE: 30
    AGENT_UNHEALTHY: 15
  # Score thresholds for each rung.
  degradedBelow: 95
  cordonBelow: 65
  drainBelow: 45
  # Attributed job failures.
  failureTallyPenalty: 5
  failureTallyCap: 25
  failureTallyHalfLifeHours: 168
  # How far automation may act on the score.
  mode: observe
  maxUnavailablePercent: 20
  cordonSuppressionMinutes: 30
  maxReplacementsPerHour: 3
  # Fault classes that evacuate a node the moment they are raised.
  evictOnFaultClasses:
    - GPU_ECC_ERRORS
  maxEvictionRestarts: 3

Penalties apply only while a condition is active, so clearing one restores its points on the node’s next health report. Attributed failures decay instead of clearing. Both paths, and the way an uncordon deliberately does not reset the score, are described in How the score recovers.

mode is observe unless you change it, which means Fuzzball records what it would have done and takes no action. See Automated Response for what each mode does and why observe is the default.

A node provisioner definition’s mode replaces the cluster’s rather than merging with it, so a GPU pool can enforce while the rest of the cluster observes.

An unrecognised mode is rejected when the configuration is validated rather than being treated as observe. A typo here is most likely someone trying to enable enforcement, and silently leaving it off would look identical to the policy simply never firing.

A condition’s penalty depends on its severity as well as its name. Correctable memory errors climbing and a single uncorrectable error are both MEMORY_ERRORS, but the first is a warning and the second is critical. A penalties entry overrides both severities for that condition.

COLLECTOR_FAILED defaults to zero: a broken sensor is missing information rather than fault evidence, so it is recorded on the node but does not move the score by default. Give it a penalty if you would rather a node with unreadable sensors score lower.

A penalty here lowers the score but does not make automation act on the node. A node with a failed collector is left alone whatever its score, exactly like a node whose telemetry has gone stale — see Guardrails. No policy action is taken and no policy event is emitted; the node is held at Unknown with the condition attached, for an operator to judge.

HEARTBEAT_MISSED and TELEMETRY_STALE never contribute to the score. A penalties entry for either is rejected when the configuration is validated, rather than accepted and then ignored. A node Fuzzball cannot see is recorded as unknown rather than scored as broken — otherwise a network interruption would look like failing hardware and could drain healthy capacity.

A node provisioner definition may set its own nodeHealth block, which replaces the cluster-wide one for the nodes that definition provisions. Use it where a pool’s hardware or risk tolerance differs from the rest of the cluster:

nodeHealth:
  cordonBelow: 65

definitions:
  - id: gpu-pool
    provisioner: aws
    # This pool reacts sooner than the rest of the cluster.
    nodeHealth:
      cordonBelow: 80
      drainBelow: 60
    provisionerSpec:
      instanceType: p5.48xlarge

Resolution is per block, not per field: a definition that sets any nodeHealth value owns scoring for its nodes outright and inherits nothing from the cluster block. Within a block, individual fields still fall back to their defaults, so a definition can override one threshold without restating the rest.

Controls how the workflow pipeline handles container image caching. Omit this section entirely to accept the defaults.

image:
  # Whether to upload converted SIF images to the object cache after a
  # cache-miss pull. Enabled by default.
  cacheWriteBack: true

Disable write-back on deployments where the network link between substrate nodes and the object cache is bandwidth-constrained — for example, segmented deployments where GPU nodes connect to the orchestrate cluster over a Tailscale or WireGuard tunnel. On such links, uploading a large SIF (8+ GB for ML framework images) after conversion can take hours and block the image stage, delaying job start.

With write-back disabled, each node converts OCI images locally on every cache miss. This trades redundant conversion work for faster job start times on slow links. On deployments with fast links between nodes and the object cache, leave write-back enabled (the default) — the one-time upload cost is repaid on every subsequent pull.

Disabling write-back does not affect images already in the object cache. A previously cached image is still served from the cache on a cache hit, regardless of this setting. Only the upload-after-conversion step on cache misses is skipped.

Federate deployments: the object cache feeds the data-locality bonus used by federate workflow routing. When write-back is disabled, newly pulled images do not create cache refs, so they stop contributing to the cluster’s locality score. On a federate deployment, disabling write-back on an orchestrate cluster reduces its attractiveness for workflows that reference images it has converted but not cached. If data locality matters for routing decisions, weigh this against the bandwidth savings before disabling write-back.

Example — segmented deployment with remote GPU nodes:

image:
  cacheWriteBack: false

Endpoints that receive node health events as signed CloudEvents. See Webhook Notifications for the payload shape and how to verify a signature.

nodeEventWebhooks:
  - url: https://alerts.example.com/fuzzball
    secret: <shared secret>
    events:
      - POLICY_ACTED
      - POLICY_HALTED
      - NODE_REPLACED
    timeoutSeconds: 5
    maxAttempts: 3

Unlike nodeHealth, this is cluster-scoped only. A definition’s nodeHealth block replaces the cluster’s outright, so webhooks living inside it would be silently dropped by any definition that tuned a single penalty — and losing notifications by accident is the failure this exists to prevent.

secret is required. Every payload names a node and the fault taking it out of service, and a receiver with no way to tell a real delivery from a forged one cannot safely act on that. An endpoint configured without one is rejected when the configuration is validated.

Bounds where cluster members may point user-created hostpath and NFS storage provisioners. When enabled, the privileged drivers (hostpath, nfs) become cluster-admin only, and any allowlists below constrain where those admins may still point them. Multi-tenant deployments turn this on automatically; single-tenant deployments leave it off unless an operator opts in for the same guardrails. Distinct from defaultStorage, which describes the provisioner Fuzzball creates itself — see the storage configuration guide.

storageRestrictions:
  enabled: true
  hostPathRoots:
    - /mnt/fuzzball
  nfsServers:
    - nfs-prod.internal
    - nfs-scratch.internal

The zero value imposes no restrictions. On operator-managed Kubernetes deployments, the operator also populates hostPathRoots with the shared hostpath location it configures for the cluster.

definitions:
  - id: p5.48xlarge
    provisioner: aws
    # Automatically run hardware discovery for this definition after a
    # configuration update. Disabled by default; discovery can always be
    # triggered manually with `fuzzball node provisioner discover`.
    autoDiscover: true
    provisionerSpec:
      instanceType: p5.48xlarge

Each entry in the definitions array is a node provisioner: a configuration that, for a chosen backend, tells Fuzzball how to obtain a class of functionally-identical compute nodes with policy attached. The serialized form of a node provisioner is referred to as a node provisioner definition.

These parameters are available for node provisioners across all backends:

ttl and ttlBuffer are both uint32 values. When a non-static provisioner definition has ttlBuffer > 0 and a non-zero allocation TTL is being set for a node, the scheduler adds ttlBuffer × nodeCount to the allocation TTL before submitting the provisioning request. Both the multiplication and addition use saturating arithmetic: if either result would exceed 4,294,967,295 (roughly 136 years), it is clamped to that value rather than wrapping around. This prevents misconfigured large values from silently producing a much shorter TTL and causing nodes to be terminated before jobs complete.

This formula does not apply to static provisioners, or when either the allocation TTL or ttlBuffer is 0.

The exclusive parameter controls how nodes provisioned by this definition are shared among jobs:

• If not specified or empty, nodes are shared and can run multiple jobs simultaneously. Multiple jobs from the same or different workflows can be scheduled on the same node based on available resources.

• job: Nodes are exclusive to a single job allocation. Once a job is assigned to the node, no other jobs can use it until the job completes and the node is cleaned up. This ensures complete isolation at the job level.

• workflow: Nodes are exclusive to a single workflow. All jobs within the same workflow can share the node, but jobs from other workflows cannot use it. This is useful for workflows that need dedicated resources but want to share nodes across their jobs.

Example:

definitions:
  # Shared nodes for general workloads
  - id: shared-compute
    provisioner: static
    exclusive: none
    provisionerSpec:
      condition: hostname() matches "shared-[0-9]+"

  # Job-exclusive nodes for sensitive workloads
  - id: exclusive-compute
    provisioner: pbs
    exclusive: job
    ttl: 3600
    provisionerSpec:
      cpu: 8
      memory: "32GiB"
      queue: "workq"

The maxNodes parameter caps the size of a dynamic definition’s node pool: the number of nodes provisioned for that definition at any one time.

• Pool cap: The pool is capped at min(maxNodes, 128). The cap applies to dynamic definitions only; static definitions are not affected.
• Blocking: When the pool is at the cap, an allocation that fits within the cap waits for an existing node to become free (node reuse) instead of provisioning new nodes. The scheduler publishes a provisioning_blocked workflow event (carrying definition_id, total_nodes, and max_nodes attributes) and a generic scheduling_blocked event, once per blocked allocation. See Troubleshooting scheduling_blocked events.
• Rejection: A multi-node job that alone requires more nodes than the definition allows fails at workflow submission. Task arrays are not rejected; their concurrent width is clamped instead.
• Unset maxNodes: If maxNodes is not configured, the pool can grow to the 128 backstop, but a single allocation is still limited to the default of 16 nodes.
• Visibility: The definition’s maxNodes value appears in fuzzball node provisioner get output and as Max Nodes in the web UI’s provisioner details (a dynamic definition without an explicit value reports the default of 16; an uncapped static definition omits the field).

Example:

definitions:
  # Pool capped at 32 nodes
  - id: small-pool
    provisioner: pbs
    ttl: 3600 # node lifetime set to 1h
    maxNodes: 32
    provisionerSpec:
      cpu: 8
      memory: "32GiB"
      queue: "workq"

  # Pool capped at the 128 backstop (maxNodes not set)
  - id: large-pool
    provisioner: slurm
    ttl: 7200 # node lifetime set to 2h
    provisionerSpec:
      cpu: 16
      memory: "64GiB"
      partition: "compute"

In the first example, once 32 small-pool nodes are provisioned, further allocations wait for a node to free up. In the second example, the pool can grow up to 128 nodes before blocking.

Every dynamic backend provisions asynchronously. The provisioning request returns as soon as the backend accepts it, long before the nodes have finished booting and installing the Fuzzball Substrate. If that bootstrap then fails — a cloud-init error, a package install failure, an NFS mount that never comes up — a node never registers with the scheduler, and nothing in the backend reports a problem. registrationDeadline bounds how long the scheduler waits for those nodes before it gives up.

• Clock start: The deadline runs from the moment the provisioning request returns, not from when the job entered the Fuzzball queue. An allocation can sit in the queue for a long time before it provisions (behind the definition’s maxNodes cap, higher-priority work ahead of it, or overall queue depth), and that wait does not count against the deadline.
• Expiry behavior: The scheduler cordons any nodes that did register, deletes every instance created by the provisioning request, then fails the workflow with an error naming how many nodes registered out of how many were requested. Expiry is terminal: the workflow does not retry and must be resubmitted.
• Partial registration: A request for three nodes where only one node registered expires exactly as a request where none registered. Once every node in a request has registered, the deadline no longer applies to it — a node lost after that point is handled as a node failure, not as a registration timeout.
• Check frequency: The scheduler evaluates deadlines every 30 seconds, so expiry is detected shortly after the deadline passes.
• Static definitions: These never issue a provisioning request, so a value set on a static definition is ignored rather than rejected.

On slurm and pbs definitions the provisioning request returns when the batch job is submitted with sbatch or qsub, not when it starts running. Time the batch job spends queued in Slurm or PBS therefore counts against registrationDeadline. On a busy backend where jobs routinely wait longer than the 15m default, raise registrationDeadline on those definitions — otherwise the scheduler fails the workflow of a legitimately queued batch job while that job is still waiting for backend resources. Cleanup signals the batch job rather than deleting it, and a job that has not started yet rejects the signal: it is left in the backend queue and must be removed with scancel or qdel.

Write the value as a duration with a unit suffix. An unsuffixed integer (900), a negative value (-5m), or anything below 1s is rejected when the configuration is loaded, rather than clamped or silently replaced with the default. Omit the field to use the 15m default.

Example:

definitions:
  # Large GPU images in a slow region: allow longer than the 15m default.
  - id: gpu-pool
    provisioner: aws
    ttl: 7200
    registrationDeadline: 30m
    provisionerSpec:
      instanceType: p5.48xlarge

  # A busy Slurm partition: the deadline must cover backend queue wait.
  - id: batch-pool
    provisioner: slurm
    ttl: 14400
    registrationDeadline: 2h
    provisionerSpec:
      cpu: 16
      memory: "64GiB"
      partition: "compute"

To diagnose a workflow that failed this way, see Troubleshooting provisioning_registration_timeout events.

Static provisioners manage physical or pre-allocated compute resources.

The condition field supports these built-in variables and functions:

# NVIDIA GPU (any model)
modalias.match("pci:v000010DEd*sv*sd*bc03sc*i*")

# Specific NVIDIA GPU models
modalias.match("pci:v000010DEd00001B06sv*sd*bc03sc*i*")  # GTX 1080 Ti
modalias.match("pci:v000010DEd00001E07sv*sd*bc03sc*i*")  # RTX 2080 Ti

# Intel Ethernet controllers
modalias.match("pci:v00008086d*sv*sd*bc02sc00i*")

# Mellanox InfiniBand adapters
modalias.match("pci:v000015B3d*sv*sd*bc0Csc06i*")

You can also easily get the modalias for all the PCI devices on a node to match a specific device with the following one-liner:

$ IFS=$'\n'; for d in $(lspci); do modalias=$(cat /sys/bus/pci/devices/0000\:${d%% *}/modalias); echo "$modalias -> ${d#* }"; done

pci:v00008086d00004641sv00001D05sd00001174bc06sc00i00 -> Host bridge: Intel Corporation 12th Gen Core Processor Host Bridge/DRAM Registers (rev 02)
pci:v00008086d0000460Dsv00000000sd00000000bc06sc04i00 -> PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 (rev 02)
pci:v00008086d000046A6sv00001D05sd00001174bc03sc00i00 -> VGA compatible controller: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] (rev 0c)
[snip...]

definitions:
  # Basic compute nodes
  - id: compute-standard
    provisioner: static
    provisionerSpec:
      condition: |-
        hostname() matches "compute-[0-9]{3}" &&
        cpuinfo.vendor_id == "GenuineIntel" &&
        cpuinfo.cpu_cores >= 16
      costPerHour: 0.40

  # GPU nodes
  - id: gpu-nodes
    provisioner: static
    provisionerSpec:
      condition: |-
        hostname() matches "gpu-[0-9]+" &&
        modalias.match("pci:v000010DEd*sv*sd*bc03sc*i*")
      costPerHour: 2.50

  # High-memory nodes
  - id: highmem-nodes
    provisioner: static
    provisionerSpec:
      condition: |-
        hostname() matches "mem-[0-9]+" &&
        cpuinfo.cpu_cores >= 64
      costPerHour: 1.75

condition: |-
  osrelease.id == "ubuntu" &&
  osrelease.version_id >= "20.04"

condition: |-
  uname.machine == "x86_64" &&
  cpuinfo.vendor_id == "GenuineIntel" &&
  cpuinfo.cpu_cores >= 16

condition: |-
  let compute_regex = "compute-[0-9]{3}";
  let gpu_regex = "gpu-[0-9]{2}";
  hostname() matches compute_regex || hostname() matches gpu_regex

condition: |-
  // Match NVIDIA GPU devices
  modalias.match("pci:v000010DEd*sv*sd*bc03sc*i*") &&
  cpuinfo.cpu_cores >= 8

condition: |-
  let is_compute_node = hostname() matches "compute-[0-9]+";
  let is_intel_cpu = cpuinfo.vendor_id == "GenuineIntel";
  let is_ubuntu = osrelease.id == "ubuntu";
  let has_enough_cores = cpuinfo.cpu_cores >= 16;

  is_compute_node && is_intel_cpu && is_ubuntu && has_enough_cores

AWS provisioners support dynamic EC2 instance provisioning.

AWS provisioners support wildcard patterns that automatically expand to individual instance types:

• t3.* expands to t3.nano, t3.micro, t3.small, etc.
• c5.* expands to c5.large, c5.xlarge, c5.2xlarge, etc.
• p3.* expands to p3.2xlarge, p3.8xlarge, p3.16xlarge

When using wildcards, the ${spec.instanceType} placeholder in the definition ID is replaced with the actual instance type.

definitions:
  # Spot instances for cost optimization
  - id: aws-${spec.instanceType}-spot
    provisioner: aws
    provisionerSpec:
      instanceType: t3.*
      spot: true
    policy: |-
      request.job_ttl <= 3600

  # On-demand compute instances
  - id: aws-${spec.instanceType}
    provisioner: aws
    provisionerSpec:
      instanceType: c5.*
      spot: false
    policy: |-
      request.job_kind == "service"

  # GPU instances for ML workloads
  - id: aws-${spec.instanceType}-gpu
    provisioner: aws
    provisionerSpec:
      instanceType: p3.*
      spot: false
    policy: |-
      request.job_resource.devices["nvidia.com/gpu"] > 0

Slurm provisioners integrate with existing Slurm clusters.

definitions:
  # Standard compute partition
  - id: slurm-compute
    provisioner: slurm
    ttl: 86400 # node lifetime set to 24h
    provisionerSpec:
      costPerHour: 0.30
      cpu: 16
      memory: "64GiB"
      partition: "compute"
    policy: |-
      request.job_resource.cpu.cores <= 16

  # GPU partition
  - id: slurm-gpu
    provisioner: slurm
    ttl: 43200 # node lifetime set to 12h
    provisionerSpec:
      costPerHour: 1.80
      cpu: 8
      memory: "32GiB"
      partition: "gpu"
    policy: |-
      request.job_resource.devices["nvidia.com/gpu"] > 0

PBS provisioners integrate with OpenPBS/PBS Pro clusters.

definitions:
  # Standard PBS queue
  - id: pbs-compute
    provisioner: pbs
    ttl: 86400 # node lifetime set to 24h
    provisionerSpec:
      cpu: 8
      memory: "32GiB"
      gpus: 0
      queue: "workq"
      costPerHour: 0.30

  # GPU queue
  - id: pbs-gpu
    provisioner: pbs
    ttl: 86400 # node lifetime set to 24h
    provisionerSpec:
      cpu: 4
      memory: "16GiB"
      gpus: 1
      queue: "gpu"
      costPerHour: 1.00

CoreWeave provisioners support dynamic instance provisioning on CoreWeave’s cloud infrastructure, optimized for GPU workloads.

definitions:
  # CPU instance for standard compute workloads
  - id: coreweave-cpu-small
    provisioner: coreweave
    provisionerSpec:
      instanceType: "cd-gp-a192-genoa"
      costPerHour: 7.78
    policy: |-
      request.job_resource.cpu.cores >= 4 &&
      request.job_resource.cpu.cores <= 32

  # GPU instance for ML/AI workloads
  - id: coreweave-gpu-a40
    provisioner: coreweave
    ttl: 3600
    provisionerSpec:
      instanceType: "cd-a40-24gb"
      costPerHour: 15.00
    policy: |-
      request.job_resource.devices["nvidia.com/gpu"] > 0

CoreWeave provisioners support both dynamic provisioning (on-demand node creation) and static provisioning (pre-existing node pools). For static provisioning with pre-existing CoreWeave node pools, see the CoreWeave Static Provisioning guide.

OCI provisioners support dynamic instance provisioning on Oracle Cloud Infrastructure.

definitions:
  # CPU instance for standard compute workloads
  - id: oci-cpu-standard
    provisioner: oci
    provisionerSpec:
      shape: "VM.Standard.E4.Flex:8:64"
    policy: |-
      request.job_resource.cpu.cores >= 4 &&
      request.job_resource.cpu.cores <= 8

  # GPU instance
  - id: oci-gpu-a10
    provisioner: oci
    ttl: 3600
    provisionerSpec:
      shape: "VM.GPU.A10.1"
    policy: |-
      request.job_resource.devices["nvidia.com/gpu"] > 0

The OCI pricing API does not publish per-GPU rates, so cost estimates for GPU shapes always use a fallback per-GPU hourly rate: the built-in default of $3.00/GPU/hr, or the defaultGPUHourlyRate value from the orchestrator’s OCI provisioner configuration when set.

Policy expressions control access to node provisioners and use the same expression language as static conditions.

Policies apply to every allocation, including the internal jobs Fuzzball creates implicitly for container image pulls and data staging (request.job_kind == "internal"). Internal jobs have a safety net user jobs do not: if every definition’s policy rejects an internal job, the scheduler falls back to placing it as if no policies were configured — a policy can never make image pulls or data staging unschedulable. A policy expression that fails to evaluate for an internal job is treated as rejecting that definition rather than failing the workflow; if no definition passes as a result, the same fallback applies — and may still place the internal job on the definition whose policy could not be evaluated. Each time the fallback places an internal job, the orchestrator logs a warning naming the allocation and the definition chosen — if internal jobs are meant to be routed somewhere specific (for example, image pulls to a data transfer node), this warning is the signal that no definition’s policy admits them and the routing policies need attention.

policy: |-
  request.owner.organization_id == "research" &&
  request.owner.account_id in ["2f0a8f4e-0a16-47d5-b541-05d3f9f44910", "c602cf05-7604-4f11-a690-79552b1fdbdd"]

policy: |-
  request.job_resource.cpu.cores <= 32 &&
  request.job_resource.mem.bytes <= (256 * 1024 * 1024 * 1024) &&
  !request.job_resource.exclusive

policy: |-
  request.job_kind == "job" &&
  request.job_ttl >= 300 &&
  request.job_ttl <= 86400

policy: |-
  let gpu_count = request.job_resource.devices["nvidia.com/gpu"];
  gpu_count > 0 && gpu_count <= 4 &&
  request.owner.organization_id == "280abb59-b765-4cdd-a538-6ab8f9b7927c"

Reject multi-node or task-array jobs that can never be satisfied by this definition’s pool. Because definition.nodes counts a static pool’s usable nodes even while they are fully occupied, a busy pool keeps accepting jobs (they queue until nodes drain) instead of being rejected at its busiest. definition.nodes never exceeds definition.max_nodes — an explicitly configured maxNodes already caps it — so gating on definition.nodes alone is sufficient.

policy: |-
  request.multinode_nodes <= definition.nodes &&
  request.task_array_concurrency <= definition.nodes

Route container image pulls and data staging to a dedicated transfer node and keep them off the compute nodes, so user workloads do not compete with network-heavy transfers:

definitions:
  - id: dtn
    provisioner: static
    provisionerSpec:
      condition: hostname() == "fz-dtn01"
    policy: request.job_kind == "internal"
  - id: compute
    provisioner: static
    provisionerSpec:
      condition: hostname() matches "fz-cmp0[1-3]"
    policy: request.job_kind != "internal"

A definition without a policy accepts every job kind, so the attract policy on the transfer node definition is not enough by itself: every other definition needs the repel policy (request.job_kind != "internal"), or internal jobs remain eligible there and may still be placed on compute nodes by resource and cost scoring.

Pulled images are shared per storage segment, so the transfer node must be in the same segment as the compute nodes that run the jobs — otherwise those jobs cannot find the pulled image. If the transfer node pool has no usable nodes (for example, the node is down), internal jobs wait for it like any policy-gated job; they do not spill onto the compute definitions.

policy: |-
  request.job_annotations["priority"] == "high" &&
  request.job_annotations["project"] in ["proj-a", "proj-b"] &&
  request.owner.email matches "*@ciq.com"

policy: |-
  request.multinode_job ?
  request.job_resource.cpu.cores >= 4 &&
  request.owner.account_id == "092403fe-12ef-4465-bce4-18292fec13c8"
  :
  request.job_resource.cpu.cores <= 16

policy: |-
  let current_hour = time.Now().Hour();
  let is_business_hours = current_hour >= 9 && current_hour <= 17;

  request.job_annotations["priority"] == "low" ? !is_business_hours : true