Priority, Preemption, and Backfill

The scheduler ranks queued work by an effective priority, can preempt running preemptible work in favor of blocked higher-priority work, and backfills idle capacity with short jobs while a large job waits. This page covers how to configure and operate these features; the underlying configuration knobs are described in the central configuration reference, and the commands for inspecting the queue are covered in Queue and Scheduling Observability.

Each allocation’s priority is recomputed on every scheduling pass from a cluster-wide expression (scheduler.priority in the central configuration). When the expression is unset, the default is the sum of the organization, account, user, and workflow priority inputs, plus one priority unit for every hour the allocation has waited in the queue (aging), so long-queued work rises over time.

The inputs are set at different levels:

Cluster admins can list every organization in the cluster with fuzzball organization list to find the organization ID to update. The --priority value on workflow start is signed and must be less than or equal to zero (default 0): users can deprioritize their own workflows but can never raise them above their entitlement.

Priority changes to an organization, group, or user propagate to that entity’s queued and running allocations, not just new submissions.

• fuzzball group list, fuzzball org member list, and fuzzball group member list include a Priority column.
• fuzzball workflow list can sort by priority.
• fuzzball workflow describe <WORKFLOW> <STAGE> shows the stage’s full priority breakdown.
• fuzzball workflow events records priority: Priority set (workflow: 0, organization: 5, group: 2, user: 1) on submission, and Priority updated: organization 5 -> 10 when an entity priority change is applied to an allocation.

When enabled, the preemption pass may evict a running, preemptible, lower-priority allocation to make room for a blocked higher-priority one. Preemption is disabled by default; set preemptionEnabled: true in the scheduler section of the central configuration to turn it on.

Nothing is ever evicted unless it opted in. A workload is preemptible when any of the following is set:

• preemptible: true at the top level of the workflow definition (applies to every job and service in the workflow);
• preemptible: true on an individual job or service, which overrides the workflow-level setting;
• fuzzball workflow start --preemptible.

See the workflow definition reference for the field details.

Eviction is deliberately conservative. A victim must be running, preemptible, past the anti-thrash minimum runtime (minPreemptionRuntime, default 60s), and below the blocked allocation’s priority by at least the configured gap (preemptionGap, default 10). The pass only fires when the node pool’s utilization is at or above preemptionThreshold (default 75) – below it, the blocked allocation simply waits for free or newly provisioned capacity – and a victim is only evicted when the capacity it frees actually lets the blocked allocation be placed. By default at most one victim is evicted per blocked allocation per pass; opt-in multi-victim preemption (preemptionMultiVictimEnabled, bounded by maxEvictionsPerTick) evicts a set of victims in one pass when no single victim frees enough.

Organization owners control which groups each group is allowed to preempt:

$ fuzzball group update <GROUP> --preempt <TARGET-GROUP>
$ fuzzball group update <GROUP> --no-preempt <TARGET-GROUP>

Preemption lists are empty by default – with preemption enabled but no lists configured, no group’s work preempts another group’s. Deleted groups are removed from all preemption lists automatically.

Preempted jobs and services are re-queued and restart cleanly on their next placement. fuzzball workflow events shows both sides of an eviction: Preempted by workflow X job Y on the evicted job and Preempted workflow X job Y on the preemptor. Prometheus metrics report preemption misses, time blocked before an eviction, near-miss priority gaps, and preempted-allocation outcomes.

While a high-priority allocation is blocked waiting for capacity, one-level (EASY-style) backfill lets shorter, lower-priority allocations run in the capacity it does not need yet – without delaying its predicted start time. Backfill is enabled by default (backfillEnabled: true); disabling it makes each node pool schedule strictly in priority order.

A job qualifies as a backfill candidate when its runtime is bounded by an execute timeout (policy.timeout.execute); the workflow DSL default satisfies this. Image and data staging jobs are also backfilled whenever they fit capacity the blocked allocation does not need. Backfill applies to static pools and to dynamic provisioner definitions running at their maxNodes cap.

Backfill placements are surfaced in fuzzball workflow events, and Prometheus metrics cover reservations held and the predicted wait of the blocked head job.