Scheduling Priority and Preemption in Federate

Federate deployments use the same scheduling priority and preemption features as standalone Fuzzball clusters, with one key difference: the scheduling attributes are managed centrally on the Federate cluster and replicate automatically to every Orchestrate cluster where scheduling actually happens. This page explains where to set those attributes, how they propagate, and what to configure on each cluster for preemption to work.

For organizations whose users log in through Federate, the Federate cluster is the source of truth for all scheduling attributes. Administrators set them with the same commands used on a standalone cluster, pointed at the Federate API:

# Set an organization's scheduling priority (cluster administrator only)
$ fuzzball organization update 992e9e64-e34f-4d84-bb4e-773c40b7c6b4 --priority 5

# Set a group's scheduling priority (organization owner)
$ fuzzball group update research --priority 10

# Set a user's scheduling priority (organization owner)
$ fuzzball user update 5b7222b3-c99a-4b3b-966e-a2ba69bcc93a --priority 2

# Allow the research group's workflows to preempt the batch group's workflows
$ fuzzball group update research --preempt batch

# Revoke that permission again
$ fuzzball group update research --no-preempt batch

The group update command accepts a group name or ID; organization update and user update require the entity’s ID, which you can find with fuzzball organization list and fuzzball user list.

Manage these values only on the Federate cluster. Each Orchestrate cluster holds a replica of every federated organization, and the replica is refreshed from the Federate cluster’s values. Treat the replicas as read-only: a change made directly on an Orchestrate cluster is overwritten by the next refresh.

Each Orchestrate cluster keeps a local replica of every federated organization, group, and user that interacts with it. Whenever a request from Federate reaches an Orchestrate cluster – most commonly a workflow submission and its routing step, but also storage and volume operations and switching to a group – that cluster refreshes its replica from the Federate cluster: the priorities of the requesting user’s organization, group, and user account, and the group’s preemption list.

Two properties of this design are worth understanding:

• Changes take effect on the next request. Editing a priority or preemption list on the Federate cluster does not push the change anywhere immediately. Each Orchestrate cluster picks up the change with the next request that carries the affected entity: an organization priority arrives with the next request by any member of the organization, a group priority or preemption list with the next request by a member of that group, and a user priority with the next request by that user. In practice this means a workflow submission always carries the submitting user’s latest attributes with it. Workflows submitted earlier keep the preemption list they were submitted with, matching standalone behavior. Priority changes additionally update allocations still waiting in that cluster’s queue.
• Preemption targets are created on demand. A group named in another group’s preemption list is created on the Orchestrate cluster automatically, even if none of its own members has ever submitted a workflow there. No administrator action is needed on the Orchestrate side.

Replicated priorities feed both stages of scheduling in a federated deployment:

• Routing. When federationPriorityRoutingEnabled is set in the Federate cluster’s central config, each candidate Orchestrate cluster computes the submission’s priority score using its own scheduler.priority expression – with the replicated organization, group, and user priorities as inputs – and reports how many allocations already waiting in its ready queue score higher. Higher-priority work therefore routes into busy clusters where it would be scheduled ahead of the existing queue. See the Configuration Reference for details.
• In-cluster scheduling. Once a workflow lands on an Orchestrate cluster, its allocations are ordered by that cluster’s scheduler.priority expression exactly as on a standalone cluster, with the replicated priorities as inputs.

Because each Orchestrate cluster applies its own scheduler.priority expression, clusters configured with different expressions can weigh the same inputs differently. Keep the expression consistent across clusters unless you specifically want per-cluster behavior.

Preemption lets a blocked higher-priority workload evict a running lower-priority one (the victim) to take its resources. In a federated deployment it is evaluated and enforced independently by each Orchestrate cluster. For a federated group’s workflows to preempt another group’s workflows on a given cluster, all of the following must hold:

1. Preemption is enabled on that Orchestrate cluster. Set preemptionEnabled: true under scheduler: in the Orchestrate cluster’s central config. This is a per-cluster setting; enable it on every cluster where preemption should operate.
2. The preempting group opts in on the Federate cluster. The victim’s group must appear in the preempting group’s preemption list (fuzzball group update <group> --preempt <victim-group>).
3. The victim workload is preemptible. Only workloads marked preemptible can be evicted: set the top-level preemptible: true Fuzzfile field, pass fuzzball workflow start --preemptible, or set preemptible: true on an individual job or service (the per-job value overrides the workflow-level setting).
4. The cluster’s preemption thresholds are met. The preemptionThreshold, preemptionGap, and minPreemptionRuntime settings of the Orchestrate cluster apply unchanged; they are described in the Configuration Reference. In particular, the effective-priority gap between the blocked and running allocation must be at least preemptionGap (default 10). Running allocations gain priority with age, so configure the preempting group’s priority delta well above the gap – a delta equal to the gap misses once the victim has aged.

Preemptions are recorded as stage events on the affected workflow. Users and administrators can see them with fuzzball workflow events <workflow-id>, including which workflow triggered the eviction and on which cluster it happened.

Scheduling attribute replication requires both the Federate cluster and the Orchestrate clusters to run a release that includes this feature; check the release notes for your installed versions. With an older Federate cluster, Orchestrate clusters fall back to the previous behavior: replicas that have never synced keep default priority 0 and empty preemption lists, so admin-raised priorities and group preemption do not take effect for federated submissions until the Federate cluster is upgraded. Values that replicated before a downgrade persist unchanged; they are never reset.