Workflow Priority

Fuzzball orders queued work by an effective priority the scheduler recomputes every tick. Workflows with a higher effective priority are placed first, and a workflow that has been waiting longer gains priority as it ages.

This page covers the one input a workflow submitter controls – the workflow’s own priority – how to set it at submission, and how to change it afterwards without killing and resubmitting the work.

By default a cluster computes effective priority as the sum of four priority inputs plus one priority unit for every hour the allocation has spent in the queue:

(now() - workflow.create_time).Hours()
  + organization.priority
  + account.priority
  + user.priority
  + workflow.priority

The first three inputs are set by administrators and apply to everything an organization, group, or user submits – see Scheduling Priority and Preemption. The fourth, workflow.priority, belongs to the individual workflow and is the subject of this page.

A cluster administrator can replace the expression entirely; see the scheduler.priority entry in the Configuration Reference.

fuzzball workflow start takes a --priority flag:

$ fuzzball workflow start --priority -10 sweep.fz

The value must be 0 or negative, and 0 is the default. A submitter can therefore only move their own work down relative to the baseline, never up: if raising your own priority were allowed, every submitter would raise it, and the input would stop meaning anything across tenants.

An owner of your group’s organization may submit a positive value. The same rule applies to workflow start, workflow score and workflow update, so a priority you can submit is one you can also set afterwards, and vice versa.

Because 0 is the ceiling, submit bulk or background work below the default so you keep room to outrank it later. A sweep submitted at --priority -10 is outranked by anything you submit afterwards at the default of 0. With the default priority expression, it still drains: the aging term adds one priority unit per hour, so a deprioritized workflow catches up on its own rather than starving. A cluster running a custom scheduler.priority expression only gets that guarantee if the replacement expression keeps an equivalent aging term – check with your administrator if you are unsure.

Submitting everything at the default of 0 is the case that leaves you stuck – with no headroom above, the only way to reorder is to lower the other workflows.

A submitted workflow’s priority is not frozen. fuzzball workflow update changes it in place:

$ fuzzball workflow update 3cc718d1-e35c-4455-b4f4-ca3412727277 --priority -20
Workflow 3cc718d1-e35c-4455-b4f4-ca3412727277 priority set to -20; the scheduler processes the change on its next pass

The new value is re-stamped onto every stage of the workflow that has not finished, and the scheduler applies it on its next pass – typically within a second. Stages that have already finished are left alone. Whether that actually changes queue order depends on the other work around it: an unchanged value, or a workflow with only running (not queued) stages, moves nothing.

The same ceiling applies: an ordinary submitter is held to <= 0, for the same reason as at submission. To move one of your own workflows to the front of your backlog, lower the ones it should outrank:

# 180 queued workflows, all at the default of 0, and one of them is now urgent.
# Drop the ones that can wait rather than trying to raise the urgent one.
$ fuzzball workflow update 8f2b0c14-2f57-4a1a-9f0c-3b6ad0f4c211 --priority -5
$ fuzzball workflow update b41d9e77-7c2a-4e0b-8d3f-15a9c7e2b0aa --priority -5

Setting a value above 0 requires ownership of the organization the workflow’s own group belongs to – not whichever organization your current context happens to be selected against. That is the only way to lift a single workflow above a backlog already sitting at the default, and it is recorded in the audit log along with who changed it. The identical rule governs workflow start, so an organization owner can submit at a positive priority directly rather than submitting at 0 and updating.

A federate organization owner cannot set a positive priority. The check resolves to the organization’s owner relation, which excludes federate owners, so the request is refused on the owning orchestrate cluster. Use an orchestrate organization owner instead.

Each re-stamped stage records a priority_updated entry in the workflow’s activity log:

$ fuzzball workflow events 3cc718d1-e35c-4455-b4f4-ca3412727277

The entry names the factor that changed (workflow), the old value, and the new one.

To see the current inputs and the resulting effective priority for one stage, pass the stage name to fuzzball workflow describe:

$ fuzzball workflow describe 3cc718d1-e35c-4455-b4f4-ca3412727277 solver

To see where the workflow now sits relative to everything else queued, use fuzzball queue show --mine – see Queue and Scheduling Observability.

• The workflow must still be active. A workflow that has finished, failed, or been canceled has no queued work left for a priority to affect, and the command reports that rather than recording a value that can never matter.
• Only your own workflows. Changing a workflow’s priority requires ownership of that workflow, the same permission fuzzball workflow stop requires.
• Reordering is relative. Priority decides the order in which queued work is considered; it does not reserve capacity or evict anything on its own. A running workflow is only displaced when the cluster has preemption enabled and the workflow’s stages were submitted as preemptible.

To move everything a group, user, or organization submits – rather than one workflow – an administrator sets the entity-level inputs instead:

# Organization priority (cluster administrator)
$ fuzzball organization update <organization-id> --priority 5

# Group priority (organization owner)
$ fuzzball group update research --priority 10

# User priority (organization owner)
$ fuzzball user update <user-id> --priority 2

These changes also propagate to work already queued, so they take effect without resubmitting anything. They move all of the entity’s queued work together, which is why they cannot substitute for a per-workflow change.