In-Workflow API Access

Every job and service container in a workflow automatically receives a workflow-scoped API token, so processes running inside a workflow can call the Fuzzball API. This supports long-running pipeline controllers — for example a Nextflow controller job that submits each pipeline task as a new Fuzzball workflow while it runs — without passing user credentials into the workflow as secrets and without logging in inside the container.

The token carries the identity of the user who submitted the workflow, acting in the group the workflow was submitted under, and is bound to the workflow it was injected into:

• It works only against the workflow and endpoints APIs. On the workflow API it covers submitting and tracking workflow execution: starting, validating, listing, describing, and stopping workflows, and reading workflow stages and stage events. On the endpoints API it covers listing and reading the workflow service endpoints the owner can reach, and minting endpoint access tokens for them. Everything else is refused, including workflow ownership management, account-wide reporting, and every other API such as volumes, secrets, and administration.
• Commands that stream from running containers — fuzzball workflow log, exec, attach, and port-forward — do not accept workflow tokens.
• It stops working as soon as its workflow finishes, fails, or is canceled, regardless of its expiry time.
• In job containers the token expires one hour after the job’s execution timeout (walltime), so a job can never outlive its token. Jobs that set no walltime (policy.timeout.execute: "0") run until they finish, and their tokens expire after seven days – as do tokens in service containers, which have no walltime either. A container that runs longer than its token can refresh the token before it expires.

Within those operations the token acts as the submitting user, so it can also list and stop that user’s other workflows – it is not confined to the workflow it was injected into. That is what lets a controller job manage the workflows it submits. To narrow the blast radius further, submit long-running controller workflows as a dedicated user.

Inside any workflow container, the Fuzzball CLI picks up FB_TOKEN and FB_API_HOST automatically when no CLI context is configured — no further configuration is needed:

$ fuzzball workflow start pipeline-step.fz
d0a45cbe-77ad-4a94-97e6-51f5b24b8de2

If a CLI configuration file with an active context is present (for example in a mounted home directory), the configured context takes precedence over the environment variables.

To call the REST API directly instead, pass the token as a bearer credential:

$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/workflows"

Both paths verify the API’s TLS certificate against the container image’s trust store. If your cluster serves the API with a private certificate authority, add that CA to the images used by workflows that call the API, or use the trustedCACerts section of the FuzzballOrchestrate resource. Development deployments that serve a self-signed certificate need --insecure (CLI) or -k (curl).

The token can also list the workflow service endpoints its owner has access to, and mint endpoint access tokens for them – so a controller workflow can discover the services it started and hand out ready-to-use endpoint credentials. The same CLI commands work unchanged inside a container:

$ fuzzball workflow endpoints list
$ fuzzball workflow endpoints generate-token ENDPOINT_ID

Or over REST:

$ curl -H "Authorization: Bearer ${FB_TOKEN}" "${FB_OPENAPI_URL}/v4/endpoints"
$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
    "${FB_OPENAPI_URL}/v4/endpoints/ENDPOINT_ID/token" -d '{}'

Access is scoped exactly as for the owner’s own credential: the token sees the endpoints the submitting user can reach across their groups and organization, and is refused for endpoints they cannot. Like every other use of the workflow token, its access to this API stops the moment the workflow finishes, fails, or is canceled.

An endpoint token minted this way is a separate credential with its own lifetime: it is not revoked when the workflow ends or when workflow tokens are disabled. Its lifetime is instead capped at the minting workflow token’s own remaining lifetime, so it can never outlive the credential that created it.

Environment variables cannot be changed in a running container, so a service that outlives its seven-day token has to renew the credential itself. Exchange a still-valid token for a fresh one with the same permissions and a new seven-day expiry:

$ curl -s -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
    "${FB_OPENAPI_URL}/v4/workflows:refreshToken"
{"token":"eyJhbGciOiJFUzI1NiIs...","expiresAt":"2026-08-10T17:04:11Z"}

The refreshed token replaces the injected one in the container’s own environment. Nothing restarts, and the previous token keeps working until its original expiry, so there is no cutover to coordinate. Only overwrite FB_TOKEN once you have a token in hand – assigning the result of a failed call would discard a credential that is still valid, and the container cannot recover from that:

new_token=$(curl -sf -X POST -H "Authorization: Bearer ${FB_TOKEN}" \
  "${FB_OPENAPI_URL}/v4/workflows:refreshToken" | jq -r '.token // empty')
if [ -n "$new_token" ]; then
  export FB_TOKEN="$new_token"
else
  echo "workflow token refresh failed; keeping the current token" >&2
fi

The CLI reads FB_TOKEN from the environment on every invocation, so a re-exported value is picked up by the next fuzzball command with no further configuration. Retrying on the next scheduled run is enough as long as the schedule leaves room inside the seven-day window.

Refresh before the token expires. An expired token is rejected outright – there is no grace period, and no way to deliver a new token into a running container from outside it. A service that needs to run indefinitely should refresh on a schedule comfortably inside the seven-day window (daily, for example), not when expiry is imminent.

Only a workflow token can be refreshed, and it can only refresh itself: the returned token always carries the same user, group, and workflow as the one presented. Your own user credential cannot refresh a workflow’s token, and there is no administrative override. All the usual restrictions still apply – refresh fails once the workflow finishes, fails, or is canceled, and stopping the workflow remains the way to revoke a workflow token. Because a token holder can keep renewing while the workflow runs, submit long-running controller and service workflows as a dedicated user.

Every cluster serves a Fuzzball CLI download that always matches the cluster’s version. The binary is larger than a job container’s writable layer so it must be downloaded into a mounted volume, for example:

volumes:
  data:
    reference: volume://user/ephemeral
jobs:
  controller:
    image:
      uri: docker://alpine:latest
    mounts:
      data:
        location: /data
    script: |
      #!/bin/sh
      cd /data
      wget --header "Cookie: fwi.access-token=${FB_TOKEN}" -O fuzzball \
          "https://ui.example.com/api/download/cli/linux/amd64"   # or linux/arm64
      chmod +x fuzzball
      ./fuzzball workflow list

This example uses ui.example.com, substitute your deployment’s DNS name, and pick linux/amd64 or linux/arm64 to match the node architecture. The downloaded CLI picks up FB_TOKEN and FB_API_HOST automatically, so it does not require login. Invoke the Fuzzball CLI by path or move it onto PATH.

The TLS note above applies here too; with a self-signed development certificate use wget --no-check-certificate and ./fuzzball --insecure.

A workflow submitted with a workflow token records the submitting workflow as its parent; workflows submitted directly by a user have no parent. The workflows a controller job launched are therefore distinguishable from unrelated work in the same group.

fuzzball workflow describe shows the parent in the Parent: row and the number of workflows this workflow submitted in the Children: row; -o json / -o yaml output includes both as parentWorkflowId and childCount. To list the workflows a given workflow submitted, pass its ID to --parent:

$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2

Like every workflow list invocation, this defaults to workflows created in the last seven days; combine --parent with --created-after to reach further back:

$ fuzzball workflow list --parent d0a45cbe-77ad-4a94-97e6-51f5b24b8de2 --created-after 30d

Cluster administrators can turn off token injection cluster-wide. On Kubernetes deployments, set the disableWorkflowTokens field in the FuzzballOrchestrate resource:

spec:
  disableWorkflowTokens: true

On deployments where the orchestrator configuration file is managed directly, set the equivalent feature flag:

features:
  disableWorkflowTokens: true

With this disabled, workflow containers receive none of the variables above, and the API rejects workflow tokens that were injected before the change — including tokens held by containers that are still running.