Deployment

After fulfilling the prerequisites from the requirements doc, you are ready to deploy Federate in an existing K8s cluster that already supports Fuzzball.

If you are running kubectl on the same node that is hosting the K8s installation and Fuzzball Orchestrate deployment (common in on-prem deployments) then you don’t need to worry about setting up a kubeconfig file. If you are using the kubectl command on a local host to control a K8s deployment on a remote server (common in a cloud deployment) you will need to double check that your kubeconfig file is pointing at the right cluster.

In AWS, the command to update your kubeconfig file will look something like the following:

$ aws eks update-kubeconfig --region us-east-1 --name <cluster name and ID>

Check the Cloud Admin Guides for information on retrieving the appropriate command to update your kubeconfig file.

Fuzzball Federate uses the Fuzzball operator that was already installed to deploy your Fuzzball Orchestrate cluster with a new Custom Resource Definition (CRD) file. We’ll call this file fuzzball-federate.yaml. Many of the values are predetermined, but there are some values you need to supply for your particular installation.

You can create an appropriate fuzzball-federate.yaml file by populating the following environment variables with their correct values and running the cat command to create the file.

Fill in your CIQ depot credentials here:

$ DEPOT_USER="" # populate with your username for CIQ Depot

$ ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

Decide on the version of Federate that you want to deploy (for example, v4.2.3):

$ FED_VERSION=v4.2.3 # change as necessary

You need to determine the certificate issuer for your externally facing URLs. This is set at Fuzzball deployment. In the AWS marketplace, it will be letsencrypt-prod. On-prem, your deployment may have (insecure) self-signed certificates in which case the appropriate value will be internal-ca-issuer, it might be letsencrypt-prod if you used letsencrypt, or it could be another issuer. You can find this information with kubectl get clusterissuer.

$ CERT_ISSUER=letsencrypt-prod # change as necessary

CERT_ISSUER sets the issuer for Federate’s externally facing certificates, and Fuzzball verifies those certificates on the gRPC connections between Orchestrate and Federate. If you deploy Federate into the same K8s cluster as Orchestrate, that cluster already trusts the CAs of its own issuers and no further action is needed. If Federate and Orchestrate are in separate K8s clusters and either one uses a private CA, each cluster must be configured to trust the other’s CA through the trustedCACerts field of its FuzzballFederate or FuzzballOrchestrate resource. See TLS Certificate Trust for instructions.

You will need to supply the domain where the cluster is hosted, the realm ID of the keycloak instance, and the email address that was set as the owner of the keycloak instance. You can obtain this information as described here.

$ DOMAIN=myfuzzballdomain.com # change as necessary

$ KC_REALMID=$(uuidgen --random | tr 'A-Z' 'a-z')

$ KC_EMAIL="" # populate with the keycloak owner email address

Federate’s audit service, database, and cluster configuration service (JetStream) all need a storage class that actually exists in your cluster. The name varies by platform: local on-prem clusters typically provide local-path, while cloud clusters expose their own classes. On AWS EKS the EBS-backed class is usually named gp2 or gp3.

List the storage classes available in your cluster and identify the one you want to use (the default class is marked (default)):

$ kubectl get storageclass
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  9d

Set the STORAGE_CLASS environment variable to the class you want Federate to use. On AWS EKS this will commonly be gp2 or gp3; on a local on-prem cluster it is often local-path:

$ STORAGE_CLASS="" # choose a class from 'kubectl get storageclass'

Choose the initial password for the Federate Keycloak admin user. Set it to a strong value of your own rather than leaving the placeholder:

$ KC_PASSWORD="" # populate with a strong initial keycloak admin password

Rather than storing the Keycloak password in the CRD at all, you can reference a Kubernetes Secret with passwordRef. See Kubernetes Secret References for details.

Once you have all of these environment variables set in your terminal, you can copy and paste the following code block to create the fuzzball-federate.yaml file in your current working directory with the values you chose.

The example fuzzball-federate.yaml configures the Federate audit service, database, and cluster configuration service (JetStream) to use the storage class you selected in STORAGE_CLASS above. The cluster configuration service runs as a 3-replica StatefulSet, each replica with its own 10Gi persistent volume by default.

$ cat >fuzzball-federate.yaml<<EOF
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballFederate
metadata:
  labels:
    app.kubernetes.io/name: fuzzball-federate
    app.kubernetes.io/part-of: fuzzball
  name: fuzzball-federate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  fuzzball:
    version: ${FED_VERSION}
    cluster:
      kind: FEDERATE
    audit:
      storage:
        class: ${STORAGE_CLASS}
      localStorage: true
    jetstream:
      storage:
        class: ${STORAGE_CLASS}
  database:
    create:
      enableDebugPod: true
      storage:
        class: ${STORAGE_CLASS}
  tls:
    # Externally facing issuer, likely letsencrypt but could be something else.
    # Issuers in the cluster can be found with 'kubectl get clusterissuer'.
    ingressIssuer:
      external:
        issuerName: ${CERT_ISSUER}
    internalIssuer:
      external:
        issuerName: internal-ca-issuer
  # Assumes existing Kong deployment from FuzzballOrchestrate
  ingress:
    external:
      domain: federate.${DOMAIN}
      className: kong
  keycloak:
    create:
      createDatabase: true
      realmId: ${KC_REALMID}
      username: keycloak
      password: ${KC_PASSWORD}  # or use passwordRef to reference a K8s Secret
      ownerEmail: ${KC_EMAIL}
EOF

The example above sets the Keycloak admin password from the KC_PASSWORD environment variable. Make sure you set it to a strong value, or use passwordRef to reference a Kubernetes Secret instead of storing the password in the CRD. See Kubernetes Secret References for details.

You can also use external cert-manager issuers. If you are unsure which ClusterIssuers are available in your cluster, run kubectl get clusterissuer. For full details on this configuration pattern, see Deploying with External cert-manager.

The example above deploys a new Keycloak instance alongside Federate (keycloak.create). If your organization already operates a Keycloak server, you can connect Federate to it instead by using keycloak.external in place of keycloak.create.

Replace the keycloak section of fuzzball-federate.yaml with the following:

  keycloak:
    external:
      url: https://keycloak.example.com   # URL of your existing Keycloak instance
      realmId: <uuid-v4-realm-id>         # Existing realm ID (UUID v4; mixed case accepted)
      realmName: ExistingRealm            # Realm display name (used as the Fuzzball org name)
      username: admin                     # Keycloak admin from the master realm (not the target realm)
      ownerEmail: admin@example.com       # Must match an existing user in the target realm
      passwordRef:
        name: keycloak-credentials        # Name of a Kubernetes Secret
        key: admin-password               # Key within the Secret

The passwordRef block above references a Kubernetes Secret that you must create yourself before applying the FuzzballFederate resource. Create the Secret in the same namespace as the Federate deployment, with a key matching passwordRef.key. See Kubernetes Secret References for the exact kubectl create secret generic invocation and other supported reference patterns.

When using keycloak.external, Federate will not create or modify the realm. The realm, its users, and its Fuzzball-specific client configuration must already exist. Federate’s coldboot process will create the local organization database record using the realm’s existing users without touching Keycloak itself.

The username and passwordRef credentials must belong to a user in the Keycloak master realm, not a user local to the target realm. A realm-scoped admin account will cause the deployment to fail.

The target realm must contain the following before deployment — these client IDs are hardcoded in Fuzzball, so the names must match exactly:

• A fuzzball-cli client, configured as a public client with Direct Access Grants and Device Authorization Grant both enabled.
• A fuzzball-ui client, configured as a confidential client with the Standard Flow enabled, a generated client secret, and Valid Redirect URIs that include the Federate UI hostname (both login and post-logout redirects).
• The user identified by ownerEmail must have the realm-management client roles assigned (manage-users, manage-clients, view-realm, etc.) so the operator can perform realm operations on its behalf.

If the external realm is an existing Orchestrate Keycloak that already contains a fuzzball-ui client, its Valid Redirect URIs must be extended to include the Federate UI hostname in addition to the Orchestrate UI hostname.

Without this pre-existing configuration the deployment will appear to succeed (the coldboot process only writes the local DB record), but UI login and CLI authentication will fail silently at first use.

By default each cluster only trusts JWTs issued by its own Keycloak. In a federated topology the Orchestrate and Federate clusters typically run separate Keycloak instances, so a token minted by one cluster’s Keycloak is rejected by the other. To let a cluster accept tokens issued by a second Keycloak — for example, so the Federate cluster honors tokens from the Orchestrate cluster’s Keycloak — add that Keycloak’s issuer URL to fuzzball.config.additionalTrustedIssuers.

Add the field to the fuzzball section of fuzzball-federate.yaml:

  fuzzball:
    config:
      additionalTrustedIssuers:
        - https://auth.${DOMAIN}/   # The other cluster's Keycloak issuer URL

Each entry must be an absolute http(s) URL. You can list more than one issuer. The same field is available on FuzzballOrchestrate, so the trust can be configured in either direction.

Trusted issuers are matched on scheme://host only — the path and query string are ignored. An entry whose scheme and host already match the cluster’s own Keycloak is discarded, so it cannot override the cluster’s built-in issuer. See the FuzzballOrchestrate CRD Reference for full details.

Adding an issuer to additionalTrustedIssuers grants full authentication trust to that issuer. Only add issuers you control or have an established trust relationship with. Any valid JWT from a trusted issuer will be accepted as authentication.

Federate clusters support the same workflow catalog as Orchestrate clusters. The official CIQ catalog is enabled by default and syncs on a schedule via a CronJob. You can control this behavior with the fuzzball.workflowCatalog field:

  fuzzball:
    workflowCatalog:
      catalogs:
        official:
          enabled: true                                         # Set to false to disable
          uri: https://github.com/ctrliq/ciq-fuzzball-catalog  # Default catalog URI

The operator always deploys a fuzzball-catalog-sync CronJob, regardless of the enabled setting. This CronJob periodically runs fuzzball workflow catalog source reload --all, which refreshes every catalog Fuzzball currently knows about.

When enabled is true (the default), the operator additionally deploys a one-time fuzzball-load-default-catalog Job during deployment that seeds the official CIQ catalog so the recurring sync has something official to refresh. Setting enabled: false skips this Job, so the official catalog is never registered with Fuzzball — the fuzzball-catalog-sync CronJob still runs, but it will only refresh other catalogs (if any) that have been registered manually. No additional configuration is required to start using the catalog.

In addition to the official CIQ catalog, any non-CIQ applications created on an Orchestrate cluster are automatically replicated to the connected Federate cluster. This happens in two ways:

• On registration: When a Federate cluster is registered with an Orchestrate cluster, all existing non-CIQ applications are bulk-synced to Federate automatically.
• On change: Any subsequent create, update, or delete of a non-CIQ application on Orchestrate is pushed to the connected Federate cluster in real time.

No configuration is required for application replication. Replicated applications are stamped with the source cluster ID so they can be distinguished from applications created natively on the Federate cluster.

When Federate forwards workflow score requests to downstream clusters, it automatically materializes all secret references embedded in the workflow specification. This includes:

• Job-scoped secrets: Image pull secrets (image.secret), image decryption secrets (image.decryption-secret), and environment variable references using secret:// URIs
• Service-scoped secrets: The same secret reference patterns are supported in the services section of a workflow and are handled identically to job-scoped secrets
• Volume secrets: S3 and other storage credentials referenced in ingress/egress configurations

Secret materialization ensures that workflows referencing secrets at any scope (user, group, organization, or cluster) will have those secrets resolved and forwarded to the target cluster. Administrators do not need to manually replicate secrets to downstream clusters; Federate handles this automatically as part of the score request.

Secret materialization applies to all secret references in the workflow specification, regardless of where they appear. Both jobs and services sections support the same secret reference syntax.

Federate deployments include a dedicated cluster configuration service — a 3-replica fuzzball-jetstream StatefulSet in the Federate namespace — that stores the cluster configuration applied with fuzzball cluster config set. This means the same configuration management commands used against Orchestrate clusters also work against a Federate cluster.

On a Federate cluster, only the global settings (for example, scheduler options) in the configuration apply. Node provisioner definitions are not used on Federate clusters — Federate receives its node provisioner information from its registered Orchestrate clusters — so any definitions entries in a configuration applied to Federate are stored but ignored.

The example fuzzball-federate.yaml above sets the service’s storage class via the fuzzball.jetstream field of the FuzzballFederate resource. The same field also lets you tune the replica count (replicas, default 3) and the volume size (storage.size, default 10Gi).

Now you can run the following command to deploy Federate into the K8s cluster that supports your Fuzzball Orchestrate installation.

$ kubectl apply -f fuzzball-federate.yaml

And you can watch as the Fuzzball Operator carries out the deployment of Federate in your K8s cluster like so. The process usually takes 10 or 15 minutes to complete.

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

[snip...]

Resources:
    ~ 3 updated
    136 unchanged

Duration: 4s

2025-03-13T17:13:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballFederate","name":"fuzzball-federate","uid":"6bdda038-b41c-489d-9ec9-f2fb5311388a","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"412932"}, "reason": "DeploymentSucceeded"}
2025-03-13T17:13:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballfederate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballFederate", "FuzzballFederate": {"name":"fuzzball-federate"}, "namespace": "", "name": "fuzzball-federate", "reconcileID": "a63ae801-a7fb-40d1-a733-bf3582bfd463"}

At this point, you can access your new Federate cluster and configure it to point to your existing Fuzzball Orchestrate cluster(s).