Deployment

After fulfilling the prerequisites from the requirements doc, you are ready to deploy Federate in an existing K8s cluster that is already support Fuzzball.

If you are running kubectl on the same node that is hosting the K8s installation and Fuzzball Orchestrate deployment (common in on-prem deployments) then you don’t need to worry about setting up a kubeconfig file. If you are using the kubectl command on a local host to control a K8s deployment on a remote server (common in a cloud deployment) you will need to double check that your kubeconfig file is pointing at the right cluster.

In AWS, the command to update your kubeconfig file will look something like the following:

$ aws eks update-kubeconfig --region us-east-1 --name <cluster name and ID>

Check the Cloud Admin Guides for information on retrieving the appropriate command to update your kubeconfig file.

Fuzzball Federate uses the Fuzzball operator that was already installed to deploy your Fuzzball Orchestrate cluster with a new Custom Resource Definition (CRD) file. We’ll call this file fuzzball-federate.yaml. Many of the values are predetermined, but you there are some values you need to supply for your particular installation.

You can create an appropriate fuzzball-federate.yaml file by populating the following environment variables with their correct values and running the cat command to create the file.

Fill in your CIQ depot credentials here:

$ DEPOT_USER="" # populate with your username for CIQ Depot

$ ACCESS_KEY="" # populate with the Depot key obtained from the CIQ sales/support team

Decide on the version of Federate that you want to deploy (for example, v3.4.0):

$ FED_VERSION=v3.4.0 # change as necessary

You need to determine the certificate issuer for your externally facing URLs. This is set at Fuzzball deployment. In the AWS marketplace, it will be letsencrypt-prod. On prem, your deployment may have (insecure) self-signed certificates in which case the appropriate value will be internal-ca-issuer, it might be letsencrypt-prod if you used letsencrypt, or it could be another issuer. You can find this information with kubectl get clusterissuer.

$ CERT_ISSUER=letsencrypt-prod # change as necessary

You will need to supply the domain where the cluster is hosted, the realm ID of the keycloak instance, and the email address that was set as the owner of the keycloak instance. You can obtain this information as described here.

$ DOMAIN=myfuzzballdomain.com # change as necessary

$ KC_REALMID=$(uuidgen --random | tr 'A-Z' 'a-z')

$ KC_EMAIL="" # populate with the keycloak owner email address

Once you have all of these environment variables set in your terminal, you can copy and paste the following code block to create the fuzzball-federate.yaml file in your current working directory with the values you chose.

The example fuzzball-federate.yaml configures the Federate audit service and database to use storage local to the nodes running on EKS.

$ cat >fuzzball-federate.yaml<<EOF
apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballFederate
metadata:
  labels:
    app.kubernetes.io/name: fuzzball-federate
    app.kubernetes.io/part-of: fuzzball
  name: fuzzball-federate
spec:
  image:
    username: ${DEPOT_USER}
    password: ${ACCESS_KEY}
    exclusive: false
  fuzzball:
    version: ${FED_VERSION}
    cluster:
      kind: FEDERATE
    audit:
      storage:
        class: local-path
      localStorage: true
  database:
    create:
      enableDebugPod: true
      storage:
        class: local-path
  tls:
    # Externally facing issuer, likely letsencrypt but could be something else.
    # Issuers in the cluster can be found with 'kubectl get clusterissuer'.
    ingressIssuer:
      external:
        issuerName: ${CERT_ISSUER}
    internalIssuer:
      external:
        issuerName: internal-ca-issuer
  # Assumes existing Kong deployment from FuzzballOrchestrate
  ingress:
    external:
      domain: federate.${DOMAIN}
      className: kong
  keycloak:
    create:
      createDatabase: true
      realmId: ${KC_REALMID}
      username: keycloak
      password: keycloak  # change this or use passwordRef to reference a K8s Secret
      ownerEmail: ${KC_EMAIL}
EOF

The example above uses a placeholder Keycloak password. Change it before deploying, or use passwordRef to reference a Kubernetes Secret instead of storing the password in the CRD. See Kubernetes Secret References for details.

You can also use external cert-manager issuers. If you are unsure which ClusterIssuers are available in your cluster, run kubectl get clusterissuer. For full details on this configuration pattern, see Deploying with External cert-manager.

The example above deploys a new Keycloak instance alongside Federate (keycloak.create). If your organization already operates a Keycloak server, you can connect Federate to it instead by using keycloak.external in place of keycloak.create.

Replace the keycloak section of fuzzball-federate.yaml with the following:

  keycloak:
    external:
      url: https://keycloak.example.com   # URL of your existing Keycloak instance
      realmId: <uuid-v4-realm-id>         # Existing realm ID (UUID v4; mixed case accepted)
      realmName: ExistingRealm            # Realm display name (used as the Fuzzball org name)
      username: admin                     # Keycloak admin from the master realm (not the target realm)
      ownerEmail: admin@example.com       # Must match an existing user in the target realm
      passwordRef:
        name: keycloak-credentials        # Name of a Kubernetes Secret
        key: admin-password               # Key within the Secret

The passwordRef block above references a Kubernetes Secret that you must create yourself before applying the FuzzballFederate resource. Create the Secret in the same namespace as the Federate deployment, with a key matching passwordRef.key. See Kubernetes Secret References for the exact kubectl create secret generic invocation and other supported reference patterns.

When using keycloak.external, Federate will not create or modify the realm. The realm, its users, and its Fuzzball-specific client configuration must already exist. Federate’s coldboot process will create the local organization database record using the realm’s existing users without touching Keycloak itself.

The username and passwordRef credentials must belong to a user in the Keycloak master realm, not a user local to the target realm. A realm-scoped admin account will cause the deployment to fail.

The target realm must contain the following before deployment — these client IDs are hardcoded in Fuzzball, so the names must match exactly:

• A fuzzball-cli client, configured as a public client with Direct Access Grants and Device Authorization Grant both enabled.
• A fuzzball-ui client, configured as a confidential client with the Standard Flow enabled, a generated client secret, and Valid Redirect URIs that include the Federate UI hostname (both login and post-logout redirects).
• The user identified by ownerEmail must have the realm-management client roles assigned (manage-users, manage-clients, view-realm, etc.) so the operator can perform realm operations on its behalf.

If the external realm is an existing Orchestrate Keycloak that already contains a fuzzball-ui client, its Valid Redirect URIs must be extended to include the Federate UI hostname in addition to the Orchestrate UI hostname.

Without this pre-existing configuration the deployment will appear to succeed (the coldboot process only writes the local DB record), but UI login and CLI authentication will fail silently at first use.

Federate clusters support the same workflow catalog as Orchestrate clusters. The official CIQ catalog is enabled by default and syncs on a schedule via a CronJob. You can control this behavior with the fuzzball.workflowCatalog field:

  fuzzball:
    workflowCatalog:
      catalogs:
        official:
          enabled: true                                         # Set to false to disable
          uri: https://github.com/ctrliq/ciq-fuzzball-catalog  # Default catalog URI

The operator always deploys a fuzzball-catalog-sync CronJob, regardless of the enabled setting. This CronJob periodically runs fuzzball admin application catalog reload --all, which refreshes every catalog Fuzzball currently knows about.

When enabled is true (the default), the operator additionally deploys a one-time fuzzball-load-default-catalog Job during deployment that seeds the official CIQ catalog so the recurring sync has something official to refresh. Setting enabled: false skips this Job, so the official catalog is never registered with Fuzzball — the fuzzball-catalog-sync CronJob still runs, but it will only refresh other catalogs (if any) that have been registered manually. No additional configuration is required to start using the catalog.

In addition to the official CIQ catalog, any non-CIQ applications created on an Orchestrate cluster are automatically replicated to the connected Federate cluster. This happens in two ways:

• On registration: When a Federate cluster is registered with an Orchestrate cluster, all existing non-CIQ applications are bulk-synced to Federate automatically.
• On change: Any subsequent create, update, or delete of a non-CIQ application on Orchestrate is pushed to the connected Federate cluster in real time.

No configuration is required for application replication. Replicated applications are stamped with the source cluster ID so they can be distinguished from applications created natively on the Federate cluster.

Now you can run the following command to deploy Federate into the K8s that supports your Fuzzball Orchestrate installation.

$ kubectl apply -f fuzzball-federate.yaml

And you can watch as the Fuzzball Operator carries out the deployment of Federate in your K8s cluster like so. The process usually takes 10 or 15 minutes to complete.

$ kubectl logs -l app.kubernetes.io/name=fuzzball-operator -n fuzzball-system -f --tail=-1

[snip...]

Resources:
    ~ 3 updated
    136 unchanged

Duration: 4s

2025-03-13T17:13:00Z	DEBUG	events	Resources have been deployed successfully	{"type": "Normal", "object": {"kind":"FuzzballFederate","name":"fuzzball-federate","uid":"6bdda038-b41c-489d-9ec9-f2fb5311388a","apiVersion":"deployment.ciq.com/v1alpha1","resourceVersion":"412932"}, "reason": "DeploymentSucceeded"}
2025-03-13T17:13:00Z	INFO	Updated Fuzzball status to ReconciliationComplete - Reconciliation completed successfully	{"controller": "fuzzballfederate", "controllerGroup": "deployment.ciq.com", "controllerKind": "FuzzballFederate", "FuzzballFederate": {"name":"fuzzball-federate"}, "namespace": "", "name": "fuzzball-federate", "reconcileID": "a63ae801-a7fb-40d1-a733-bf3582bfd463"}

At this point, you can access your new Federate cluster and configure it to point to your existing Fuzzball Orchestrate cluster(s).