Requirements

You need a few things before you begin deploying your new Federate instance.

The Federate deployment method leverages the K8s infrastructure of an existing Fuzzball Orchestrate cluster. This is appropriate because the purpose of Federate is to manage Orchestrate clusters. If you have not already done so, consult the Local Admin Guides or Cloud Admin Guides to learn how to deploy a Fuzzball cluster in the environment of your choice.

You can deploy Federate and use it to manage a single Orchestrate cluster. But the real value of Federate emerges when you have 2 or more existing Fuzzball Orchestrate clusters to manage. Therefore, we recommend installing and using Federate when you have 2 or more clusters that you want to manage together.

This guide assumes access to CIQ Depot. Depot provides a secure delivery platform for a number of asset types, including RPM packages and SIF/OCI container images. It will be used to obtain the Federate images that will be deployed.

You should have the kubectl command installed either locally (if your local environment is configured with the context pointing to your Fuzzball deployment) or on the node where you set up K8s and are running Fuzzball. You should already have this requirement satisfied since you used it to deploy your Fuzzball cluster.

Fuzzball verifies TLS certificates on the gRPC connections between Orchestrate and Federate. Each cluster reaches the other at its external API endpoint, so the certificate being verified is the one issued by that cluster’s ingress issuer. If a cluster’s ingress issuer is a private CA, then the cluster on the other side must be configured to trust that CA through the trustedCACerts field of its FuzzballOrchestrate or FuzzballFederate resource.

No action is needed when either of the following is true:

• The ingress issuer is a public CA such as letsencrypt-prod. Its root is already in the default trust store.
• Federate and Orchestrate share a K8s cluster and its issuers, which is the common case when you deploy Federate alongside an existing Orchestrate cluster. Each cluster already trusts the CAs of its own issuers.

You need to add a CA explicitly when Federate and an Orchestrate cluster are separate K8s clusters and either one’s ingress issuer is a private CA. Note that two clusters that each use the internal-ca-issuer ClusterIssuer have different internal CAs, generated per cluster, so they still need to exchange them.

The operator also tries to add an external ingress issuer’s CA to the trust bundle automatically, but it can only do so for issuer types that expose their CA: a ca issuer, or an acme, vault, or venafi issuer with caBundle set. When extraction fails the operator records a warning event and continues the deployment, so the CA has to be added manually as described below.

The steps below involve two clusters. The issuing cluster is the one whose ingress certificates are signed by the private CA. The verifying cluster is the one that connects to it and needs to trust that CA. Repeat the steps for each direction that needs it: a Federate cluster must trust every Orchestrate cluster it manages, and each Orchestrate cluster must trust the Federate cluster it registers with.

First, point your kubectl context at the issuing cluster and obtain the CA certificate that signs its ingress certificates. When the ingress issuer is internal-ca-issuer, read the CA from the root-ca-cert Secret in the cert-manager namespace:

$ kubectl get secret -n cert-manager root-ca-cert \
  -o "jsonpath={.data['ca\.crt']}" | base64 --decode > remote-ingress-ca.pem

When the issuing cluster instead uses a custom ingress CA supplied through tls.ingressIssuer.create.customCA, that CA is stored in the root-ingress-ca Secret in the same namespace. When it uses an external ingress issuer, obtain the certificate from that issuer or from your PKI administrator.

Next, point your kubectl context at the verifying cluster and store the certificate in a Secret in its fuzzball-system namespace:

$ kubectl create secret generic remote-ingress-ca \
  --namespace fuzzball-system \
  --from-file=ca.crt=remote-ingress-ca.pem

Then reference that Secret from the trustedCACerts field of the verifying cluster’s resource — FuzzballFederate when Federate needs to trust an Orchestrate cluster, or FuzzballOrchestrate when Orchestrate needs to trust Federate:

spec:
  trustedCACerts:
    refs:
      - name: remote-ingress-ca
        key: ca.crt

A Federate cluster that manages several Orchestrate clusters with different private CAs needs one refs entry per CA. You can also supply certificates directly with trustedCACerts.inline — see Egress Proxy and Trusted CA Configuration for the full parameter reference.

The operator distributes the certificates through the cluster trust bundle mounted at /etc/ssl/certs in every Fuzzball service pod, alongside the default public CAs. Updating trustedCACerts does not restart those pods, and each service caches its trust store for the lifetime of the process, so roll the pods out to pick up the new bundle:

$ kubectl rollout restart deployment -n fuzzball

Use -n fuzzball-federate on a Federate cluster. You can confirm the CA reached the bundle by listing its contents. The ConfigMap is named trusted-ca-certs-bundle in the fuzzball namespace on an Orchestrate cluster, and fuzzball-federate-trusted-ca-certs-bundle in the fuzzball-federate namespace on a Federate cluster:

$ kubectl get configmap -n fuzzball trusted-ca-certs-bundle \
  -o "jsonpath={.data['ca-bundle\.crt']}" \
  | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
  | openssl pkcs7 -print_certs -noout | grep subject

Installing the CA into a node’s host trust store, with update-ca-trust or update-ca-certificates, has no effect. Fuzzball services run in containers that never read the node’s trust store; they mount the operator-managed bundle at /etc/ssl/certs instead, which shadows the trust store baked into the container image.

If the CA is not trusted, the affected connection fails with:

x509: certificate signed by unknown authority

On a Federate cluster this surfaces when Federate forwards a user request — a workflow, account, scheduler, or node operation — to the Orchestrate cluster whose CA is missing. On an Orchestrate cluster it surfaces when the cluster replicates workflow, application, volume, or provisioner state to Federate.

You should have the AWS CLI installed locally and configured to access your account. If you already deployed your a Fuzzball cluster in AWS via the marketplace listing, you should already have this requirement satisfied.

If you are deploying Federate alongside your existing Orchestrate cluster on-prem you will need a slightly larger machine to accommodate all of the running pods.