Deployment

If there are any left-over stacks or resources from a previous deployment they may interfere with your ability to deploy a fresh Fuzzball cluster. Please make sure that all stacks, nested stacks, and resources from previous deployments have been fully and successfully deleted before initiating a new one.

Make sure you are logged into your AWS account as the fuzzballAdmin IAM user and navigate to the Fuzzball Cloud Marketplace listing. Review the information paying special attention to the pricing options. When you are ready to subscribe, press the “View Purchase Options” button.

On the subscription page you can accept the public or private offer

After subscribing, you can press the “Launch your Software” button.

The launch page will include a link to the AWS CloudFormation page as well as the link to the CloudFormation template for your selected version of Fuzzball. Note down the highlighted URI of the template.

With the template URI in hand you can head over to the CloudFormation page either by following the link on the launch page or going directly to CloudFormation in the AWS console. On the CloudFormation page select “Create stack” and on the following page add the Fuzzball CloudFormation template URI

Make sure you are in the region where you want to install Fuzzball and where you have the appropriate service quotas configured. Use the dropdown on the top right to change to the correct region if necessary.

After clicking “Next” you will be taken to the configuration form. Here, you can supply the values that are specific to your Fuzzball cluster.

Most of these values are self-explanatory or are adequately explained by the accompanying text in the form itself. There are a few notes below to better explain some of the values.

• ClusterAdminUsers: These are the IAM usernames that will have access to the Kubernetes cluster through the AWS UI or kubectl. This value is technically optional, but you may have trouble accessing your cluster (because you won’t be able to retrieve login information) if you leave it blank. Note that the IAM users specified in this parameter do not need to actually exist before the deployment process, but can be added to the AWS account (with appropriate permissions) after deployment is completed. We suggest using the fuzzballAdmin IAM user as created in the previous section.

• SSOAdminRoleArn: This is the arn that corresponds to the role your users assume when they login via AWS’ SSO, or IAM Identity Center. If you plan to connect to the cluster as an individual IAM user, you can ignore this parameter and follow the instructions for ClusterAdminUsers above.

• Domain: This is the domain name of the Hosted Zone that you should have configured in Route53 by either purchasing a new domain, or creating a new Hosted Zone and adding it to the DNS records of your existing top-level domain. See the section on setting up a hosted zone with Route53 in the appendices for more details.

• DomainHostedZoneID: This is the actual ID of the Hosted Zone that you should have configured in Route53. Once again, see the section on setting up a hosted zone with Route53 in the appendices for more details.

• PostgresEngineVersion: Fuzzball supports the v13.x series of the PostgreSQL engine. New point versions within the v13.x series are regularly available, and older versions are automatically deprecated when newer versions are released. This means that the version that is pre-filled in the CloudFormation form may be deprecated. To avoid this issue, it is recommended that you run the following command and locate the latest version in the v13.x series. In this example, you would place 13.20 in the PostgresEngineVersion field.

• EnableRDSDeletionProtection: When this parameter is set to true, it will prevent the AWS RDS instance deployed by the CloudFormation template from being deleted. By default, this parameter is set to false.

• EnableRDSDRSnapshot: When this parameter is set to true, it will enable automated RDS backup replication in a secondary region. By default, this parameter is set to true.

$ aws rds describe-db-engine-versions --engine postgres \
    --query 'DBEngineVersions[].EngineVersion' --output json | \
    grep "^\s*\"13\.[0-9]*\""

    "13.15",
    "13.16",
    "13.18",
    "13.20",
    "13.21",
    "13.22",

• StandardComputeNodeInstanceType, AdvancedComputeNodeInstanceType: Instance types for which you can automatically create a node description for the provisioner once your Fuzzball cluster is deployed. The defaults we provide are a CPU and single GPU node but you can select any two instance types appropriate for your workloads. If you choose other nodes make sure your account has sufficient quotas for the node types you select. You can also add additional node types manually once Fuzzball has been deployed.

In addition, there are parameters that relate to third-party components that fuzzball utilizes within its stack. Below are these parameters with more context about the dependencies they correspond to.

• Keycloak: Fuzzball utilizes Keycloak as its main authentication provider. A Keycloak instance is deployed by the operator and torn down with the cluster. Keycloak provides fuzzball with SSO, user management, and secure authentication/authorization for Fuzzball users and services.

  • KeycloakRealmName: The name of the Keycloak realm to be created for your deployment. The realm name will also be used as the organization name by Fuzzball.

  • KeycloakOwnerEmail: The email address of the owner or administrator for the Keycloak realm.

  • KeycloakUsername: The username for the Keycloak admin account. This account will have administrative privileges within the master realm. By default, the username will be “keycloak”

  • KeycloakPassword: The password for the Keycloak admin account (KeycloakUsername).

  • KeycloakDefaultUserPassword: The default password assigned to new users created in the Keycloak realm, including the organization owner.

• Let’s Encrypt: Fuzzball utilizes Let’s Encrypt to automatically provision and manage TLS certificates, enabling secure HTTPS connections for its services. Let’s Encrypt is deployed by the operator and torn down with the cluster

  • LetsEncryptEmail: The email address that will be associated with the Let’s Encrypt account for your deployment

Once you’ve filled in the appropriate values and pressed next, you will be presented with several optional configuration parameters. You can fill these in as your use-case dictates, and then check the two acknowledgement boxes at the bottom of the page if you understand the permissions you are giving Fuzzball.

After clicking “Next” you will be presented with one more screen to review all of the parameters you provided and create your cluster.

Once you are satisfied with the options you have chosen you can select “Submit”. You will be presented with a screen to watch your cluster as it is deployed. There are several options available for watching the various services in your stacks and nested stacks as they are deployed.

During the deployment process, CloudFormation establishes all of the resources necessary to host the Fuzzball stack, and then the Fuzzball Kubernetes operator deploys the Kubernetes Orchestrate platform. These steps take about 90 minutes and 30 minutes respectively, so the entire deployment usually takes around 2 hours to complete.

Once the main stack reports that it is in the Completed state, you can begin configuring and using Fuzzball.

In order to run the deployment, we have to create a few IAM roles to establish the required permissions. Each of these roles have their AssumeRolePolicy scoped to the relevant AWS service and therefore can not be used by a user. In addition, each policy is scoped either to a specific set of resources, or requires explicit tagging in order to apply said access. Meaning the roles can only affect fuzzball resources in your account.

• ECSRunnerLambdaExecutionRole: This role defines the permissions the lambda needs to trigger the pulumi runner. It can only be assumed by the lambda service. It focuses on giving permission to start and manage ecs tasks in the ecs cluster.

• FuzzballPulumiRunnerTaskExecutionRole: This role defines the permissions needed to create the container the pulumi runner will use. It is used by the FuzzballPulumiRunnerTaskDefinition. It can only be assumed by the ecs service. It gives access to pull the image from our marketplace repository, and adds some ec2 resources to the container.

• FuzzballPulumiRunnerTaskRole: This role defines the permissions needed to run the pulumi program that deploys Fuzzball and its dependencies. Because of the varied resource footprint, it mentions many actions. The permissions themselves are broken up into multiple managed policies also included in the template (FuzzballPulumiRunnerTaskRolePolicyECRS3KMS, FuzzballPulumiRunnerTaskRolePolicySTSRDSEFS, etc). It can only be assumed by the ecs service.

Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

After fulfilling the prerequisites listed in the requirements and discovering your cluster’s domain using the domain discovery procedure, you are ready to deploy Fuzzball on CoreWeave.

The Fuzzball operator manages the deployment and lifecycle of Fuzzball on your Kubernetes cluster.

First, use your Depot credentials to authenticate with the Helm registry:

$ DEPOT_USER="your-depot-username"

$ ACCESS_KEY="your-depot-access-key"

Replace your-depot-username and your-depot-access-key with the credentials provided by CIQ.

$ helm registry login depot.ciq.com --username "${DEPOT_USER}" --password "${ACCESS_KEY}"

Install the Fuzzball operator using the Helm chart from the depot:

$ VERSION="v3.2.0"

$ CHART="oci://depot.ciq.com/fuzzball/fuzzball-images/helm/fuzzball-operator"

$ STORAGE_CLASS="shared-vast"

$ helm upgrade --install fuzzball-operator "${CHART}" \
  --namespace fuzzball-system \
  --create-namespace \
  --version "${VERSION}" \
  --set "image.tag=${VERSION}" \
  --set "imagePullSecrets.name=repository-ciq-com" \
  --set "imagePullSecrets.inline.registry=depot.ciq.com" \
  --set "imagePullSecrets.inline.username=${DEPOT_USER}" \
  --set "imagePullSecrets.inline.password=${ACCESS_KEY}" \
  --set "storageClassName=${STORAGE_CLASS}"

Check that the operator pod is running:

$ kubectl get pods -n fuzzball-system
NAME                                              READY   STATUS    RESTARTS   AGE
fuzzball-operator-controller-manager-xxxxx-xxxxx  2/2     Running   0          2m

Expected output should show the operator pod in Running state with 2/2 containers ready.

Create a FuzzballOrchestrate custom resource to deploy Fuzzball on CoreWeave. Here’s a complete example configuration:

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball-coreweave
  namespace: fuzzball-system
spec:
  # Image registry configuration
  image:
    username: <depot-username>
    password: <depot-password>
    exclusive: false

  # Database configuration
  database:
    create:
      enableDebugPod: false
      storage:
        class: shared-vast

  # Fuzzball version and cluster configuration
  fuzzball:
    version: v3.2.0
    cluster:
      name: fuzzball-coreweave

    # CoreWeave provisioner configuration
    orchestrator:
      provisioner:
        enabled: true
        coreweave:
          enabled: true
          storage:
            accessMode: ReadWriteMany
            class: shared-vast
            size: 100Gi  # Adjust based on workflow needs

    # Shared container image cache
    config:
      sharedPVC:
        accessMode: ReadWriteMany
        class: shared-vast
        size: 10Gi  # Adjust based on caching needs

  # Ingress and networking configuration
  ingress:
    create:
      # Use your discovered CoreWeave domain
      domain: <YOUR_COREWEAVE_DOMAIN>
      proxy:
        type: LoadBalancer
        annotations:
          # Public LoadBalancer for internet access
          service.beta.kubernetes.io/coreweave-load-balancer-type: public
          # Wildcard DNS for all services
          service.beta.kubernetes.io/external-hostname: '*.<YOUR_COREWEAVE_DOMAIN>'

  # Keycloak identity management
  keycloak:
    create:
      createDatabase: true
      # Generate with: uuidgen
      realmId: <uuid-v4-realm-id>
      ownerEmail: <owner-email>
      # Change this password after first login!
      defaultUserPassword: <initial-user-password>

  # TLS certificate configuration
  tls:
    # cert-manager for certificate management
    certManager:
      create: {}
    # trust-manager for CA certificate distribution
    trustManager:
      create: {}
    # Let's Encrypt certificate issuer
    ingressIssuer:
      create:
        letsEncrypt:
          email: <letsencrypt-email>
          issuer: letsencrypt-prod

Replace the placeholder values with your configuration. Use the domain discovered in the domain discovery procedure.

For detailed explanations of these configuration options and additional settings, see the CRD reference material.

Save the configuration to a file and apply it with kubectl:

$ kubectl apply -f fuzzball-orchestrate-coreweave.yaml

Watch the deployment status:

$ kubectl get fuzzballorchestrate -A -w

Wait until the status shows Ready. This typically takes several minutes for a full deployment.

Check the deployed resources:

$ kubectl get fuzzballorchestrate -A

$ kubectl get pvc -n fuzzball

$ kubectl get pods -n fuzzball

For detailed deployment status and troubleshooting, check the operator logs:

$ kubectl logs -l control-plane=fuzzball-operator-controller-manager -n fuzzball-system -f

The deployment steps above configure dynamic provisioning, where Fuzzball creates and destroys CoreWeave nodes on-demand. If you prefer to manage node pools yourself for predictable capacity or faster startup times, see Static Node Pool Provisioning for an alternative deployment approach.