Requirements

The AWS cloud marketplace listing of Fuzzball assumes that you will deploy a cluster in your own account. CIQ does not provide Fuzzball as a SaaS solution at this time. You may use either your own personal account or an organization account. The account should be dedicated to Fuzzball.

You will need administrator-level access to the AWS account. This can be the account’s root user, an IAM user with the AdministratorAccess policy attached, or access to a role with the AdministratorAccess policy attached.

You will need to submit requests to increase the following service quotas for your account in the region where you will deploy Fuzzball. For the us-east-2 region these quotas can be found on the AWS Console EC2 service quota page. Use the region dropdown to change to your desired region.

• Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances: 120 vCPUs
• Running On-Demand P instances (or other instance types you need for compute nodes): 32 vCPUs
• EC2-VPC Elastic IPs: 10

The standard instance type quota will cover the nodes used by Fuzzball Orchestrate as well as some basic compute nodes. P instances (or other GPU/high-memory types you require) can be included in the InitialInstanceTypes parameter during deployment to pre-configure compute nodes. You can also add or change instance types after deployment.

In our experience AWS may not grant the full quota requested. You will need the 10 elastic IPs as well as 120 vCPUs of standard instances.

You will need the AWS CLI installed locally and configured to access your account with sufficient permissions.

You will need an IAM user with the AdministratorAccess policy attached or an equivalent role for the installation and administration of Fuzzball. We recommend creating a dedicated IAM user for Fuzzball with the commands below. Note the commands assume that a $PW environment variable is set to temporary password for AWS console login.

$ aws iam create-user --user-name fuzzballAdmin

$ aws iam create-login-profile --user-name fuzzballAdmin --password "$PW" --password-reset-required

$ aws iam attach-user-policy --user-name fuzzballAdmin --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Once you have created the fuzzballAdmin user you can create a key for the user with the following command:

$ aws iam create-access-key --user-name fuzzballAdmin > fuzzballAdminKey.json

These keys can be used to configure AWS CLI access in a different environment or profile.

Several service-linked roles are required during the Fuzzball installation. The easiest way to create any that are missing is to run the fuzzball cluster aws preflight command with the --provision-roles flag before deploying:

$ fuzzball cluster aws preflight --provision-roles

This checks all required service-linked roles and creates any that are absent. You can also run the check without provisioning (to see which roles are missing before taking action):

$ fuzzball cluster aws preflight

Manual creation (fallback): If you prefer to create the roles by hand, or if fuzzball cluster aws preflight is not yet available, use the AWS CLI:

$ export AWS_PAGER=""

$ aws iam create-service-linked-role --aws-service-name elasticfilesystem.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name rds.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks-nodegroup.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name elasticloadbalancing.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name autoscaling.amazonaws.com

You can verify the roles are present with:

$ aws iam list-roles --path-prefix /aws-service-role/ | jq -r '.Roles[] | .RoleName' | nl

     1  AWSServiceRoleForAmazonEKS
     2  AWSServiceRoleForAmazonEKSNodegroup
     3  AWSServiceRoleForAmazonElasticFileSystem
     4  AWSServiceRoleForAutoScaling
     5  AWSServiceRoleForAwsUserNotifications
     6  AWSServiceRoleForCloudFormationStackSetsOrgMember
     7  AWSServiceRoleForCloudTrail
     8  AWSServiceRoleForECS
     9  AWSServiceRoleForElasticLoadBalancing
    10  AWSServiceRoleForOrganizations
    11  AWSServiceRoleForRDS
    12  AWSServiceRoleForResourceExplorer
    13  AWSServiceRoleForServiceQuotas
    14  AWSServiceRoleForSSO
    15  AWSServiceRoleForSupport
    16  AWSServiceRoleForTrustedAdvisor

In order to connect to your Fuzzball cluster, you need a DNS resolvable URL where it can be hosted. You are expected to provide this by creating a hosted zone in Route53 to act as a subdomain to a domain name you own. You can either use a domain that you already own, your organization’s domain if your administrators will assist you in creating a new DNS record for your Fuzzball subdomain, or you can purchase a new domain to host your Fuzzball cluster. If you don’t already have a personal domain name, it’s easy to purchase one through Route53. Instructions for purchasing a new domain, setting up a hosted zone, and adding the appropriate NS record can be found in the appendix.

After deploying your Fuzzball cluster, you can get all of the information that you need to access it by running a convenience script provided by CIQ. This script will run all relevant commands and format the output to make it easy to read. It requires that you have set your local K8s context to point to your cloud deployment with the AWS CLI and that you have the kubectl command installed locally.

If you have an existing instance of DataDog, you can use it to monitor your Fuzzball cluster. During deployment, you can optionally provide your DataDog API key to configure access.

We find direnv to be a convenient tool for managing environments for different deployments. Here is how you would set up a .envrc file for direnv with the appropriate keys and environment variables for easier management of your Fuzzball deployment using the FuzzballAdmin IAM user we configured earlier. This requires direnv and jq to be installed and and the aws cli installed and configured with a profile for a admin user.

$ opsdir="PATH/TO/fuzzball-aws-dir"

$ mkdir -p "$opsdir"

Create keys for the fuzzballAdmin user if you don’t have them already.

$ aws iam create-access-key --user-name fuzzballAdmin >  "${opsdir}/fuzzballAdminKey.json"

$ cd "$opsdir"

# create an .envrc file that is loaded every time you enter this directory
$ cat <<__EOF__ > .envrc
AWS_ACCESS_KEY_ID=$(jq '.AccessKey.AccessKeyId' fuzzballAdminKey.json)
AWS_SECRET_ACCESS_KEY=$(jq '.AccessKey.SecretAccessKey' fuzzballAdminKey.json)
AWS_DEFAULT_REGION=XXXX
XDG_CONFIG_HOME=\$PWD/.config
KUBECONFIG=\${XDG_CONFIG_HOME}/kubernetes/kubeconfig.yaml
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION \
    XDG_CONFIG_HOME KUBECONFIG
__EOF__

$ direnv allow

Now if you run aws sts get-caller-identity you should see that you are using the new IAM user’s identity. This will also keep your Fuzzball cli and kubectl configurations confined to this directory.

The Fuzzball CLI is used to deploy Fuzzball in GCP using the command fuzzball cluster gcp deploy. See the appendix for information on installing the Fuzzball CLI.

Fuzzball is deployed into your own GCP environment. CIQ does not provide Fuzzball as a SaaS solution at this time.

You may either use a personal GCP project or one managed by your organization. You should have a project that is dedicated to running Fuzzball. Billing must be enabled within the project.

You will need to install the gcloud CLI tool and make sure you are properly authenticated using the commands below:

$ gcloud auth login

$ gcloud auth application-default login

Many of the commands that you will run to set up your GCP project and deploy Fuzzball require information about your environment. The following commands will help you populate environment variables that you need to proceed.

First, list the projects that you can access. Make note of the PROJECT_ID and the PROJECT_NUMBER where you want to deploy Fuzzball.

$ gcloud projects list --format='table(name,projectId,projectNumber)'
NAME          PROJECT_ID     PROJECT_NUMBER
my-project1   my-proj-1234   012345678901
my-project2   my-proj-3456   987654321098

The project NAME and PROJECT_ID may be the same in some cases, but this is arbitrary.

$ PROJECT_ID=<desired project ID from above>

$ PROJECT_NUMBER=<desired project number from above>

Assuming your local gcloud config files are set up appropriately, you can fill in the following environment variables automatically. If the commands fail, follow the instructions in the terminal to set these values in your local config.

$ SA="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

$ REGION=$(gcloud config get-value compute/region)

$ ZONE=$(gcloud config get-value compute/zone)

$ VERSION=v3.4.0

You can also record the following information in environment variables to aid in deployment:

$ DEPOT_USERNAME=<your depot username>

$ DEPOT_PASSWORD=<your depot password>

Your GCP user needs the following roles (or equivalent):

• Compute Admin
• Kubernetes Engine Admin
• Cloud SQL Admin
• DNS Admin
• Service Account Admin
• Storage Admin
• Cloud KMS Admin
• Infrastructure Manager Admin (roles/config.admin)

The following APIs must be enabled on your GCP project. The CLI automatically enables config.googleapis.com (Infrastructure Manager) before creating deployments, and the Terraform template auto-enables all other APIs via google_project_service resources. You can ensure that every necessary API is enabled with the following:

$ gcloud services enable \
    cloudresourcemanager.googleapis.com \
    config.googleapis.com \
    cloudbuild.googleapis.com \
    compute.googleapis.com \
    container.googleapis.com \
    sqladmin.googleapis.com \
    dns.googleapis.com \
    storage.googleapis.com \
    iam.googleapis.com \
    cloudkms.googleapis.com \
    cloudasset.googleapis.com \
    secretmanager.googleapis.com \
    run.googleapis.com \
    cloudfunctions.googleapis.com \
    artifactregistry.googleapis.com \
    servicenetworking.googleapis.com \
    file.googleapis.com \
    monitoring.googleapis.com \
    pubsub.googleapis.com

The following table gives more details about the required APIs and their purpose.

Deployments use the project’s default Compute Engine service account ({PROJECT_NUMBER}-compute@developer.gserviceaccount.com). This service account needs a number of roles. The CLI automatically grants roles/resourcemanager.projectIamAdmin and roles/secretmanager.admin to the default Compute Engine service account before creating the deployment. The deploying user must have roles/resourcemanager.projectIamAdmin (or Owner) to perform these grants. If the automatic grants fail, you can grant them manually like so:

$ for ROLE in \
    roles/config.agent \
    roles/storage.admin \
    roles/resourcemanager.projectIamAdmin \
    roles/secretmanager.admin
  do
    gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member="serviceAccount:${SA}" \
      --role="${ROLE}"
  done

The following gives details about each role.

The deployment uses a Cloud Run Pulumi runner image hosted in your GCP project’s Artifact Registry. You must create a Docker repository named fuzzball in your deployment region and push the runner image before deploying. The CLI expects the runner image to be available at {region}-docker.pkg.dev/{project}/fuzzball/gcp-pulumi-runner:{version}. You can do this using the following commands:

$ gcloud artifacts repositories create fuzzball \
  --repository-format=docker \
  --location=$REGION \
  --project=$PROJECT_ID

$ docker login depot.ciq.com \
  --username $DEPOT_USERNAME \
  --password $DEPOT_PASSWORD

$ docker pull depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION}

$ docker tag depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION} \
  ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

$ gcloud auth configure-docker ${REGION}-docker.pkg.dev

$ docker push ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

Fuzzball uses a custom container runtime called Substrate on compute nodes. In GCP, Fuzzball needs access to OS images that have Substrate pre-installed. You should download these images from CIQ Depot, create a GCS bucket, upload the images, and import them as Compute Engine images to make them usable within GCP. Use the following commands.

First, navigate to CIQ Depot and get links to download the two tarballs corresponding to the normal node and GPU enabled node images.

Set the substrate version in an environment variable to make the remaining commands easier to run:

$ SUBSTRATE_VERSION=<version from Depot>

The download commands will look similar to this, but your URLs will be different because they include a token generated specifically for you.

$ wget https://ciq:<token>@depot.ciq.dev/my/fuzzball/fuzzball-substrate-os-images/fuzzball-substrate-rlc-pro-9-7-${SUBSTRATE_VERSION}.tar.gz

$ wget https://ciq:<token>@depot.ciq.dev/my/fuzzball/fuzzball-substrate-os-images/fuzzball-substrate-rlc-pro-9-7-nvidia-${SUBSTRATE_VERSION}.tar.gz

Now you can create a GCS bucket with the following command. Do not change the name of the bucket.

$ gcloud storage buckets create gs://${PROJECT_ID}-substrate-images \
    --project=${PROJECT_ID} \
    --location=${REGION}

After the bucket has been created you are ready to upload the tarballs that you downloaded from CIQ Depot.

$ gcloud storage cp fuzzball-substrate-rlc-pro-9-7-${SUBSTRATE_VERSION}.tar.gz \
    gs://${PROJECT_ID}-substrate-images/

$ gcloud storage cp fuzzball-substrate-rlc-pro-9-7-nvidia-${SUBSTRATE_VERSION}.tar.gz \
    gs://${PROJECT_ID}-substrate-images/

Now you can import them as images ready to use with GCP:

$ gcloud compute images create fuzzball-substrate-rlc-pro-9-7-${SUBSTRATE_VERSION} \
    --project=${PROJECT_ID} \
    --source-uri=gs://${PROJECT_ID}-substrate-images/fuzzball-substrate-rlc-pro-9-7-${SUBSTRATE_VERSION}.tar.gz \
    --family=fuzzball-substrate-rlc-pro-9-7 \
    --guest-os-features=VIRTIO_SCSI_MULTIQUEUE,UEFI_COMPATIBLE,GVNIC

$ gcloud compute images create fuzzball-substrate-rlc-pro-9-7-nvidia-${SUBSTRATE_VERSION} \
    --project=${PROJECT_ID} \
    --source-uri=gs://${PROJECT_ID}-substrate-images/fuzzball-substrate-rlc-pro-9-7-nvidia-${SUBSTRATE_VERSION}.tar.gz \
    --family=fuzzball-substrate-rlc-pro-9-7-nvidia \
    --guest-os-features=VIRTIO_SCSI_MULTIQUEUE,UEFI_COMPATIBLE,GVNIC

Fuzzball Orchestrate recognizes the images to pull for compute nodes by the image family name. The name of the actual tarballs used to create these images does not matter, but the --family values in the commands above cannot be changed.

The Pulumi runner automatically looks up Cloud DNS zones by domain name and creates apex and wildcard A records pointing to the Fuzzball load balancer IP. For this to function, you need the following:

• A registered domain name that you will use to host Fuzzball
• The ability to configure DNS records or nameservers

The simplest way to fulfill this requirement in GCP is to create or use an existing GCP Cloud DNS zone, and provide the zone name and project. To do so from scratch, you can issue commands like the following:

$ DOMAIN=<desired domain name>

$ gcloud services enable domains.googleapis.com --project=$PROJECT_ID

$ gcloud domains registrations search-domains ${DOMAIN} --project=$PROJECT_ID

Select a domain from the list and a name for the managed zone. Then run the following.

$ SELECTED_DOMAIN=<selected domain name>

$ MANAGED_ZONE=<selected managed zone name>

$ gcloud dns managed-zones create ${MANAGED_ZONE} \
    --dns-name="${SELECTED_DOMAIN}." \
    --description="Fuzzball testing domain" \
    --project=$PROJECT_ID

$ gcloud domains registrations register ${SELECTED_DOMAIN} --project=$PROJECT_ID

$ gcloud domains registrations describe ${SELECTED_DOMAIN} --project=$PROJECT_ID

When the domain is listed as Active, it is ready for use. You will need the values of $SELECTED_DOMAIN and $MANAGED_ZONE when you deploy Fuzzball.

The CLI uses Helm to push third-party Helm charts to your Artifact Registry during deployment. This deployment workflow requires Helm OCI registry support, including helm registry login and helm push to oci://... destinations. A local helm installation is required and must be authenticated to Artifact Registry before deploying.

Follow the Helm installation guide and install Helm >=v3.8.0 on your system.

The GCP deployment uses Infrastructure Manager (which runs Terraform via Cloud Build). The Terraform template is embedded in the CLI binary. So it is not necessary to install Terraform locally to deploy Fuzzball in GCP.

Once you’ve satisfied all of the requirements, you are ready to proceed with the deployment.

Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

You will need access to a CoreWeave Kubernetes cluster with appropriate permissions to deploy workloads and create resources. This includes the ability to create namespaces, deploy Helm charts, and manage Custom Resource Definitions (CRDs).

You will need the kubectl command line tool installed locally and configured with appropriate cluster access to your CoreWeave Kubernetes cluster.

You will need Helm version 3.x or later installed locally for deploying the Fuzzball operator chart, and optionally the CoreWeave static provisioner chart.

The Fuzzball CLI is essential for post-deployment configuration, provisioner setup, and cluster administration. You should install the CLI before beginning the deployment process.

Installation instructions for various operating systems are available in the Installing the CLI appendix.

Before deploying Fuzzball, you need to discover the domain name for your CoreWeave cluster. CoreWeave automatically assigns domains in the format: <tenant-id>-<cluster-name>.coreweave.app

See the Domain Discovery Procedure below for detailed steps on how to determine your cluster’s domain before deployment.

CoreWeave provides shared storage accessible via the shared-vast storage class. Before deploying Fuzzball, you must request that CoreWeave configure your tenant’s Vast view to use “Native Protocol Limit” instead of the default “Lowest Common Denominator” view policy.

Before deploying Fuzzball, you must discover the domain name assigned to your CoreWeave cluster. CoreWeave does not expose the assigned domain through standard Kubernetes APIs. Instead, you can discover it by creating a temporary LoadBalancer service that triggers CoreWeave’s domain assignment.

Create a YAML file for a minimal LoadBalancer service:

# dns-discovery.yaml
apiVersion: v1
kind: Service
metadata:
  name: dns-discovery
  namespace: default
  annotations:
    service.beta.kubernetes.io/coreweave-load-balancer-type: public
    service.beta.kubernetes.io/external-hostname: '*.coreweave.app'
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: dns-discovery

Apply the service definition to your CoreWeave cluster:

$ kubectl apply -f dns-discovery.yaml

Wait approximately 30 seconds for CoreWeave to process the request.

Check the Kubernetes events to find your assigned domain:

$ kubectl get events --sort-by='.lastTimestamp' | grep dns-discovery | grep "DomainNotAllowed"

You should see output similar to:

34s   Warning   DomainNotAllowed   service/dns-discovery   Domain *.coreweave.app is not allowed, appending a1b2c3-my-cluster.coreweave.app

The domain after “appending” (a1b2c3-my-cluster.coreweave.app in this example) is your cluster’s domain.

Note down both forms of the domain:

• Base domain: a1b2c3-my-cluster.coreweave.app
• Wildcard domain: *.a1b2c3-my-cluster.coreweave.app

You will use these values when configuring your FuzzballOrchestrate resource.

Delete the test service:

$ kubectl delete service dns-discovery -n default

Support for Oracle Cloud Infrastructure (OCI) within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on OCI, we recommend contacting CIQ as part of your deployment planning process.

You will need an Oracle Cloud Infrastructure account with Administrator permissions on the compartment where Fuzzball will be deployed. The deployment creates dynamic groups and IAM policies for instance principal authentication, which requires manage permissions on dynamic-groups and policies at the tenancy level.

Alternatively, you can create a group with the following granular IAM policies scoped to your deployment compartment:

Allow group FuzzballAdmins to manage instance-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage cluster-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage virtual-network-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage database-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage file-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage object-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage dns in compartment <compartment_name>
Allow group FuzzballAdmins to manage load-balancers in compartment <compartment_name>
Allow group FuzzballAdmins to manage dynamic-groups in tenancy
Allow group FuzzballAdmins to manage policies in compartment <compartment_name>
Allow group FuzzballAdmins to manage alarms in compartment <compartment_name>
Allow group FuzzballAdmins to manage log-groups in compartment <compartment_name>
Allow group FuzzballAdmins to manage metrics in compartment <compartment_name>

Dynamic group policies must be at the tenancy level; they cannot be scoped to a compartment.

Create a dedicated compartment in your OCI tenancy for the Fuzzball deployment. All resources will be provisioned within this compartment. Using a dedicated compartment simplifies resource management and cleanup.

You will need the OCI CLI installed locally and configured with API key authentication. Run oci setup config to create a configuration file at ~/.oci/config with your tenancy OCID, user OCID, region, and API signing key.

Verify your configuration is working:

$ oci iam region list --output table

The Fuzzball CLI is essential for deploying, managing, and monitoring your Fuzzball cluster on OCI. Install the CLI before beginning the deployment process.

Installation instructions for various operating systems are available in the Installing the CLI appendix.

You will need to verify that your tenancy has sufficient service limits in the region where you will deploy Fuzzball. The following service limits should be checked and increased if necessary through the OCI Console under Governance > Limits, Quotas and Usage:

• Compute: Sufficient quota for standard shapes (e.g., VM.Standard.E4.Flex) and GPU shapes (e.g., VM.GPU.A10.1 or BM.GPU4.8) depending on your workload requirements
• VCN: At least 1 Virtual Cloud Network with adequate subnet count
• Load Balancers: At least 1 flexible load balancer
• Block Storage: Sufficient volume quota for compute and database instances
• File Storage: At least 1 file system and mount target for shared NFS storage
• Container Engine (OKE): At least 1 cluster with sufficient node pool capacity
• Database with PostgreSQL: At least 1 PostgreSQL database system

You will need a domain name that can be used to host your Fuzzball cluster. The deployment creates an OCI DNS zone and configures the necessary records. You can either use a subdomain of a domain you already own or purchase a new domain.

If you are using a domain managed outside of OCI, you will need to configure DNS delegation by adding NS records pointing to the OCI-managed DNS zone at your domain registrar after the deployment creates the DNS zone.

Before deploying Fuzzball, you should determine the domain name you will use for your cluster. If you already own a domain or subdomain, you can use it directly by providing it during the deployment configuration.

Select a domain or subdomain for your Fuzzball cluster. For example:

• fuzzball.example.com (subdomain of an existing domain)
• fuzzball-cluster.com (a new dedicated domain)

Ensure you have the ability to create DNS records for your chosen domain. If using a subdomain, confirm with your domain administrator that you can add NS records to delegate the subdomain to OCI DNS.

Note down your chosen domain. You will provide this value during the Fuzzball CLI deployment configuration when prompted for the Domain parameter.