Requirements

The AWS cloud marketplace listing of Fuzzball assumes that you will deploy a cluster in your own account. CIQ does not provide Fuzzball as a SaaS solution at this time. You may use either your own personal account or an organization account. The account should be dedicated to Fuzzball.

You will need administrator-level access to the AWS account. This can be the account’s root user, an IAM user with the AdministratorAccess policy attached, or access to a role with the AdministratorAccess policy attached.

You will need to submit requests to increase the following service quotas for your account in the region where you will deploy Fuzzball. For the us-east-2 region these quotas can be found on the AWS Console EC2 service quota page. Use the region dropdown to change to your desired region.

• Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances: 120 vCPUs
• Running On-Demand P instances (or other instance types you need for compute nodes): 32 vCPUs
• EC2-VPC Elastic IPs: 10

The standard instance type quota will cover the nodes used by Fuzzball Orchestrate as well as some basic compute nodes. P instances are used as the default advanced compute node instance types in the Fuzzball deployment. However, that is configurable so if you plan to use a different node type you will have to check your quotas for the corresponding instance type. In addition, you can add additional node types after deployment.

In our experience AWS may not grant the full quota requested. You will need the 10 elastic IPs as well as 120 vCPUs of standard instances.

You will need the AWS CLI installed locally and configured to access your account with sufficient permissions.

You will need an IAM user with the AdministratorAccess policy attached or an equivalent role for the installation and administration of Fuzzball. We recommend creating a dedicated IAM user for Fuzzball with the commands below. Note the commands assume that a $PW environment variable is set to temporary password for AWS console login.

$ aws iam create-user --user-name fuzzballAdmin

$ aws iam create-login-profile --user-name fuzzballAdmin --password "$PW" --password-reset-required

$ aws iam attach-user-policy --user-name fuzzballAdmin --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Once you have created the fuzzballAdmin user you can create a key for the user with the following command:

$ aws iam create-access-key --user-name fuzzballAdmin > fuzzballAdminKey.json

These keys can be used to configure AWS CLI access in a different environment or profile.

Some service-linked roles are required during the Fuzzball installation. These roles are usually created and attached to the AWS account as needed when you access the corresponding resources (e.g. when you create an EFS filesystem). However, the Fuzzball install requires these roles to be created ahead of time. This can be done with the aws cli:

$ export AWS_PAGER=""

$ aws iam create-service-linked-role --aws-service-name elasticfilesystem.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name rds.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks-nodegroup.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name elasticloadbalancing.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name autoscaling.amazonaws.com

You can check the list of current service-linked roles with the following command:

$ aws iam list-roles --path-prefix /aws-service-role/ | jq -r '.Roles[] | .RoleName' | nl

     1  AWSServiceRoleForAmazonEKS
     2  AWSServiceRoleForAmazonEKSNodegroup
     3  AWSServiceRoleForAmazonElasticFileSystem
     4  AWSServiceRoleForAutoScaling
     5  AWSServiceRoleForAwsUserNotifications
     6  AWSServiceRoleForCloudFormationStackSetsOrgMember
     7  AWSServiceRoleForCloudTrail
     8  AWSServiceRoleForECS
     9  AWSServiceRoleForElasticLoadBalancing
    10  AWSServiceRoleForOrganizations
    11  AWSServiceRoleForRDS
    12  AWSServiceRoleForResourceExplorer
    13  AWSServiceRoleForServiceQuotas
    14  AWSServiceRoleForSSO
    15  AWSServiceRoleForSupport
    16  AWSServiceRoleForTrustedAdvisor

In order to connect to your Fuzzball cluster, you need a DNS resolvable URL where it can be hosted. You are expected to provide this by creating a hosted zone in Route53 to act as a subdomain to a domain name you own. You can either use a domain that you already own, your organization’s domain if your administrators will assist you in creating a new DNS record for your Fuzzball subdomain, or you can purchase a new domain to host your Fuzzball cluster. If you don’t already have a personal domain name, it’s easy to purchase one through Route53. Instructions for purchasing a new domain, setting up a hosted zone, and adding the appropriate NS record can be found in the appendix.

After deploying your Fuzzball cluster, you can get all of the information that you need to access it by running a convenience script provided by CIQ. This script will run all relevant commands and format the output to make it easy to read. It requires that you have set your local K8s context to point to your cloud deployment with the AWS CLI and that you have the kubectl command installed locally.

If you have an existing instance of DataDog, you can use it to monitor your Fuzzball cluster. During deployment, you can optionally provide your DataDog API key to configure access.

We find direnv to be a convenient tool for managing environments for different deployments. Here is how you would set up a .envrc file for direnv with the appropriate keys and environment variables for easier management of your Fuzzball deployment using the FuzzballAdmin IAM user we configured earlier. This requires direnv and jq to be installed and and the aws cli installed and configured with a profile for a admin user.

$ opsdir="PATH/TO/fuzzball-aws-dir"

$ mkdir -p "$opsdir"

Create keys for the fuzzballAdmin user if you don’t have them already.

$ aws iam create-access-key --user-name fuzzballAdmin >  "${opsdir}/fuzzballAdminKey.json"

$ cd "$opsdir"

# create an .envrc file that is loaded every time you enter this directory
$ cat <<__EOF__ > .envrc
AWS_ACCESS_KEY_ID=$(jq '.AccessKey.AccessKeyId' fuzzballAdminKey.json)
AWS_SECRET_ACCESS_KEY=$(jq '.AccessKey.SecretAccessKey' fuzzballAdminKey.json)
AWS_DEFAULT_REGION=XXXX
XDG_CONFIG_HOME=\$PWD/.config
KUBECONFIG=\${XDG_CONFIG_HOME}/kubernetes/kubeconfig.yaml
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION \
    XDG_CONFIG_HOME KUBECONFIG
__EOF__

$ direnv allow

Now if you run aws sts get-caller-identity you should see that you are using the new IAM user’s identity. This will also keep your Fuzzball cli and kubectl configurations confined to this directory.

The Fuzzball CLI is used to deploy Fuzzball in GCP using the command fuzzball cluster gcp deploy. See the appendix for information on installing the Fuzzball CLI.

Fuzzball is deployed into your own GCP environment. CIQ does not provide Fuzzball as a SaaS solution at this time.

You may either use a personal GCP project or one managed by your organization. You should have a project that is dedicated to running Fuzzball. Billing must be enabled within the project.

You will need to install the gcloud CLI tool and make sure you are properly authenticated using the commands below:

$ gcloud auth login

$ gcloud auth application-default login

Many of the commands that you will run to set up your GCP project and deploy Fuzzball require information about your environment. The following commands will populate environment variables that you need to proceed.

$ PROJECT_ID=$(gcloud config get-value project)

$ PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')

$ SA="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

$ REGION=$(gcloud config get-value compute/region)

$ ZONE=$(gcloud config get-value compute/zone)

$ VERSION=v3.3.0

You can also record the following information in environment variables to aid in deployment:

$ DEPOT_USERNAME=<your depot username>

$ DEPOT_PASSWORD=<your depot password>

Your GCP user needs the following roles (or equivalent):

• Compute Admin
• Kubernetes Engine Admin
• Cloud SQL Admin
• DNS Admin
• Service Account Admin
• Storage Admin
• Cloud KMS Admin
• Infrastructure Manager Admin (roles/config.admin)

The following APIs must be enabled on your GCP project. The CLI automatically enables config.googleapis.com (Infrastructure Manager) before creating deployments, and the Terraform template auto-enables all other APIs via google_project_service resources. You can ensure that every necessary API is enabled with the following:

$ gcloud services enable \
    cloudresourcemanager.googleapis.com \
    config.googleapis.com \
    cloudbuild.googleapis.com \
    compute.googleapis.com \
    container.googleapis.com \
    sqladmin.googleapis.com \
    dns.googleapis.com \
    storage.googleapis.com \
    iam.googleapis.com \
    cloudkms.googleapis.com \
    cloudasset.googleapis.com \
    secretmanager.googleapis.com \
    run.googleapis.com \
    cloudfunctions.googleapis.com \
    artifactregistry.googleapis.com \
    servicenetworking.googleapis.com \
    file.googleapis.com \
    monitoring.googleapis.com \
    pubsub.googleapis.com

The following table gives more details about the required APIs and their purpose.

Deployments use the project’s default Compute Engine service account ({PROJECT_NUMBER}-compute@developer.gserviceaccount.com). This service account needs a number of roles. The CLI automatically grants roles/resourcemanager.projectIamAdmin and roles/secretmanager.admin to the default Compute Engine service account before creating the deployment. The deploying user must have roles/resourcemanager.projectIamAdmin (or Owner) to perform these grants. If the automatic grants fail, you can grant them manually like so:

$ for ROLE in \
    roles/config.agent \
    roles/storage.admin \
    roles/resourcemanager.projectIamAdmin \
    roles/secretmanager.admin
  do
    gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member="serviceAccount:${SA}" \
      --role="${ROLE}"
  done

The following gives details about each role.

The deployment uses a Cloud Run Pulumi runner image hosted in your GCP project’s Artifact Registry. You must create a Docker repository named fuzzball in your deployment region and push the runner image before deploying. The CLI expects the runner image to be available at {region}-docker.pkg.dev/{project}/fuzzball/gcp-pulumi-runner:{version}. You can do this using the following commands:

$ gcloud artifacts repositories create fuzzball \
  --repository-format=docker \
  --location=$REGION \
  --project=$PROJECT_ID

$ docker login depot.ciq.com \
  --username $DEPOT_USERNAME \
  --password $DEPOT_PASSWORD

$ docker pull depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION}

$ docker tag depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION} \
  ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

$ gcloud auth configure-docker ${REGION}-docker.pkg.dev

$ docker push ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

The Pulumi runner automatically looks up Cloud DNS zones by domain name and creates apex and wildcard A records pointing to the Fuzzball load balancer IP. For this to function, you need the following:

• A registered domain name that you will use to host Fuzzball
• The ability to configure DNS records or nameservers

The simplest way to fulfill this requirement in GCP is to create or use an existing GCP Cloud DNS zone, and provide the zone name and project. To do so from scratch, you can issue commands like the following:

$ DOMAIN=<desired domain name>

$ gcloud services enable domains.googleapis.com --project=$PROJECT_ID

$ gcloud domains registrations search-domains ${DOMAIN} --project=$PROJECT_ID

Select a domain from the list and a name for the managed zone. Then run the following.

$ SELECTED_DOMAIN=<selected domain name>

$ MANAGED_ZONE=<selected managed zone name>

$ gcloud dns managed-zones create ${MANAGED_ZONE} \
    --dns-name="${SELECTED_DOMAIN}." \
    --description="Fuzzball testing domain" \
    --project=$PROJECT_ID

$ gcloud domains registrations register ${SELECTED_DOMAIN} --project=$PROJECT_ID

$ gcloud domains registrations describe ${SELECTED_DOMAIN} --project=$PROJECT_ID

When the domain is listed as Active, it is ready for use. You will need the values of $SELECTED_DOMAIN and $MANAGED_ZONE when you deploy Fuzzball.

The CLI uses Helm to push third-party Helm charts to your Artifact Registry during deployment. This deployment workflow requires Helm OCI registry support, including helm registry login and helm push to oci://... destinations. A local helm installation is required and must be authenticated to Artifact Registry before deploying.

Follow the Helm installation guide and install Helm >=v3.8.0 on your system.

The GCP deployment uses Infrastructure Manager (which runs Terraform via Cloud Build). The Terraform template is embedded in the CLI binary. So it is not necessary to install Terraform locally to deploy Fuzzball in GCP.

Once you’ve satisfied all of the requirements, you are ready to proceed with the deployment.

Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

You will need access to a CoreWeave Kubernetes cluster with appropriate permissions to deploy workloads and create resources. This includes the ability to create namespaces, deploy Helm charts, and manage Custom Resource Definitions (CRDs).

You will need the kubectl command line tool installed locally and configured with appropriate cluster access to your CoreWeave Kubernetes cluster.

You will need Helm version 3.x or later installed locally for deploying the Fuzzball operator chart, and optionally the CoreWeave static provisioner chart.

The Fuzzball CLI is essential for post-deployment configuration, provisioner setup, and cluster administration. You should install the CLI before beginning the deployment process.

Installation instructions for various operating systems are available in the Installing the CLI appendix.

Before deploying Fuzzball, you need to discover the domain name for your CoreWeave cluster. CoreWeave automatically assigns domains in the format: <tenant-id>-<cluster-name>.coreweave.app

See the Domain Discovery Procedure below for detailed steps on how to determine your cluster’s domain before deployment.

CoreWeave provides shared storage accessible via the shared-vast storage class. Before deploying Fuzzball, you must request that CoreWeave configure your tenant’s Vast view to use “Native Protocol Limit” instead of the default “Lowest Common Denominator” view policy.

Before deploying Fuzzball, you must discover the domain name assigned to your CoreWeave cluster. CoreWeave does not expose the assigned domain through standard Kubernetes APIs. Instead, you can discover it by creating a temporary LoadBalancer service that triggers CoreWeave’s domain assignment.

Create a YAML file for a minimal LoadBalancer service:

# dns-discovery.yaml
apiVersion: v1
kind: Service
metadata:
  name: dns-discovery
  namespace: default
  annotations:
    service.beta.kubernetes.io/coreweave-load-balancer-type: public
    service.beta.kubernetes.io/external-hostname: '*.coreweave.app'
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: dns-discovery

Apply the service definition to your CoreWeave cluster:

$ kubectl apply -f dns-discovery.yaml

Wait approximately 30 seconds for CoreWeave to process the request.

Check the Kubernetes events to find your assigned domain:

$ kubectl get events --sort-by='.lastTimestamp' | grep dns-discovery | grep "DomainNotAllowed"

You should see output similar to:

34s   Warning   DomainNotAllowed   service/dns-discovery   Domain *.coreweave.app is not allowed, appending a1b2c3-my-cluster.coreweave.app

The domain after “appending” (a1b2c3-my-cluster.coreweave.app in this example) is your cluster’s domain.

Note down both forms of the domain:

• Base domain: a1b2c3-my-cluster.coreweave.app
• Wildcard domain: *.a1b2c3-my-cluster.coreweave.app

You will use these values when configuring your FuzzballOrchestrate resource.

Delete the test service:

$ kubectl delete service dns-discovery -n default