Requirements

The AWS cloud marketplace listing of Fuzzball assumes that you will deploy a cluster in your own account. CIQ does not provide Fuzzball as a SaaS solution at this time. You may use either your own personal account or an organization account. The account should be dedicated to Fuzzball.

You will need administrator-level access to the AWS account. This can be the account’s root user, an IAM user with the AdministratorAccess policy attached, or access to a role with the AdministratorAccess policy attached.

You will need to submit requests to increase the following service quotas for your account in the region where you will deploy Fuzzball. For the us-east-2 region these quotas can be found on the AWS Console EC2 service quota page. Use the region dropdown to change to your desired region.

• Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances: 120 vCPUs
• Running On-Demand P instances (or other instance types you need for compute nodes): 32 vCPUs
• EC2-VPC Elastic IPs: 10

The standard instance type quota will cover the nodes used by Fuzzball Orchestrate as well as some basic compute nodes. P instances are used as the default advanced compute node instance types in the Fuzzball deployment. However, that is configurable so if you plan to use a different node type you will have to check your quotas for the corresponding instance type. In addition, you can add additional node types after deployment.

In our experience AWS may not grant the full quota requested. You will need the 10 elastic IPs as well as 120 vCPUs of standard instances.

You will need the AWS CLI installed locally and configured to access your account with sufficient permissions.

You will need an IAM user with the AdministratorAccess policy attached or an equivalent role for the installation and administration of Fuzzball. We recommend creating a dedicated IAM user for Fuzzball with the commands below. Note the commands assume that a $PW environment variable is set to temporary password for AWS console login.

$ aws iam create-user --user-name fuzzballAdmin

$ aws iam create-login-profile --user-name fuzzballAdmin --password "$PW" --password-reset-required

$ aws iam attach-user-policy --user-name fuzzballAdmin --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Once you have created the fuzzballAdmin user you can create a key for the user with the following command:

$ aws iam create-access-key --user-name fuzzballAdmin > fuzzballAdminKey.json

These keys can be used to configure AWS CLI access in a different environment or profile.

Some service-linked roles are required during the Fuzzball installation. These roles are usually created and attached to the AWS account as needed when you access the corresponding resources (e.g. when you create an EFS filesystem). However, the Fuzzball install requires these roles to be created ahead of time. This can be done with the aws cli:

$ export AWS_PAGER=""

$ aws iam create-service-linked-role --aws-service-name elasticfilesystem.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name rds.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name eks-nodegroup.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name elasticloadbalancing.amazonaws.com

$ aws iam create-service-linked-role --aws-service-name autoscaling.amazonaws.com

You can check the list of current service-linked roles with the following command:

$ aws iam list-roles --path-prefix /aws-service-role/ | jq -r '.Roles[] | .RoleName' | nl

     1  AWSServiceRoleForAmazonEKS
     2  AWSServiceRoleForAmazonEKSNodegroup
     3  AWSServiceRoleForAmazonElasticFileSystem
     4  AWSServiceRoleForAutoScaling
     5  AWSServiceRoleForAwsUserNotifications
     6  AWSServiceRoleForCloudFormationStackSetsOrgMember
     7  AWSServiceRoleForCloudTrail
     8  AWSServiceRoleForECS
     9  AWSServiceRoleForElasticLoadBalancing
    10  AWSServiceRoleForOrganizations
    11  AWSServiceRoleForRDS
    12  AWSServiceRoleForResourceExplorer
    13  AWSServiceRoleForServiceQuotas
    14  AWSServiceRoleForSSO
    15  AWSServiceRoleForSupport
    16  AWSServiceRoleForTrustedAdvisor

In order to connect to your Fuzzball cluster, you need a DNS resolvable URL where it can be hosted. You are expected to provide this by creating a hosted zone in Route53 to act as a subdomain to a domain name you own. You can either use a domain that you already own, your organization’s domain if your administrators will assist you in creating a new DNS record for your Fuzzball subdomain, or you can purchase a new domain to host your Fuzzball cluster. If you don’t already have a personal domain name, it’s easy to purchase one through Route53. Instructions for purchasing a new domain, setting up a hosted zone, and adding the appropriate NS record can be found in the appendix.

After deploying your Fuzzball cluster, you can get all of the information that you need to access it by running a convenience script provided by CIQ. This script will run all relevant commands and format the output to make it easy to read. It requires that you have set your local K8s context to point to your cloud deployment with the AWS CLI and that you have the kubectl command installed locally.

If you have an existing instance of DataDog, you can use it to monitor your Fuzzball cluster. During deployment, you can optionally provide your DataDog API key to configure access.

We find direnv to be a convenient tool for managing environments for different deployments. Here is how you would set up a .envrc file for direnv with the appropriate keys and environment variables for easier management of your Fuzzball deployment using the FuzzballAdmin IAM user we configured earlier. This requires direnv and jq to be installed and and the aws cli installed and configured with a profile for a admin user.

$ opsdir="PATH/TO/fuzzball-aws-dir"

$ mkdir -p "$opsdir"

Create keys for the fuzzballAdmin user if you don’t have them already.

$ aws iam create-access-key --user-name fuzzballAdmin >  "${opsdir}/fuzzballAdminKey.json"

$ cd "$opsdir"

# create an .envrc file that is loaded every time you enter this directory
$ cat <<__EOF__ > .envrc
AWS_ACCESS_KEY_ID=$(jq '.AccessKey.AccessKeyId' fuzzballAdminKey.json)
AWS_SECRET_ACCESS_KEY=$(jq '.AccessKey.SecretAccessKey' fuzzballAdminKey.json)
AWS_DEFAULT_REGION=XXXX
XDG_CONFIG_HOME=\$PWD/.config
KUBECONFIG=\${XDG_CONFIG_HOME}/kubernetes/kubeconfig.yaml
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION \
    XDG_CONFIG_HOME KUBECONFIG
__EOF__

$ direnv allow

Now if you run aws sts get-caller-identity you should see that you are using the new IAM user’s identity. This will also keep your Fuzzball cli and kubectl configurations confined to this directory.

The Fuzzball CLI is used to deploy Fuzzball in GCP using the command fuzzball cluster gcp deploy. See the appendix for information on installing the Fuzzball CLI.

Fuzzball is deployed into your own GCP environment. CIQ does not provide Fuzzball as a SaaS solution at this time.

You may either use a personal GCP project or one managed by your organization. You should have a project that is dedicated to running Fuzzball. Billing must be enabled within the project.

You will need to install the gcloud CLI tool and make sure you are properly authenticated using the commands below:

$ gcloud auth login

$ gcloud auth application-default login

Many of the commands that you will run to set up your GCP project and deploy Fuzzball require information about your environment. The following commands will populate environment variables that you need to proceed.

$ PROJECT_ID=$(gcloud config get-value project)

$ PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')

$ SA="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"

$ REGION=$(gcloud config get-value compute/region)

$ ZONE=$(gcloud config get-value compute/zone)

$ VERSION=v4.0.0

You can also record the following information in environment variables to aid in deployment:

$ DEPOT_USERNAME=<your depot username>

$ DEPOT_PASSWORD=<your depot password>

Your GCP user needs the following roles (or equivalent):

• Compute Admin
• Kubernetes Engine Admin
• Cloud SQL Admin
• DNS Admin
• Service Account Admin
• Storage Admin
• Cloud KMS Admin
• Infrastructure Manager Admin (roles/config.admin)

The following APIs must be enabled on your GCP project. The CLI automatically enables config.googleapis.com (Infrastructure Manager) before creating deployments, and the Terraform template auto-enables all other APIs via google_project_service resources. You can ensure that every necessary API is enabled with the following:

$ gcloud services enable \
    cloudresourcemanager.googleapis.com \
    config.googleapis.com \
    cloudbuild.googleapis.com \
    compute.googleapis.com \
    container.googleapis.com \
    sqladmin.googleapis.com \
    dns.googleapis.com \
    storage.googleapis.com \
    iam.googleapis.com \
    cloudkms.googleapis.com \
    cloudasset.googleapis.com \
    secretmanager.googleapis.com \
    run.googleapis.com \
    cloudfunctions.googleapis.com \
    artifactregistry.googleapis.com \
    servicenetworking.googleapis.com \
    file.googleapis.com \
    monitoring.googleapis.com \
    pubsub.googleapis.com

The following table gives more details about the required APIs and their purpose.

Deployments use the project’s default Compute Engine service account ({PROJECT_NUMBER}-compute@developer.gserviceaccount.com). This service account needs a number of roles. The CLI automatically grants roles/resourcemanager.projectIamAdmin and roles/secretmanager.admin to the default Compute Engine service account before creating the deployment. The deploying user must have roles/resourcemanager.projectIamAdmin (or Owner) to perform these grants. If the automatic grants fail, you can grant them manually like so:

$ for ROLE in \
    roles/config.agent \
    roles/storage.admin \
    roles/resourcemanager.projectIamAdmin \
    roles/secretmanager.admin
  do
    gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member="serviceAccount:${SA}" \
      --role="${ROLE}"
  done

The following gives details about each role.

The deployment uses a Cloud Run Pulumi runner image hosted in your GCP project’s Artifact Registry. You must create a Docker repository named fuzzball in your deployment region and push the runner image before deploying. The CLI expects the runner image to be available at {region}-docker.pkg.dev/{project}/fuzzball/gcp-pulumi-runner:{version}. You can do this using the following commands:

$ gcloud artifacts repositories create fuzzball \
  --repository-format=docker \
  --location=$REGION \
  --project=$PROJECT_ID

$ docker login depot.ciq.com \
  --username $DEPOT_USERNAME \
  --password $DEPOT_PASSWORD

$ docker pull depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION}

$ docker tag depot.ciq.com/fuzzball/fuzzball-images/gcp-pulumi-runner:${VERSION} \
  ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

$ gcloud auth configure-docker ${REGION}-docker.pkg.dev

$ docker push ${REGION}-docker.pkg.dev/${PROJECT_ID}/fuzzball/gcp-pulumi-runner:${VERSION}

The Pulumi runner automatically looks up Cloud DNS zones by domain name and creates apex and wildcard A records pointing to the Fuzzball load balancer IP. For this to function, you need the following:

• A registered domain name that you will use to host Fuzzball
• The ability to configure DNS records or nameservers

The simplest way to fulfill this requirement in GCP is to create or use an existing GCP Cloud DNS zone, and provide the zone name and project. To do so from scratch, you can issue commands like the following:

$ DOMAIN=<desired domain name>

$ gcloud services enable domains.googleapis.com --project=$PROJECT_ID

$ gcloud domains registrations search-domains ${DOMAIN} --project=$PROJECT_ID

Select a domain from the list and a name for the managed zone. Then run the following.

$ SELECTED_DOMAIN=<selected domain name>

$ MANAGED_ZONE=<selected managed zone name>

$ gcloud dns managed-zones create ${MANAGED_ZONE} \
    --dns-name="${SELECTED_DOMAIN}." \
    --description="Fuzzball testing domain" \
    --project=$PROJECT_ID

$ gcloud domains registrations register ${SELECTED_DOMAIN} --project=$PROJECT_ID

$ gcloud domains registrations describe ${SELECTED_DOMAIN} --project=$PROJECT_ID

When the domain is listed as Active, it is ready for use. You will need the values of $SELECTED_DOMAIN and $MANAGED_ZONE when you deploy Fuzzball.

The CLI uses Helm to push third-party Helm charts to your Artifact Registry during deployment. This deployment workflow requires Helm OCI registry support, including helm registry login and helm push to oci://... destinations. A local helm installation is required and must be authenticated to Artifact Registry before deploying.

Follow the Helm installation guide and install Helm >=v3.8.0 on your system.

The GCP deployment uses Infrastructure Manager (which runs Terraform via Cloud Build). The Terraform template is embedded in the CLI binary. So it is not necessary to install Terraform locally to deploy Fuzzball in GCP.

Once you’ve satisfied all of the requirements, you are ready to proceed with the deployment.

The Fuzzball CLI is used to deploy Fuzzball in Azure using the command fuzzball cluster azure deploy. See the appendix for information on installing the Fuzzball CLI.

Fuzzball is deployed into your own Azure subscription. CIQ does not provide Fuzzball as a SaaS solution at this time. You may use a personal subscription or one managed by your organization. Billing must be enabled.

You will need to install the Azure CLI and make sure you are properly authenticated using the command below:

$ az login

If you have access to multiple subscriptions, select the one you want to deploy into:

$ az account set --subscription <subscription-id>

Many of the commands that you will run to set up your Azure environment and deploy Fuzzball require information about your environment. The following commands will populate environment variables that you need to proceed.

$ SUBSCRIPTION_ID=$(az account show --query id -o tsv)

$ LOCATION=eastus

$ RESOURCE_GROUP=fuzzball

$ VERSION=v4.0.0

You can also record the following information in environment variables to aid in deployment:

$ DEPOT_USER=<your depot username>

$ DEPOT_ACCESS_TOKEN=<your depot access token>

Your Azure user (or service principal) needs the Contributor role at the subscription scope (or on the resource group you intend to deploy into). The Contributor role is sufficient because the deployment creates its own managed identities and grants them the narrowly-scoped roles each component needs. If you cannot grant Contributor at the subscription scope, scope it to the deployment resource group instead.

If you want to use Azure AD group-based admin access to the Kubernetes cluster, you will also need the Object ID of the AD group whose members should receive AKS admin access. Capture it like so:

$ AZURE_AD_ADMIN_GROUP_ID=$(az ad group show --group <group-name> --query id -o tsv)

The following resource providers must be registered on your subscription before deployment. The CLI will surface a clear error if any of these are not registered when you run deploy.

$ for RP in \
    Microsoft.Compute \
    Microsoft.ContainerService \
    Microsoft.DBforPostgreSQL \
    Microsoft.KeyVault \
    Microsoft.ManagedIdentity \
    Microsoft.Network \
    Microsoft.Storage
  do
    az provider register --namespace "$RP"
  done

The following table gives more details about each provider and its purpose.

Verify that your subscription has sufficient quota in your deployment region for the VM families you plan to use. At minimum, you will need quota for the AKS system node pool, AKS user node pool, and any compute VM families you expect to provision for workflows. The Fuzzball CLI provides a quota helper:

$ fuzzball cluster azure quota --location $LOCATION --instance-types Standard_D4s_v5,Standard_NC6s_v3

This walks through the VM families you intend to use and reports current quota vs. recommended minimums, plus the exact az quota commands to request increases when needed.

The Azure deployment supports several DNS providers for ACME (Let’s Encrypt) certificate issuance and apex/wildcard record management:

• Azure DNS (--dns-provider azuredns) — recommended when you already manage DNS in Azure
• Route53 (--dns-provider route53) — if your authoritative DNS lives in AWS
• Cloudflare (--dns-provider cloudflare)
• Google Cloud DNS (--dns-provider clouddns)

For Azure DNS, the simplest path is to create or use an existing Azure DNS zone in the same subscription:

$ DOMAIN=<your domain name>

$ DNS_ZONE_RG=<resource group containing the DNS zone>

$ az network dns zone create \
    --resource-group $DNS_ZONE_RG \
    --name $DOMAIN

Note the nameservers reported by az network dns zone show and configure them with your domain registrar before deploying.

When you deploy, pass these values via --dns-provider azuredns --dns-azuredns-subscription $SUBSCRIPTION_ID --dns-azuredns-resource-group $DNS_ZONE_RG --dns-azuredns-zone-name $DOMAIN, or let interactive mode prompt you.

You need to install kubectl so you can interact with the AKS cluster that the deployment creates. After the deployment finishes, you can configure it for your cluster with:

$ az aks get-credentials --resource-group $RESOURCE_GROUP --name <cluster-name>

Like the GCP path, the Azure deployment uses Helm to publish third-party charts to your Azure Container Registry (ACR) during deployment. Helm OCI registry support (helm registry login, helm push oci://...) is required.

Follow the Helm installation guide and install Helm

=v3.8.0 on your system.

The Azure deployment uses an ARM template plus a Pulumi runner that runs inside the deployment resource group. Both the ARM template and the Pulumi runner image are managed by the CLI — you do not need to install or run Pulumi or Bicep locally.

Once you’ve satisfied all of the requirements, you are ready to proceed with the deployment.

Support for CoreWeave within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on CoreWeave, we recommend contacting CIQ as part of your deployment planning process.

You will need access to a CoreWeave Kubernetes cluster with appropriate permissions to deploy workloads and create resources. This includes the ability to create namespaces, deploy Helm charts, and manage Custom Resource Definitions (CRDs).

You will need the kubectl command line tool installed locally and configured with appropriate cluster access to your CoreWeave Kubernetes cluster.

You will need Helm version 3.x or later installed locally for deploying the Fuzzball operator chart, and optionally the CoreWeave static provisioner chart.

The Fuzzball CLI is essential for post-deployment configuration, provisioner setup, and cluster administration. You should install the CLI before beginning the deployment process.

Installation instructions for various operating systems are available in the Installing the CLI appendix.

Before deploying Fuzzball, you need to discover the domain name for your CoreWeave cluster. CoreWeave automatically assigns domains in the format: <tenant-id>-<cluster-name>.coreweave.app

See the Domain Discovery Procedure below for detailed steps on how to determine your cluster’s domain before deployment.

CoreWeave provides shared storage accessible via the shared-vast storage class. Before deploying Fuzzball, you must request that CoreWeave configure your tenant’s Vast view to use “Native Protocol Limit” instead of the default “Lowest Common Denominator” view policy.

Before deploying Fuzzball, you must discover the domain name assigned to your CoreWeave cluster. CoreWeave does not expose the assigned domain through standard Kubernetes APIs. Instead, you can discover it by creating a temporary LoadBalancer service that triggers CoreWeave’s domain assignment.

Create a YAML file for a minimal LoadBalancer service:

# dns-discovery.yaml
apiVersion: v1
kind: Service
metadata:
  name: dns-discovery
  namespace: default
  annotations:
    service.beta.kubernetes.io/coreweave-load-balancer-type: public
    service.beta.kubernetes.io/external-hostname: '*.coreweave.app'
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: dns-discovery

Apply the service definition to your CoreWeave cluster:

$ kubectl apply -f dns-discovery.yaml

Wait approximately 30 seconds for CoreWeave to process the request.

Check the Kubernetes events to find your assigned domain:

$ kubectl get events --sort-by='.lastTimestamp' | grep dns-discovery | grep "DomainNotAllowed"

You should see output similar to:

34s   Warning   DomainNotAllowed   service/dns-discovery   Domain *.coreweave.app is not allowed, appending a1b2c3-my-cluster.coreweave.app

The domain after “appending” (a1b2c3-my-cluster.coreweave.app in this example) is your cluster’s domain.

Note down both forms of the domain:

• Base domain: a1b2c3-my-cluster.coreweave.app
• Wildcard domain: *.a1b2c3-my-cluster.coreweave.app

You will use these values when configuring your FuzzballOrchestrate resource.

Delete the test service:

$ kubectl delete service dns-discovery -n default

Support for Oracle Cloud Infrastructure (OCI) within Fuzzball is in preview status and is currently subject to more rapid change to address customer requirements than other features of Fuzzball. If you are interested in using Fuzzball on OCI, we recommend contacting CIQ as part of your deployment planning process.

You will need an Oracle Cloud Infrastructure account with Administrator permissions on the compartment where Fuzzball will be deployed. The deployment creates dynamic groups and IAM policies for instance principal authentication, which requires manage permissions on dynamic-groups and policies at the tenancy level.

Alternatively, you can create a group with the following granular IAM policies scoped to your deployment compartment:

Allow group FuzzballAdmins to manage instance-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage cluster-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage virtual-network-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage database-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage file-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage object-family in compartment <compartment_name>
Allow group FuzzballAdmins to manage dns in compartment <compartment_name>
Allow group FuzzballAdmins to manage load-balancers in compartment <compartment_name>
Allow group FuzzballAdmins to manage dynamic-groups in tenancy
Allow group FuzzballAdmins to manage policies in compartment <compartment_name>
Allow group FuzzballAdmins to manage alarms in compartment <compartment_name>
Allow group FuzzballAdmins to manage log-groups in compartment <compartment_name>
Allow group FuzzballAdmins to manage metrics in compartment <compartment_name>

Dynamic group policies must be at the tenancy level; they cannot be scoped to a compartment.

Create a dedicated compartment in your OCI tenancy for the Fuzzball deployment. All resources will be provisioned within this compartment. Using a dedicated compartment simplifies resource management and cleanup.

You will need the OCI CLI installed locally and configured with API key authentication. Run oci setup config to create a configuration file at ~/.oci/config with your tenancy OCID, user OCID, region, and API signing key.

Verify your configuration is working:

$ oci iam region list --output table

The Fuzzball CLI is essential for deploying, managing, and monitoring your Fuzzball cluster on OCI. Install the CLI before beginning the deployment process.

Installation instructions for various operating systems are available in the Installing the CLI appendix.

You will need to verify that your tenancy has sufficient service limits in the region where you will deploy Fuzzball. The following service limits should be checked and increased if necessary through the OCI Console under Governance > Limits, Quotas and Usage:

• Compute: Sufficient quota for standard shapes (e.g., VM.Standard.E4.Flex) and GPU shapes (e.g., VM.GPU.A10.1 or BM.GPU4.8) depending on your workload requirements
• VCN: At least 1 Virtual Cloud Network with adequate subnet count
• Load Balancers: At least 1 flexible load balancer
• Block Storage: Sufficient volume quota for compute and database instances
• File Storage: At least 1 file system and mount target for shared NFS storage
• Container Engine (OKE): At least 1 cluster with sufficient node pool capacity
• Database with PostgreSQL: At least 1 PostgreSQL database system

You will need a domain name that can be used to host your Fuzzball cluster. The deployment creates an OCI DNS zone and configures the necessary records. You can either use a subdomain of a domain you already own or purchase a new domain.

If you are using a domain managed outside of OCI, you will need to configure DNS delegation by adding NS records pointing to the OCI-managed DNS zone at your domain registrar after the deployment creates the DNS zone.

Before deploying Fuzzball, you should determine the domain name you will use for your cluster. If you already own a domain or subdomain, you can use it directly by providing it during the deployment configuration.

Select a domain or subdomain for your Fuzzball cluster. For example:

• fuzzball.example.com (subdomain of an existing domain)
• fuzzball-cluster.com (a new dedicated domain)

Ensure you have the ability to create DNS records for your chosen domain. If using a subdomain, confirm with your domain administrator that you can add NS records to delegate the subdomain to OCI DNS.

Note down your chosen domain. You will provide this value during the Fuzzball CLI deployment configuration when prompted for the Domain parameter.