Connecting External Static Compute Nodes

A cloud Fuzzball Orchestrate deployment normally provisions its own compute nodes on demand from the corresponding cloud provider. In addition it is possible to attach compute nodes that Fuzzball does not provision — for example on-premises GPU servers, bare-metal machines, or instances in another cloud — and have Orchestrate schedule jobs onto them alongside its dynamically provisioned nodes. These are referred to as external static nodes: they run the Substrate agent, join the cluster over a network you control, and are never created or destroyed by Fuzzball.

This guide describes the architecture, the network connectivity that must exist between the nodes and Orchestrate, and an end-to-end procedure using AWS (EKS) and Tailscale as a concrete example. The same approach applies to the other supported clouds.

Generating the external node configuration requires the deployment’s dynamic provisioner to be enabled (spec.fuzzball.orchestrator.provisioner.enabled in the FuzzballOrchestrate resource). The generate-substrate-config command reads the substrate configuration that the operator renders only when that provisioner is enabled. This is the default for cloud deployments.

An external static node runs Substrate and communicates with Orchestrate in two directions:

• Node → Orchestrate. On startup the node fetches the cluster’s service endpoints from the public .well-known/services endpoint of the substrate-bridge service, then connects to NATS to self-register and receive work. NATS advertises the cluster’s private IP addresses, so the node must be able to route to the cluster’s internal network.

• Orchestrate → Node. Orchestrate dials back to the node on the Substrate gRPC port(s) to grant leases and to drive operations such as ResourceShow and PullJobImage. Each Substrate publishes its dial-back address as part of its NATS client name:

  client_id = hostname/<source-IP-of-NATS-traffic>/7331
  

  <source-IP-of-NATS-traffic> is the local IP address the node’s kernel sources from when it reaches the NATS host. For the dial-back to succeed, that address must be routable from the cluster back to the node. With a VPN such as Tailscale, this is the node’s VPN address, so the cluster must have a route to the VPN address range.

Because Orchestrate must reach nodes on private addresses and nodes must reach Orchestrate on private addresses, a bidirectional overlay network (VPN) between the two is the simplest way to satisfy both directions.

Fuzzball assumes that all of the external nodes of a node segment (a named group of nodes that share resources - configured in Step 4) able to share work also share an on-disk image cache, and it will use the cheapest available node to pull and convert container images. Configure your set of external nodes so that:

• The image directory is located on a filesystem shared by all of the external nodes of a node segment (for example an NFS mount). This lets one node pull and convert an image that the others can then run.

If nodes that can be scheduled together do not share an image cache, jobs may fail when a node cannot find an image another node pulled. A similar caveat applies to volume provisioning (see Update the Orchestrate configuration).

Regardless of which VPN or overlay you use, the following must be reachable:

In addition, the node must be able to resolve DNS for the substrate-bridge host, and the cluster must have a route back to the node’s source address (see Architecture).

Use the generate-substrate-config subcommand for your cloud provider to extract the Substrate and orchestrate-extension configuration from the running deployment. It is available under each cloud command:

fuzzball cluster <CLOUDPROVIDER> generate-substrate-config

The examples below use AWS. The command connects to the deployment, reads the required information and renders static configuration files in the output directory:

$ fuzzball cluster aws generate-substrate-config ./substrate-config --stack-name my-fuzzball \
    --image-dir /data/image

Useful flags:

• --stack-name — the deployment (CloudFormation stack) to read from. If omitted you are prompted to select one. (--profile and --region select the AWS account and region.)
• --image-dir — override the substrate image cache directory (services.image.directory). Set this to the shared filesystem path your external nodes mount.
• --namespace/-n — Kubernetes namespace of the Fuzzball workload (default fuzzball).
• --force/-f — overwrite files in a non-empty output directory.

The output directory contains four files:

substrate-config/
  fuzzball-substrate.yaml    # main substrate configuration
  orchestrate.yaml           # orchestrate extension config (embeds the NATS credentials)
  ca.crt                     # debugging aid only
  nginx-ca.crt               # debugging aid only

ca.crt and nginx-ca.crt are provided for inspection only. Both certificate authorities are already embedded in the configuration URLs, so you do not need to copy the .crt files to the nodes.

The command prints per-node next steps when it finishes; the following sections walk through them in detail.

Each external node needs the connectivity described in Network connectivity requirements. In general you must:

1. Advertise the cluster’s private subnet range to the nodes, so they can reach NATS and other internal services.
2. Route the nodes’ address range back into the cluster’s subnets, so Orchestrate’s dial-back reaches the nodes.
3. Allow the required ports through any firewalls in between.

How you accomplish this depends on your environment. The rest of this section shows one concrete implementation using Tailscale with an AWS-deployed Orchestrate.

In this example the static nodes have public egress so they can reach the public .well-known/services endpoint directly. A Tailnet carries the private traffic: it lets Substrate reach the cluster’s private NATS address, and lets Orchestrate dial back to the node’s Substrate gRPC ports. On nodes routing to AWS NATS over Tailscale, the source address is the node’s Tailscale IP (from the 100.64.0.0/10 range). The EKS pod network does not know about that range by default, so a single router instance plus one VPC route entry per subnet bridges the two networks:

                   advertise 10.0.0.0/16        accept 10.0.0.0/16 route
                   ─────────────────────────►   ─────────────────────────►
                  ┌───────────────────────────┐                        ┌──────────────┐
EKS pods ◄───────►│ EC2 (Tailscale router)    │◄──── tailnet ─────────►│ external     │
   (VPC subnets)  │  • tailscaled --advertise │  (encrypted, WireGuard)│   substrate  │
                  │  • IP forwarding on       │                        │   tag:fuzzball-static
                  │  • src/dst check off      │                        └──────────────┘
                  └────────────┬──────────────┘
                               │
   VPC route table:            │  100.64.0.0/10 → eni-...
   100.64.0.0/10 → eni-of-EC2  ▼
                  (added to every subnet EKS nodes run in)

The CIDRs below are examples; substitute your VPC CIDR and Tailnet range.

Launch a small instance in your Orchestrate VPC, on one of the subnets used by the EKS cluster, with the security group applied to EKS nodes and the key pair associated with the cluster:

1. List key pairs and pick the one associated with your active cluster:

$ aws ec2 describe-key-pairs

2. List the subnets associated with the EKS cluster of your Fuzzball deployment:

$ kluster=YOUR_CLUSTER_NAME
$ subnets=( $(aws eks describe-cluster --name $kluster \
    --query 'cluster.resourcesVpcConfig.subnetIds' | jq -r '. | @tsv') )

3. Find the EKS node security group on one of the subnets:

$ aws ec2 describe-network-interfaces \
  --filters "Name=subnet-id,Values=${subnets[0]}" \
  --query 'NetworkInterfaces[].Groups[].[GroupId,GroupName]' \
  --output text | sort -u | grep 'eks-cluster'

4. Launch a small instance to route from your EKS cluster subnet to your tailnet in any of the EKS subnets:

$ aws ec2 run-instances \
  --image-id ami-... \
  --instance-type t3.micro \
  --subnet-id subnet-... \
  --security-group-ids sg-... \
  --key-name your-key \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fuzzball-tailscale-router}]' \
| tee fuzzball-tailscale-router-instance.json

$ router_id="$(jq -r '.Instances[0].InstanceId' fuzzball-tailscale-router-instance.json)"
$ router_eni="$(jq -r '.Instances[0].NetworkInterfaces[0].NetworkInterfaceId' fuzzball-tailscale-router-instance.json)"

5. Disable the source/destination check on the router’s network interface so it can forward traffic that is not addressed to itself:

$ aws ec2 modify-instance-attribute --instance-id $router_id --no-source-dest-check

6. Connect to the instance to configure it. If it has no public interface, you can use the fuzzball-admin-0 pod as a bastion. That pod is a StatefulSet the Fuzzball operator deploys into the Orchestrate namespace as an in-VPC break-glass shell; it has SSH keys mounted at /app/.ssh/fuzzball/id_rsa that authenticate against the same key pair the operator provisions EC2 instances with. Note that IPs and IP ranges are examples and will have to be adjusted for your deployment.

$ kubectl exec -n fuzzball fuzzball-admin-0 -it -- /bin/sh

(admin-0)# ssh -i /app/.ssh/fuzzball/id_rsa ec2-user@10.0.84.247 # from inside the pod (ec2-user is the default Amazon Linux user)

On the router instance, install Tailscale, enable IP forwarding, and advertise the VPC CIDR:

(router)# curl -fsSL https://tailscale.com/install.sh | sh

(router)$ cat <<'EOF' | sudo tee /etc/sysctl.d/99-tailscale.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

(router)$ sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

(router)$ sudo tailscale up \
  --advertise-routes=10.0.0.0/16 \
  --hostname=fuzzball-tailscale-router \
  --auth-key=...

Note that the auth key for joining systems to your tailnet is created in the Tailscale console.

7. Approve the advertised route in the Tailscale admin console, apply the fuzzball-router tag to this node, and set an ACL/tag policy that permits the required traffic:

{
  "tagOwners": {
    "tag:fuzzball-router": ["..."],
    "tag:fuzzball-static": ["..."]
  },
  "acls": [
    // external substrates reach the VPC (NATS, internal DNS, etc.)
    { "action": "accept", "src": ["tag:fuzzball-static"], "dst": ["10.0.0.0/16:*"] },
    // external substrates can reach each other
    { "action": "accept", "src": ["tag:fuzzball-static"], "dst": ["tag:fuzzball-static:*"] },
    // orchestrate (via the router) reaches static substrates on the substrate gRPC ports
    { "action": "accept", "src": ["tag:fuzzball-router"], "dst": ["tag:fuzzball-static:7331,7332,7333"] }
  ]
}

Finally, add a route in each EKS subnet’s route table sending the Tailnet CIDR to the router’s ENI:

for s in ${subnets[@]} ; do
  table="$(aws ec2 describe-route-tables \
    --filters "Name=association.subnet-id,Values=$s" \
  | jq -r '.RouteTables[] | .RouteTableId')"
  aws ec2 create-route --route-table-id $table \
    --destination-cidr-block 100.64.0.0/10 --network-interface-id $router_eni
done

If a node’s hostname does not match a pattern you can use to identify it in the cluster configuration, set one that does before joining the Tailnet:

(node)$ sudo hostnamectl set-hostname static-001

Install Tailscale and accept the advertised routes, then tag the node fuzzball-static in the Tailscale admin console.

(node)$ curl -fsSL https://tailscale.com/install.sh | sudo sh
(node)$ sudo tailscale up --auth-key=... --accept-routes

Depending on how this node is configured you may have to configure your firewall to accept the required incoming traffic and accept forwards.

If your nodes have Docker installed, the DOCKER chains in the packet-filter tables can interfere with Tailscale. Since these hosts will be dedicated Fuzzball compute nodes, uninstalling Docker and rebooting or flushing packet-filter rules may be necessary.

From the external node:

(node)$ tailscale status
(node)$ tailscale ping fuzzball-tailscale-router

# Confirm the node can reach NATS on the private cluster address (port 4222).
# Obtain the NATS address from the well-known endpoint; the response is protobuf,
# so pipe through `strings` to find the :4222 endpoints:
#   curl -sku '<basic-auth-from-orchestrate.yaml>' \
#     https://substrate-bridge.<domain>/.well-known/services | strings
(node)$ nc -zv 10.0.X.X 4222

On each external node, install the Substrate runtime, the orchestrate extension, and any device plugins you need (for example the NVIDIA GPU plugin), but do not start the service yet. Packages are available from Depot; use the versions that match your Orchestrate release. The example below uses deb packages.

(node)$ sudo apt install \
  ./fuzzball-substrate-<version>.linux-amd64.deb \
  ./fuzzball-substrate-orchestrate_<version>_amd64.deb \
  ./fuzzball-substrate-nvidia-gpu-plugin-<version>.amd64.deb

Create the image and compute directories chosen in Step 1. The image directory should be on the filesystem shared across your external nodes; the compute directory is node-local:

(node)$ sudo mkdir -p /data/image   # shared filesystem

Install the two configuration files generated in Step 1. The Substrate package owns /etc/fuzzball-substrate/ and ships a default device-plugins.yml — leave that file in place:

(node)$ sudo install -m 0644 substrate-config/fuzzball-substrate.yaml \
    /etc/fuzzball-substrate/fuzzball-substrate.yml
(node)$ sudo install -d -m 0755 /etc/fuzzball-substrate/extension.conf.d
(node)$ sudo install -m 0600 substrate-config/orchestrate.yaml \
    /etc/fuzzball-substrate/extension.conf.d/orchestrate.yaml

The Substrate package installs a default configuration. Make sure you deploy the generated configuration files after installing or updating the package so your settings are not overwritten by the defaults.

Do not start Substrate until the cluster configuration has been updated in the next step.

These changes must be made as a cluster administrator. The CLI is shown here; the same changes can be made in the web UI.

First save the current configuration so you can refer back to it:

$ fuzzball cluster config get | tee cluster_config.orig.yaml

A cloud deployment’s node definitions typically reference the cloud provisioner, for example:

definitions:
  - id: t3.small
    provisioner: aws
    provisionerSpec:
      instanceType: t3.small
  - id: t3.large
    provisioner: aws
    provisionerSpec:
      instanceType: t3.large

Replace (or supplement) these with a static provisioner definition whose condition matches your external nodes. The example below matches hosts named static-000 through static-999. Use named segments to specify topology of nodes required for Fuzzball to locate image pulls and volume placement.

$ cat > new_cluster_config.yaml <<'EOF'
definitions:
  - id: external-static
    segment: external-static
    provisioner: static
    provisionerSpec:
      costPerHour: 0.01
      condition: |-
        hostname() matches "static-[0-9]{3}"
  - id: t3.small
    provisioner: aws
    segment: aws
    provisionerSpec:
      instanceType: t3.small
  - id: t3.large
    provisioner: aws
    segment: aws
    provisionerSpec:
      instanceType: t3.large
EOF

$ fuzzball cluster config set new_cluster_config.yaml

The hostname match is only an example — see the Central Configuration reference for the full condition expression syntax.

Like node provisioners, storage volume provisioners can be associated with a segment to indicate that it is limited to a subset (a segment) of nodes. Unlike for node provisioners, the segment of a volume provisioner is immutable after creation. So if there are pre-existing volume provisioners that are specific to your cloud resources you need to delete and re-create them. Start by listing your provisioners:

$ fuzzball volume provisioner list

ID                                   | NAME         | STATUS | DETAIL | CLUSTER
00abf6b2-d5dc-3664-96bc-fa9787282e57 | default      | Ready  |        | my.fuzzball

If any of the existing provisioners need to be restricted to the cloud resources you can dump them to a file, remove them, and then re-create them.

$ fuzzball volume provisioner info default > default.yaml

$ cat default.yaml
default:
  description: Default EFS Storage (built-in)
  driver:
    type: efs
    filesystemId: fs-xxxxxxxxxxxxx
    region: us-east-2
  access: all
  create:
  - admin@ciq.com
  ephemeral: all

$ echo "  segment: aws" >> default.yaml

If there are no volumes yet attached to the provisioner deleting and re-creating is straight forward:

$ fuzzball volume provisioner remove default

$ fuzzball volume provisioner add default -f default.yaml

If volumes are already attached you will have to ignore them and then re-scan for volumes after you add back the modified provisioner:

$ fuzzball volume provisioner remove default
Remove storage provisioner "default"? [y/N] y
provisioner removal: cannot remove provisioner "00abf6b2-d5dc-3664-96bc-fa9787282e57": volumes are
still associated with it; use --ignore-volumes to remove anyway

$ fuzzball volume provisioner remove default --ignore-volumes
WARNING: Removing provisioner "default" with --ignore-volumes will soft-delete all
associated volume records. Physical data on the backing store will NOT be deleted.
Volumes can be recovered by creating a new provisioner and running scan.

Remove storage provisioner "default" and soft-delete its volumes? [y/N] y
Storage provisioner "default" removed

$ fuzzball volume provisioner add default -f default.yaml # the modified yaml file
Storage provisioner "00abf6b2-d5dc-3664-96bc-fa9787282e57" added

$ fuzzball volume provisioner scan --dry-run default
Discovered 1 volume(s):
  - myvol

$ fuzzball volume provisioner scan default
Discovered 1 volume(s):
  - myvol

Imported 2 new volume(s):
  myvol               imported (id: 019f5cd1-80cf-7722-858f-ab9d06df74ff)

Then add a provisioner for volumes shared across your external node segment only. In the example below this is a hostpath provisioner:

$ cat > hostpath_provisioner.yaml <<'EOF'
external:
  segment: external-static
  driver:
    type: hostpath
    path: /data/fuzzball
  access: all
  create:
  - your-admin-account@example.com
  ephemeral: all
EOF

$ fuzzball volume provisioner add external-static -f hostpath_provisioner.yaml

This lets your-admin-account@example.com create persistent volumes under /data/fuzzball on the external nodes (again, shared across the set of nodes).

Start and enable Substrate on each external node:

(node)$ sudo systemctl enable --now fuzzball-substrate

Watch the service log to confirm it detects hardware and picks up the provisioner definition:

(node)$ sudo journalctl -u fuzzball-substrate -f

You should see it settle on your provisioner definition, for example:

... msg":"provisioner annotations","id":"external-static"}

where external-static is the definition ID you created in Step 4.

The nodes should now appear in the cluster:

$ fuzzball node list

Run a job to confirm scheduling works end to end (this example requests a GPU):

$ fuzzball run --image docker://ubuntu:22.04 --device nvidia.com/gpu=1

You can confirm where the image pull and job were allocated with fuzzball workflow events:

$ fuzzball workflow events WORKFLOW_ID

The events show the node the image pull and the job ran on. When the image cache is shared, the image pull can be allocated to a cheaper node while the job runs on another node in the set — this is expected and relies on the shared image cache described in Architecture.