Using GPUs with Fuzzball

Fuzzball intelligently passes GPU resources from the host through to the containers that support your jobs. Once an administrator has properly added and configured resources that include GPUs, you can access them simply by specifying them in your Workflow.

Observe the following Fuzzfile:

version: v4
jobs:
  gpu-check:
    image:
      uri: docker://rockylinux:9
    script: |
      #!/bin/sh
      nvidia-smi
    resource:
      cpu:
        cores: 1
      memory:
        size: 1GB
      devices:
        nvidia.com/gpu: 1

Note that the gpu-check job in this workflow is based on the official Rocky Linux (v9) container on Docker Hub. This container does not have a GPU driver or any NVIDIA related software installed. However, the Fuzzfile specifies that this job should run the program nvidia-smi to check the status of any GPUs that are present.

The resources section has a field devices that allows you to specify that the job should use GPUs. Since our administrator has configured GPUs for use with nvidia.com/gpu, we can specify that we need to use one. Fuzzball will cause all of the driver-related software to be available in the container.

If you are using the workflow editor in the web UI to create your Fuzzfile, you can open the Resources tab in the flyout menu on the right and “Add” a device. Once again, in the example configuration, the nvidia.com/gpu string is appropriate to add a GPU to our job.

After submitting the workflow, note that the nvidia-smi command has no trouble completing even though the Rocky Linux container has no NVIDIA-related software.

Thu Feb 12 18:13:24 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.44.01              Driver Version: 590.44.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:00:1E.0 Off |                    0 |
| N/A   23C    P8              8W /   70W |       0MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

It’s important to understand that not only is installing GPU drivers within your containers unnecessary for GPU workflows, it’s actually considered a poor practice that can lead to compatibility issues. Here’s why:

When executing GPU workloads, Fuzzball automatically identifies the appropriate libraries and binaries associated with the GPU driver installed on the host system and dynamically injects them into your container at runtime. This sophisticated mechanism ensures perfect compatibility between the kernel modules running on the host and the libraries available inside the container.

If you manually install GPU drivers inside your container, you risk creating conflicts — the software in your container might discover and attempt to use these drivers instead of the properly injected ones, potentially disrupting the correspondence between libraries and kernel modules and leading to mysterious failures or performance issues.

Important distinction: There is a difference between the GPU driver and libraries like CUDA or HIP. This is confusing since CUDA and HIP are often packaged with their corresponding GPU drivers. While you should not install GPU drivers in your container, you should install the appropriate computational libraries to support your GPU-accelerated applications.

The standard fuzzball cluster docker-compose deploy command brings up a CPU-only stack. On hosts that have GPUs you can opt in to a GPU-enabled substrate image. The same image bundles both the NVIDIA and AMD device plugins; the substrate automatically loads whichever matches the GPU present on the host, so one --gpu image works on either vendor:

$ fuzzball cluster docker-compose deploy --gpu [other deploy flags]

Or to update an existing deployment for GPU-enabled substrate image:

$ fuzzball cluster docker-compose update --gpu

• The host’s GPU vendor driver must be installed and loaded (the vendor’s status tool — e.g. nvidia-smi for NVIDIA — should work on the host).
• The vendor-specific device nodes must be present (the driver creates these at boot).

$ fuzzball node list --available
NODE ID         | HOSTNAME              | CPU TYPE  | AVAILABLE CORES | AVAILABLE MEMORY (GB) | AVAILABLE DEVICES | RUNNING JOBS | CLUSTER
172.21.0.5/7334 | substrate-localnode-2 | cpu/arm64 | 20              | 127.9                 | nvidia.com/gpu:1  | 0            | local-dev

If a GPU device type (e.g. nvidia.com/gpu:N or amd.com/gpu:N) shows up in the AVAILABLE DEVICES column, the substrate-side plumbing is correct and you can submit GPU workflows like the gpu-check Fuzzfile above.

Fuzzball supports AMD ROCm GPUs using the same device-plugin mechanism as NVIDIA, with the amd.com/gpu device string. The device plugin passes the GPU device nodes (/dev/kfd and /dev/dri/renderD*) through to your job and makes the host’s ROCm driver libraries available in the container. Request one by adding amd.com/gpu under resource.devices (or via the Resources tab in the web UI workflow editor, the same way you would add nvidia.com/gpu):

version: v4
jobs:
  amd-gpu-details:
    image:
      uri: docker://python:3-slim
    script: |
      #!/usr/bin/env python3
      import glob, os

      def read(p):
          try:
              with open(p) as f:
                  return f.read().strip()
          except OSError:
              return None

      NAMES = {"0x1586": "AMD Radeon 8050S/8060S (Strix Halo, gfx1151)"}
      found = False

      for dev in sorted(glob.glob("/sys/class/drm/card*/device")):
          if read(f"{dev}/vendor") != "0x1002":
              continue
          found = True

          did = read(f"{dev}/device") or "unknown"
          print(f"GPU      : {NAMES.get(did, 'AMD GPU')} [{did}]")
          print(f"PCI addr : {os.path.basename(os.path.realpath(dev))}")

          # Read strict hardware buffer
          vt = read(f"{dev}/mem_info_vram_total")
          vu = read(f"{dev}/mem_info_vram_used")

          # Read flexible system memory pool (GTT) used by ROCm/AI
          gt = read(f"{dev}/mem_info_gtt_total")
          gu = read(f"{dev}/mem_info_gtt_used")

          # Calculate Total Unified Memory Available to your AI Code
          total_vram = (int(vt or 0) + int(gt or 0)) / 2**30
          used_vram = (int(vu or 0) + int(gu or 0)) / 2**30

          print(f"Unified VRAM Total : {total_vram:.1f} GiB (Dedicated: {int(vt or 0)/2**30:.1f} GiB + GTT: {int(gt or 0)/2**30:.1f} GiB)")
          print(f"Unified VRAM Used  : {used_vram:.2f} GiB")

          busy = read(f"{dev}/gpu_busy_percent")
          if busy is not None:
              print(f"GPU busy : {busy}%")

          for hw in glob.glob(f"{dev}/hwmon/hwmon*/temp1_input"):
              t = read(hw)
              if t:
                  print(f"Temp     : {int(t) / 1000:.0f} C")

          print()
          print("READY" if os.path.exists("/dev/kfd") else "NOT READY: /dev/kfd missing")

      if not found:
          print("No AMD GPU visible in /sys/class/drm -- is /sys mounted in the job?")
    resource:
      cpu:
        cores: 1
      memory:
        size: 1GB
      devices:
        amd.com/gpu: 1

This job reads the GPU’s attributes from sysfs – name, VRAM, utilization, temperature, and PCI address – and confirms that /dev/kfd is present, indicating the GPU is passed through and ready for ROCm workloads.

Unlike NVIDIA – where the host nvidia-smi is injected and runs in a stock container image – AMD’s rocm-smi is a Python-based tool that is not injected. To run ROCm applications or inspect the GPU with rocm-smi/rocminfo, use a container image that ships its own ROCm userspace (for example a rocm/* image). As with NVIDIA, do not install the GPU driver itself inside the container.

On nodes with an NVIDIA GPU, Fuzzball collects per-device health alongside the device inventory above: ECC error counts, Xid faults from the kernel log, and thermal or power throttling reported by the driver. A failing card raises a condition on its node, lowers the node’s reliability score, and can cordon or drain the node if a cluster has opted into automated response.

Nothing extra needs installing: the collector uses the nvidia-smi that ships with the NVIDIA driver, and a node without it reports GPU health as unavailable rather than as an error.

AMD GPUs are inventory-only. They are scheduled and passed through exactly as described above, but ROCm health counters are not collected, so an AMD node’s reliability score reflects host signals only.

See Node Health for the conditions raised and how Xid numbers are graded.