Object Cache

The Fuzzball Object Cache is a built-in store for sharing files, data, and container images across workflows and clusters. Objects are addressed with fb:// URIs, organized into group and user namespaces, and managed through the fuzzball object commands or the web UI. Objects can be given a time-to-live (TTL) so the cache cleans up old data automatically.

The object cache is also used transparently to speed up container image pulls — see Automatic SIF image caching below.

Objects are referenced with an fb:// URI (the fuzzball:// prefix is also accepted):

fb://group/path/to/object
fb://user/path/to/object

There are two namespaces:

• fb://group/... — objects shared within your currently selected group. Visible to members of the group.
• fb://user/... — objects private to your user. Objects placed in the user namespace are keyed to your user and do not appear in the group listing, and vice versa.

A reference may be pinned to a specific content digest:

fb://group/images/myapp.sif@sha256:abc123...

The fuzzball object command group manages cached objects.

$ fuzzball object put ./report.csv fb://group/reports/report.csv

Default behavior: If an object already exists at the destination path, object put will succeed only if the new content has an identical digest. Attempts to overwrite with different content are rejected with an “already exists” error.

Force mode: Use --force to replace an existing object with new content:

$ fuzzball object put ./data.csv fb://group/data.csv --force

Recursive upload:

$ fuzzball object put ./dataset fb://user/datasets/run1 --recursive --ttl 7d

If the destination URI omits the object name (it ends in a trailing slash, such as fb://group/), the name is inferred from the source basename. The following two commands are equivalent:

$ fuzzball object put ./report.csv fb://group/
$ fuzzball object put ./report.csv fb://group/report.csv

The same applies recursively — fuzzball object put -R ./dataset fb://group/ uploads the tree under fb://group/dataset/.

Common errors: If you encounter a 401 Unauthorized error, run fuzzball context login to authenticate. A 403 Forbidden error indicates insufficient permissions — contact your group administrator.

$ fuzzball object get fb://group/reports/report.csv ./report.csv

If the local destination names a directory (it ends in a trailing slash, or is . / ..), the name is inferred from the object basename. The following two commands are equivalent:

$ fuzzball object get fb://group/reports/report.csv ./
$ fuzzball object get fb://group/reports/report.csv ./report.csv

The same applies recursively — fuzzball object get -R fb://group/dataset ./ downloads the tree into ./dataset/.

$ fuzzball object list fb://group/reports

The CLI walks every page automatically and prints the full result set.

$ fuzzball object describe fb://group/reports/report.csv

Shows detailed information about an object, including its size, digest, and versions.

$ fuzzball object delete fb://group/reports/report.csv

When a workflow pulls an external container image — for example via a docker:// or oras:// URI — Fuzzball converts the image to the SIF format and caches the result in the object cache automatically. Subsequent pulls of the same image are served from the cache instead of the external registry, which speeds up workflow startup and reduces dependence on external registry availability.

1. Pull and convert: When a job or service specifies a container image URI (e.g., docker://nvcr.io/nvidia/pytorch:24.01-py3), Fuzzball pulls the OCI image from the registry and converts it to SIF format.

2. Cache storage: The SIF image is stored in the object cache under a key derived from the image URI and target architecture; the blob itself is content-addressed by digest. The cached image is addressable with an fb:// URI if needed for manual inspection.

3. Subsequent pulls: When another workflow requests the same image URI, Fuzzball checks the cache first. If a cached SIF exists for that URI (and target architecture), the cached version is used immediately without contacting the external registry.

4. Cache refresh: Cached images are keyed by URI, so a mutable tag (e.g., latest) continues to be served from the cache even if the tag moves in the remote registry. The cache picks up a new version only after the cached entry expires or is deleted, triggering a fresh pull on the next request. Because the TTL counts from the entry’s last use, a frequently pulled tag can stay cached (and stale) indefinitely — to force a refresh, delete the cached entry or pin the image by digest.

Automatically cached entries are scoped to the group of the workflow that pulled the image; workflows in other groups populate their own cache entries on their first pull.

In deployments where storage provisioners serve different backends (different NFS servers, different availability zones), each provisioner can be assigned a segment identifier. Segments ensure the substrate-side image cache — the staged SIF copy used at job startup — is local to each provisioner’s shared filesystem so that nodes in that segment can access it.

Without segments, all provisioners share a single staging namespace. If provisioner A and provisioner B point to different filesystems, a SIF staged on A’s filesystem is not accessible to nodes using B’s filesystem. Segments solve this by giving each provisioner its own staging scope and staged copy.

As a secondary benefit, segments also isolate the image-pull lock per segment, allowing concurrent pulls of the same URI across segments instead of serializing behind a single lock.

Each segment stages its own copy of cached images on its shared filesystem. The object cache itself keeps a single shared copy of each converted SIF; only the substrate-side staging is duplicated.

Segments are configured by storage administrators in provisioner definitions. Users do not need to modify workflow YAML. See the Segments section in the provisioner documentation for details.

Cached images respect the object cache’s TTL policies. Automatically cached images carry no per-object TTL, so they inherit the configured default (group, organization, or cluster policy); if none is configured, the system default of 7 days applies. A TTL counts from the object’s last access or update, so entries in active use are retained. Expired entries are removed by a periodic purge job. Under disk pressure, the purge becomes more aggressive: entries are treated as expired at half their configured TTL and removed oldest-accessed first; at critical disk usage, new uploads are rejected. For manually uploaded objects, pass the --ttl flag to fuzzball object put to enforce a specific lifetime.

No configuration is required to benefit from automatic image caching.

The object cache stores converted SIF images with architecture-specific keys. When a workflow pulls an external container image (e.g., docker://ubuntu:22.04), Fuzzball converts it to SIF format and caches the result under a reference that includes the runtime architecture (such as amd64 or arm64).

This means:

• Heterogeneous clusters (clusters with both amd64 and arm64 nodes, for example) will maintain separate cache entries for the same logical image on each architecture. A workflow that pulls docker://ubuntu:22.04 on an amd64 node will not reuse the cache entry created by a pull on an arm64 node.
• Cache warming must be performed per architecture. If you pre-populate the cache with images on amd64 nodes, workflows scheduled on arm64 nodes will still need to pull and convert those images independently.
• Cache hit rates are architecture-local. Moving a workflow from an amd64 node to an arm64 node (or vice versa) will typically result in a cache miss for images that have not yet been pulled on the target architecture.

No configuration is required to enable architecture-specific caching — Fuzzball automatically ensures that nodes receive the correct image variant for their hardware.

For clusters with a single architecture, this behavior is transparent and has no operational impact. Mixed-architecture deployments should plan cache warming and storage capacity accordingly.

The Fuzzball web UI includes an Object Cache browser for uploading, searching, and inspecting objects across the group and user namespaces, including object metadata and version history.

At the top of the Object Cache browser, you’ll see a namespace selector with Group and User (private) options. This selector controls which namespace you’re viewing:

• Group — shows all objects in the group namespace (fb://group/...), visible to all members of your group.
• User (private) — shows objects in your personal user namespace (fb://user/...), visible only to you.

Click either option to switch between namespaces. The object list, breadcrumb, and all operations update to reflect the selected namespace.

The breadcrumb bar at the top of the browser shows your current location:

• In the group namespace, the breadcrumb starts with Group followed by any path segments (e.g., Group > datasets > images).
• In the user namespace, it starts with User (private) followed by path segments (e.g., User (private) > experiments > run-42).

Click any segment to navigate to that level. Clicking the namespace label (Group or User (private)) returns you to the root of that namespace. All operations in the browser — viewing object details, deleting objects, and creating references — act on the currently selected namespace only.

Click Upload to open the Upload Object dialog. Uploads are always relative to your current location: the object is stored in the namespace and folder you are browsing, so you don’t need to retype the full path.

The dialog has the following fields:

• File — the file or files to upload. Drag them onto the drop zone, or use Choose File / Choose Folder. You can select multiple files, and folder uploads preserve their directory structure.
• Object Name — the name the object is stored under. For a single file it is filled in automatically from the file’s name; edit it to store the object under a different name. (Shown only when uploading a single file on its own; folder and multi-file uploads keep each file’s own name.)
• Prefix — an optional subfolder, relative to your current location. The dialog shows that location — for example, Relative to Group / datasets / images. Leave it blank to upload directly into the current folder, or enter a value such as run-42/ to place the object in a subfolder beneath the current folder.
• TTL — an optional lifetime after which the object is automatically removed: 1 day, 7 days, 30 days, or No expiry (the default).

Click Upload in the dialog to store the object. For example, while browsing Group > datasets > images with an Object Name of cat.png and a Prefix of train/, the file is stored at fb://group/datasets/images/train/cat.png.

For the command-line equivalent, see Upload.

The browser encodes the namespace and current path in the URL query string:

• Group namespace root: ?prefix=
• User namespace root: ?prefix=user/
• Group namespace path: ?prefix=group/datasets/images/
• User namespace path: ?prefix=user/experiments/run-42/

You can bookmark or share these URLs; opening a URL with a prefix parameter selects the correct namespace and navigates to the specified path.

URLs pointing to the user namespace are personal. If you share a ?prefix=user/... URL with a colleague, they will see their own user namespace at that path, not yours. To share objects with others, store them in the group namespace.