Closing Notes and Next Steps

Each workflow can create/use an arbitrary number of volumes and carry out any amount of data ingress and egress. However, it is a good practice to minimize data transfer since it is time consuming and (potentially) costly. If you find yourself building workflows with multiple ingress and egress operations, it may help to consider a different strategy. Reach out to an administrator or contact CIQ to learn more about building optimized workflows.

As mentioned previously, it is often necessary to use secrets to access online containers and data. Secrets are covered in detail in another section of these guides.

Container images referenced in workflows are automatically converted to SIF format and cached. In deployments where storage provisioners point to different backends (different NFS servers or cloud filesystems), each provisioner can be assigned a segment so that converted images are staged locally on its filesystem. Without segments, nodes on one provisioner’s filesystem would not be able to access images staged on another provisioner’s filesystem.

Segments are configured by storage administrators in provisioner definitions and require no changes to workflow YAML. See the Segments section in the provisioner documentation for details.

If your workflow submission fails with a volume validation error, check the following:

A volume name must be 1-255 characters and must not contain path separators (/ or \), path-traversal sequences (. or ..), or null bytes. Letters in any case, digits, spaces, and other punctuation are allowed. Common issues:

• Path separators: Remove any / or \ from the name (e.g. change data/set to data-set)
• Path traversal: Rename ., .., or any name containing .. (e.g. change my..data to my-data)
• Empty or too long: Provide a non-empty name no longer than 255 characters

If you reference a persistent volume by name that does not exist, the workflow will be rejected. Verify the volume exists:

$ fuzzball volume list

If the volume does not exist, create it before submitting the workflow:

$ fuzzball volume create <provisioner-name> <volume-name>

If you attempt to create an ephemeral volume on a provisioner your group does not have access to, the workflow will be rejected. Contact your cluster administrator to request access or choose a different provisioner.