Using Volumes

HPC jobs typically require data. Volumes allow your jobs to access external data, allow multiple jobs within the same workflow to share data, and provide a way to save data after a workflow is completed.

Fuzzball supports the use of ephemeral volumes that last for the lifetime of a given workflow, or persistent volumes that retain data after workflow completion. Volumes can access data from URLs such as http, https, ftp, etc., as well as (API-compliant) S3 object stores.

Your organization’s storage is managed through storage provisioners, which determine the storage backend and control which groups can access volumes. As an end user, you specify volumes in your workflow definitions and Fuzzball handles the rest — selecting the right provisioner, creating the volume if needed, and mounting it into your job containers.

In many cases, it is necessary to provide a password, token, or other secret credential to access volumes such as S3 buckets, private GitHub repos, ftp servers etc. Secrets are saved within the Fuzzball secret store (via the web UI or CLI) and then referenced by scope and name within workflow specification files. Details are provided in the Secrets section of the User manual.

Because of the fact that many storage locations require secrets, it is impossible to provide working examples in all cases. Most of the examples presented in these documents are therefore meant as a guide to help you understand how to use volumes rather than a hands-on tutorial that you can follow.