Creating Your First Workflow

The first step in creating a workflow is designing it. This involves determining what jobs need to be run, and defining the specific behavior and resource requirements for each of those jobs. These details about a workflow get defined in a Fuzzfile. (A YAML file with the Fuzzball workflow specification.)

Fuzzfiles are super versatile, allowing you to capture everything from the simplest single-job workflow, like a printer job that outputs numbers, to highly complex workflows involving tasks like training machine learning models.

You can create Fuzzfiles interactively through Fuzzball’s web UI, or you can simply write the file yourself in any text editor.

Many users find it easiest to use the web UI to create the first iteration of a Fuzzfile and then edit the file manually to fill in the details.

Below, a simple Fuzzfile is presented. Detailed information on developing Fuzzfiles and a complete syntax guide are presented elsewhere in the user guide.

The workflow used to complete this tutorial runs for 5 minutes and counts from 1 to 300 (once per second). In the real world, this workflow is completely useless. But it serves as a useful demonstration for this tutorial since it produces output that can be gathered via Fuzzball logging and runs for 5 minutes allowing time for user interaction. More useful “real world” examples can be obtained from this Fuzzball examples Git repo.

We refer to the following workflow as “printer” and the Fuzzfile as printer.fz.

version: v1
jobs:
  printer:
    image:
      uri: docker://alpine:latest
    policy:
      timeout:
        execute: 10m0s
    env:
      - PRINTER_START_INT=1
      - PRINTER_END_INT=300
      - PRINTER_SLEEP_INTERVAL=1
    command:
      - /bin/sh
      - '-c'
      - for i in $(seq $PRINTER_START_INT $PRINTER_END_INT); do echo $i; sleep $PRINTER_SLEEP_INTERVAL; done
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GB

Preview workflow in web UI

The .fz file extension is not necessary. Fuzzfiles also commonly carry .yml or .yaml extensions since they are written in YAML.

If you want detailed information about the sections in this Fuzzfile, expand the section below. Feel free to skip this section for now if you are just trying to learn how to submit your workflows to Fuzzball.

CLICK TO EXPAND: Explanation of example Fuzzfile ↕

You might find it useful to start with this Fuzzfile and edit it to suit your needs. Therefore, a brief explanation is provided. Feel free to skip this section for now and head directly to the section on Creating the Fuzzfile if you are just trying to understand how to run a workflow.

We will explain each Fuzzfile section in turn.

The order of sections at the same level of indentation is arbitrary.

version: v1
jobs:

The first line specifies that we are using version 1 Fuzzfile syntax, and the second line starts the “jobs” section.

  printer:
    image:
      uri: docker://alpine:latest

This starts the section associated with the job called “printer” (which is the only job in this Fuzzfile), and states that the job should run in the latest alpine container obtained from Docker Hub. (See the documentation on container URLs for more info about specifying containers.)

    policy:
      timeout:
        execute: 10m0s

The policy section states that the job should timeout (be canceled) if it runs longer than 10 minutes and 0 seconds.

    env:
      - PRINTER_START_INT=1
      - PRINTER_END_INT=300
      - PRINTER_SLEEP_INTERVAL=1

The environment variable section sets environment variables PRINTER_START_INT, PRINTER_END_INT, and PRINTER_SLEEP_INTERVAL to 1, 300, and 1 repsectively.

    command:
      - /bin/sh
      - '-c'
      - for i in $(seq $PRINTER_START_INT $PRINTER_END_INT); do echo $i; sleep $PRINTER_SLEEP_INTERVAL; done

This section specifies the command(s) that should run in the job (within the given container). It is broken into parts that will be concatenated into a single command at runtime. Using the /bin/sh -c syntax, we can submit more than one command as a single string to the shell interpreter.

The actual commands in this example specify to run a loop with 300 iterations configured through the envionment variables. Within each cycle of the loop, echo the current iteration, and wait (sleep) for one second. So this workflow just counts seconds from 1 to 300 (a total of five minutes).

    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GB

The final section specifies the computational resources that the job needs. In this case the job requires 1 core and 1GB memory. Affinity refers to the way in which the CPU for this job should access its memory cache. NUMA is fine for most jobs, but the values SOCKET and CORE are also available.

Creating the Fuzzfile

Now that we have an example Fuzzfile specifying a simple workflow, we can use it to demonstrate how to build and run workflows using either the web UI or the CLI.

select

Please select either the web UI or CLI tab to see the appropriate instructions for your environment.

web UI

The web UI has a full-featured interactive workflow building tool that can be used to generate and submit Fuzzfiles. This will be covered extensively in another section of the user manual. For the purposes of running this quick tutorial, we will simply copy and paste the above Fuzzfile into the web UI editor.

If you click “Workflow Editor” and “Create New”, you will see a blank page in the workflow editor.

Fuzzball create new workflow section

Now you can either click the ellipses (...) menu in the lower right and select “Edit YAML” or simply press e on your keyboard. An editor with a Fuzzfile stub will appear.

Fuzzball editor newly opened

You can delete the current contents and copy and paste the contents of printer.fz from above.

Fuzzball editor copied and pasted printer

Now pressing “save” will return you to the interactive workflow editor. You will now see the printer job instead of a blank editor page.

Fuzzball workflow editor with printer

The Fuzzball web UI will automatically validate the yaml file for syntax errors.

From here, you can click on the job and edit its properties in the tabs to the right as described in another section if you wish.

CLI

Building your Fuzzfile outside of the GUI is simply a matter of using a text editor. First, let’s create a new directory to experiment in:

$ mkdir ~/fb-work

$ cd !$

Now either copy and paste the contents of printer.fz described above into the appropriately named file using the text editor of your choice, or run the following commands.

version: v1
jobs:
  printer:
    image:
      uri: docker://alpine:latest
    policy:
      timeout:
        execute: 10m0s
    env:
      - PRINTER_START_INT=1
      - PRINTER_END_INT=300
      - PRINTER_SLEEP_INTERVAL=1
    command:
      - /bin/sh
      - '-c'
      - for i in $(seq $PRINTER_START_INT $PRINTER_END_INT); do echo $i; sleep $PRINTER_SLEEP_INTERVAL; done
    resource:
      cpu:
        cores: 1
        affinity: NUMA
      memory:
        size: 1GB

Fuzzball will automatically check and validate your Fuzzfile for syntax errors when you submit the workflow, but you can do so manually with the following command:

$ fuzzball workflow validate ./printer.fz
"./printer.fz" has been validated.

A Fuzzfile with syntax errors will return an error similar to this:

$ fuzzball workflow validate ./printer.fz
yaml: unmarshal errors:
  line 2: field jerbs not found in type api.WorkflowDefinition