Workflows

Cirrus workflows are the component that puts the “pipe” in “pipeline”. Workflows model a transformation of an input item or set of input items with a process definition into one or more output items via processing from one or more tasks. Workflows are composed of one or more “tasks”.

Cirrus workflows, being implemented via AWS Step Functions, are written in the AWS States Language. Workflows are defined in state-machine.json files and require a companion README.md.

These state-machine.json files define the entire step function workflow. A non exhaustive list of configuarable are :

• task definitions

• task input parameters

• task output parameters

• job definition

• job name

• task retry config

• error handling

• resource ARNs

The state database tracks the state of items processed at the workflow level.

Workflow topics:

• Writing workflow definitions
  • Simple example
  • Basic step function structure
  • Triggering workflows off one another
  • Error handling
  • Workflow best practices
• Batch tasks in workflows
  • pre-batch and post-batch
  • Put it all together with a parallel block
  • Batch retries vs step function retries
• Workflow chaining
  • Filtering output items
• Workflow callbacks