Workflow chaining

Workflow chaining is a mechanism to allow multiple workflow executions to be chained together in a declarative fashion, such that the output of one becomes the inputs of the next. Chaining is implemented as an extension to the Cirrus Process Payload format via support for an array of process definitions under the payload process key, where each element in that process array represents a link in the chain. The payload for the simple case of two chained workflows would have a process array something like this:

process = [
    { <workflow 1 definition> },
    { <workflow 2 definition> }
]

This example models a simple chain:

flowchart LR 1(Workflow 1) 2(Workflow 2) 1 --> 2

In this case, the Cirrus Process Payload output from workflow 1 is passed as the input to workflow 2 after

popping the workflow 1 process definition from the array.
deleting the payload id (as the id necessarily has to be different for the workflow 2 execution).

Each step in the workflow chain additionally supports a nested array, to model branching within the chain. For example, the following would branch into two separate workflow executions after the end of workflow 1:

process = [
    { <workflow 1 definition> },
    [{ <workflow 2a definition> }, { <workflow 2b definition> }],
    { <workflow 3 definition> }
]

This example would result in an execution chain that looks like this:

flowchart LR 1(Workflow 1) 2a(Workflow 2a) 2b(Workflow 2b) 3a(Workflow 3) 3b(Workflow 3) 1 --> 2a 1 --> 2b 2a --> 3a 2b --> 3b

Note that chaining alone can effectively model one-to-one and one-to-many dependency relationships. If needing to model many-to-one or many-to-many relationships, see workflow callbacks for an effective way to consolidate the outputs of multiple workflows together.

Filtering output items

In cases where a successive workflow in a chain does not need all items produced by an multi-output-item proceeding workflow, chaining supports an output item filter via a JSONPath filter expression. To set such a filter, specify it in the successive workflow’s process definition using the chain_filter key. For example, if we only wanted items with an id starting with SWE that had a confidence property greater than 80, we could do something like this:

process = [
    { <workflow 1 definition> },
    {
        <workflow 2 definition>,
        "chain_filter": "@.id =~ 'SWE*' & @.properties.confidence > 80"
    }
]

The chain_filter string is specifically a JSONPath filter expression string to apply to the output payload features. In other words, the above filter expression example actually results in the following JSONPath query applied against the JSON workflow 1 output payload:

$.features[?(@.id =~ 'SWE*' & @.properties.confidence > 80)]