Cirrus architecture
Cirrus overview
An overall picture of Cirrus looks like this:
As shown in that diagram, input Cirrus Process Payloads are generated by a Feeder and published into an
input SQS queue, either manually by a user or from an externals source such as an SNS notification. These enqueued messages are
then processed by
the process Lambda Function. The process function creates or updates the
State DB record for the payload and starts a
Workflow execution by dispatching the payload to the
specified Workflow.
Any output items and their assets are persisted to S3, and the output items are published to an output SNS topic that can notify downstream subscribers, like stac-server, of newly-processed items.
At the end of a workflow execution, an EventBridge event is triggered, which
notifies the listening update-state lambda that the execution has
completed. update-state updates the State DB record for the processed
payload in accordance with the final workflow state (success, failed, aborted,
or invalid), and triggers any necessary follow-up actions. For example, failed
or invalid executions can trigger notifications to the Failure/Invalid SNS
topic(s), and successful payloads using workflow chaining can be re-enqueued for the next stage of the
processing chain.
Workflow structure
Workflows are composed of one or more Tasks, which use an input Cirrus Process Payload to define input metadata/assets for the Task processing. A workflow could look something like this:
This example workflow begins with a preprocessing step to do some initial metadata modifications. Following that, a batch task to perform some computationally-intensive manipulation of the input item data assets, and a create preview task then generates a new preview image asset. The workflow ends with a publish task step, which pushes the final generated item to S3 for persistence and the Cirrus publish SNS topic for and subscribed downstream consumers of this Workflow’s items.
In the event of any task failure, the failure step in the step function ensures
that the step function execution will end in a FAILED state. If all steps
complete successfully then the execution will end in a SUCCEEDED state.
Glossary
execution: An individual run of a workflow on a payload
payload: A Cirrus Process Payload used as input to a workflow
publish: Uploading output STAC Items to s3 and publishing the Items to the Cirrus Publish SNS topic
statedb: A database used to track the state of executions
task: A single function operating on a Cirrus Process Payload running on Lambda or Batch
workflow: An AWS Step Function consisting of 1 or more tasks
feeder: Anything that generates a Cirrus Process Payload and queues it for processing.