Cirrus architecture =================== Cirrus overview --------------- An overall picture of Cirrus looks like this: .. image:: images/arch-overview.png As shown in that diagram, input Cirrus Process Payloads are generated by a :doc:`Feeder ` and published into an input SQS queue, either manually by a user or from an externals source such as an SNS notification. These enqueued messages are then processed by the ``process`` Lambda Function. The ``process`` function creates or updates the :doc:`State DB <70_statedb>` record for the payload and starts a :doc:`Workflow ` execution by dispatching the payload to the specified Workflow. Any output items and their assets are persisted to S3, and the output items are published to an output SNS topic that can notify downstream subscribers, like stac-server, of newly-processed items. At the end of a workflow execution, an EventBridge event is triggered, which notifies the listening ``update-state`` lambda that the execution has completed. ``update-state`` updates the State DB record for the processed payload in accordance with the final workflow state (success, failed, aborted, or invalid), and triggers any necessary follow-up actions. For example, failed or invalid executions can trigger notifications to the Failure/Invalid SNS topic(s), and successful payloads using :doc:`workflow chaining ` can be re-enqueued for the next stage of the processing chain. Workflow structure ------------------ Workflows are composed of one or more :doc:`Tasks `, which use an input Cirrus Process Payload to define input metadata/assets for the Task processing. A workflow could look something like this: .. image:: images/workflow-example.png This example workflow begins with a preprocessing step to do some initial metadata modifications. Following that, a batch task to perform some computationally-intensive manipulation of the input item data assets, and a create preview task then generates a new preview image asset. The workflow ends with a publish task step, which pushes the final generated item to S3 for persistence and the Cirrus publish SNS topic for and subscribed downstream consumers of this Workflow's items. In the event of any task failure, the failure step in the step function ensures that the step function execution will end in a ``FAILED`` state. If all steps complete successfully then the execution will end in a ``SUCCEEDED`` state. Glossary -------- **execution**: An individual run of a workflow on a payload **payload**: A Cirrus Process Payload used as input to a workflow **publish**: Uploading output STAC Items to s3 and publishing the Items to the Cirrus Publish SNS topic **statedb**: A database used to track the state of executions **task**: A single function operating on a Cirrus Process Payload running on Lambda or Batch **workflow**: An AWS Step Function consisting of 1 or more tasks **feeder**: Anything that generates a :doc:`Cirrus Process Payload <../30_payload>` and queues it for processing. Naming conventions ------------------