Tips and Tricks

Processing Queue Priority

By default, there is only one process queue. This can cause difficulty when doing back-processing or failure re-run, as thousands of older, lower-priority payloads can pile up in front of newly-generated, higher-priority payloads. A solution to this is to create an addiitonal “bulk” queue and put all of the lower-priority payloads into it.

In the CloudFormation resources file, define a new queue named ProcessBulkQueue and associated DLQ with the same settings as the default ProcessQueue.

Create an environment variable in cirrus.yml to reference this queue:

CIRRUS_PROCESS_QUEUE_URL: !Ref ProcessQueue
CIRRUS_PROCESS_BULK_QUEUE_URL: !Ref ProcessBulkQueue

Modify the process lambda definition to trigger on either of these queues with maximumConcurrency set:

lambda:
memorySize: 128
timeout: 30
reservedConcurrency: 16
handler: lambda_function.lambda_handler
events:
   - sqs:
      arn: !GetAtt ProcessQueue.Arn
      maximumConcurrency: 4
   - sqs:
      arn: !GetAtt ProcessBulkQueue.Arn
      maximumConcurrency: 12

To use this “bulk” queue, replace uses of CIRRUS_PROCESS_QUEUE_URL with CIRRUS_PROCESS_BULK_QUEUE_URL.