Tips and Tricks
===============

Processing Queue Priority
-------------------------

By default, there is only one `process` queue. This can cause difficulty when doing
back-processing or failure re-run, as thousands of older, lower-priority payloads
can pile up in front of newly-generated, higher-priority payloads. A solution to this
is to create an addiitonal "bulk" queue and put all of the lower-priority payloads into it.

In the CloudFormation resources file, define a new queue named `ProcessBulkQueue` and
associated DLQ with the same settings as the default `ProcessQueue`.

Create an environment variable in cirrus.yml to reference this queue::

   CIRRUS_PROCESS_QUEUE_URL: !Ref ProcessQueue
   CIRRUS_PROCESS_BULK_QUEUE_URL: !Ref ProcessBulkQueue

Modify the `process` lambda definition to trigger on either of these queues with
`maximumConcurrency` set::

   lambda:
   memorySize: 128
   timeout: 30
   reservedConcurrency: 16
   handler: lambda_function.lambda_handler
   events:
      - sqs:
         arn: !GetAtt ProcessQueue.Arn
         maximumConcurrency: 4
      - sqs:
         arn: !GetAtt ProcessBulkQueue.Arn
         maximumConcurrency: 12


To use this "bulk" queue, replace uses of CIRRUS_PROCESS_QUEUE_URL with
CIRRUS_PROCESS_BULK_QUEUE_URL.