Amazon Simple Queue Service (SQS) is a fully managed message queuing service. SQS architecture allows “pull / poll” of messages versus “push” in case of SNS’ Publish-Subscribe architecture.

 

Key Components of Simple Queue Service (SQS)

Queue is the transient storage where received messages are kept until they are received by (polling) application(s) or the defined period (max up to 14 days) has ended without any application reading it.

There are two architecture implementations of Queue with Amazon SQS:

  • FIFO (First In First Out) Queue – this queue preserves the order in which messages were received.
    • Provides high throughput (up to 300 send, receive, or delete operations per second), but is still low compared to throughput of Standard Queue
    • Exactly-Once Processing – message is guaranteed to be delivered only once.
      • If consumer application does not deletes the message within defined period, the message is assumed to be not processed and becomes available again
  • Standard Queue – this queue attempts to preserve order, but does not guarantee it.
    • Provides nearly unlimited number of transactions per second
    • At-least-Once Delivery – the message is typically delivered once, but occasionally may get delivered more than once

 

Dead Letter Queue (DLQ)

This is a special queue created with the purpose to collect messages from source queue(s) that do not get processed by any consumer.

  • Each DLQ can be associated with one or more source queues.
  • It can be important queue to analyze lost messages for root cause(s).
  • The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.

 

Message

  • Message is text only, and can be of any text format – such as plain text, XML, JSON, etc.
  • Message can be up to 256 KB
  • Messages are retained in queues for up to 14 days
    • Default is 4 days, and you can configure retention period to be as low as 1 minute.
  • When a message is received (by a consumer application) the message is locked and hidden for a defined period from other polling applications to avoid duplicate receipt
    • The consumer has to confirm processing by deleting the message within defined period, or otherwise the lock expires and the message is made available again in the queue
    • This defined period is configured as VisibilityTimeout period

 

Key Points

  • SQS is highly available and support strong durability of data
    • SQS stores queue and messages copies across multiple AZs within a specific Region
    • SQS is a regional service
  • You can have unlimited SQS queues with unlimited number of messages
  • You can batch messages to efficient processing, and at the same time reducing cost
  • SQS can encrypt the messages (as soon as it receives) using KMS keys
  • Two types of polling are supported
    • Short Polling – the request to queue checks for messages, and if none available the response comes back empty immediately.
    • Long Polling – the request to queue waits up to 20 seconds if none is immediately available. This reduces the extraneous polling to queues and dramatically reduces the cost.
  • Once a queue is created, you cannot change queue type between Standard and FIFO. This is configured at the time of creation of the queue.
  • Amazon provides the Amazon SQS Java Messaging Library that implements the JMS 1.1 specification and uses Amazon SQS as the JMS provider.
  • SQS can be a key component of Fan-out architecture to decouple services.
  • SQS is PCI DSS compliant and HIPAA eligible.

 

Pricing

SQS is billed for following components:

  • Requests (send, receive, delete) – per million (first million is free each month)
    • Rates are different for Standard and FIFO queue (FIFO being more expensive)
    • A single request can have 1 to 10 messages, up to a maximum total payload of 256 KB
    • Each 64 KB chunk of payload is billed as 1 request (API action with 256 KB payload is billed as 4 requests)
  • Data Transfer – standard AWS Data Transfer (out) charges apply
  • KMS usage – when using KMS for SQS server-side encryption, standard KMS calls charges apply

 

External Resources

 

 

AWS Data Integration Queue SQS Storage