Message queue#

The SQS service (Simple Queue Service) allows software components in the cloud to asynchronously exchange messages without losing them. Thanks to its distributed architecture, the service ensures reliable storage and high availability of multiple messages.

The queue serves as a buffer where some components put messages and others retrieve them for further processing. Messages are locked during processing, preventing the same message from being processed simultaneously by different consumers.

SQS can be used with any software components provided that they can send HTTPS requests or use the AWS SDK.

SQS overview#

The messaging system has a distributed architecture and consists of three main parts:

  • software components (producers and consumers);

  • standard queue;

  • messages in the queue.

Producers put messages to a queue, and consumers retrieve them from the queue for processing. To ensure redundancy, each message in the queue is stored on multiple SQS servers in different availability zones (if available in the region).

Message lifecycle#

The life cycle consists of three stages:

  1. A producer puts a message to the queue. The message is stored on multiple SQS servers.

  2. A consumer requests a message from the queue when it is ready to process it. While the message is being processed, it is locked, i.e. it remains in the queue but is unavailable to other consumers.

  3. Once processing is complete, the consumer deletes the message from the queue. If the message could not be processed within the allotted time (for example, due to a consumer failure), it becomes available to other consumers.

Standard queue#

K2 Cloud employs a standard queue engine, which ensures that a message will be delivered at least once. However, in some situations (for example, due to network latency), a message may be re-delivered. This should be taken into consideration when developing applications in order to ensure that processing re-delivered messages does not change the system state.

Messages are delivered in the order they were sent, whenever possible. However, in some cases like recovery after failure, messages may come out of order. If the order matters, such applications must employ an internal message ordering logic.

The standard queue can handle many API method calls per second, such as SendMessage, ReceiveMessage, and DeleteMessage. Thanks to the high throughput, it is ideal for tasks, which require fast, asynchronous processing of lots of messages and tolerate message order variations and duplicates. Such tasks may include:

  • asynchronous processing of background jobs, such as sending SMS, push notifications, and emails

  • buffering during peak loads, when incoming message flow exceeds processing capacity;

  • reliable message delivery in cases when duplicates are acceptable (logging, metrics collection);

  • asynchronous message exchange between microservices, etc.

Queue parameters#

You can specify different settings for different message queues.

Visibility timeout — The period of time during which a message requested by a consumer is unavailable to other consumers. This mechanism ensures processing idempotence in case of concurrent access. In other words, it prevents multiple consumers from simultaneously processing the same message. The maximum timeout is 12 hours.

Send delay — This parameter determines the time interval (in seconds) during which a new message in the queue must remain invisible to consumers after being sent. As a result, the message becomes available for processing only after a certain period of time has passed since it entered the queue. This parameter can be set for the entire queue or for an individual message. The maximum delay is 20 seconds.

Message wait time — The maximum time to wait for a message after a ReceiveMessage request comes when the queue is empty. If there are messages in the queue or a message comes before the wait time expires, a response will be sent immediately. If a message does not come within the specified time, an empty response will be sent. This Long Polling mechanism helps reduce the number of requests resulting in such empty responses.

Message retention period – Messages can remain in the queue for up to four days.

Maximum message size – Message size cannot exceed 1 MB.

Receipt handle#

Only the message consumer can delete it from the queue or modify its visibility timeout.

When a request is made, a consumer receives a message and receipt handle, which identifies the request to receive the message, but not the message itself. To delete a message or modify the visibility timeout, a unexpired receipt handle must be passed.

A new receipt handle is passed with each message request. Therefore, to delete messages from the queue, the last receive handle must be passed before the current visibility timeout expires.

Managing queues and messages#

Important

To automate the use of queues and messages, use SQS API.

Note

To be able to manage queues and messages, a user needs to have SQSFullAccess project privileges. For instance, such privileges are available to administrators in the CloudAdministrators group. If necessary, you can create a separate user, add the user to the project, and either attach the SQSFullAccess policy to it or add the user to the group of cloud administrators in this project.

Managing queues#

Create a queue#

  1. Go to SQS Queues.

  2. Click Create.

  3. In the window that opens, set the queue parameters (see more in Queue parameters).

    • queue name;

    • visibility timeout;

    • message delivery delay;

    • message wait time;

    • Message retention period

    • Maximum message size

  4. Once all parameters are set, click Create.

Edit queue#

  1. Go to SQS Queues.

  2. In the resource table, select the desired queue and click Edit.

  3. In the window that opens, set up new queue parameters (for details, see the section Queue parameters).

    • visibility timeout;

    • message delivery delay;

    • message wait time;

    • Message retention period

    • Maximum message size

  4. Once all parameters are set, click Save.

You can also edit queue parameters in the Information tab on the queue page.

Purge a queue#

Important

All messages, including those under processing, will be deleted from the queue.

  1. Go to SQS Queues.

  2. In the resource table, select the desired queue and click Purge queue. If this button is not shown, open the additional action menu (More).

  3. In the dialog window, confirm the action.

You can also purge the queue in the Information tab on the queue page. If you need to delete specific messages from the queue, use the instruction.

Delete a queue#

  1. Go to SQS Queues.

  2. In the resource table, select the queue and click Delete.

  3. In the dialog window, confirm the action.

You can also delete the queue in the Information tab on the queue page.

Managing messages#

Send a message#

  1. Go to SQS Queues.

  2. Find the desired queue in the resource table and click its name to go to the queue page.

  3. In the Messages tab, click Send message.

  4. In the dialog window:

    • enter a message in the Message text field;

    • (optional) modify the default delivery delay.

  5. Click Send.

Poll a queue#

Note

If there are more messages in the queue than N, the value of the Maximum number of messages parameter, the N oldest messages will be returned. If the visibility timeout is less than the poll duration, messages will be received more than once during the poll period.

  1. Go to SQS Queues.

  2. Find the desired queue in the resource table and click its name to go to the queue page.

  3. Open the Messages tab and make the poll settings if necessary:

    • poll duration (no more than 30 sec);

    • maximum number of requested messages (no more than 10).

  4. Click Poll queue.

The poll progress is shown by the respective indicator.

Stop polling#

Queue polling can be aborted at any time.

  1. Go to SQS Queues.

  2. Find the desired queue in the resource table and click its name to go to the queue page.

  3. To interrupt the current poll, click Stop polling in the Messages tab.

Receive message details#

  1. Go to SQS Queues.

  2. Find the desired queue in the resource table and click its name to go to the queue page.

  3. In the Messages tab, select the desired message in the resource table. In the window that opens, the following message information will be displayed:

    • ID;

    • date and time the message was sent;

    • message size;

    • MD5 message hash.

  4. To view the message text, click Show.

  5. To close the message details window, click .

Delete a message#

Note

To delete all messages from the queue, click Purge.

Important

A message can be deleted from the queue only during the poll. If the queue receives a delete command after the current visibility timeout expires, the message will not be deleted from the queue (for details, see Receipt handle).

  1. Go to SQS Queues.

  2. Find the desired queue in the resource table and click its name to go to the queue page.

  3. In the Messages tab, select the message you want to delete.

  4. Click an icon in the Actions field.

Information about queues and messages#

General information about queues can be viewed in the resource table in SQS Queues. To view detailed information about a particular queue and messages in it, find the desired queue in the table and click its ID. This will open the queue page.

The Information tab displays the queue data:

  • queue type (currently only Standard type is supported);

  • queue URL to which requests should be sent;

  • queue ARN;

  • queue creation date and time;

  • maximum message size;

  • date and time the queue was last updated;

  • default message retention period;

  • default visibility timeout;

  • default message delivery delay;

  • message wait time;

  • number of messages in the queue;

  • number of messages being processed;

  • number of messages whose delivery was delayed.

Here you can modify the queue parameters, purge the queue and delete the queue.

The Messages tab displays a resource table with general information about messages in the queue:

  • message ID;

  • date and time the message was sent;

  • message size;

  • how many times the message was requested.

Moreover, here you can view the current poll progress, change the poll duration and the maximum number of returned messages.

If you need to know the message hash and/or view its text, use the instruction. Here you can also create a new message, poll the queue, stop polling, and delete selected messages.