Message brokers#

General information#

The message broker service enables messaging between various applications, systems, and services in the cloud. It allows applications to communicate with each other via asynchronous messages, without the need for a permanent connection or knowledge of their reciprocal location. K2 Cloud supports popular message brokers Apache Kafka and RabbitMQ.

Apache Kafka#

Apache Kafka is a high-performance and reliable message broker designed to process large amounts of data in real time. It allows distributed system components to exchange messages quickly and efficiently.

Its distributed architecture makes Apache Kafka highly available and efficient for mission-critical applications. It guarantees message delivery and supports various data processing models, including streaming and batch processing. These features make Apache Kafka a frequent choice in various domains that require fast and reliable exchange of large amounts of data.

Apache Kafka offers a flexible system of topics and partitions that allow you to split the message stream into separate sub-streams, thus simplifying data processing. This structure provides higher throughput and allows users to scale the system to their requirements. The broker supports both individual messages and message batches, which makes it possible to optimize the performance and efficiency of data transfer.

Apache Kafka’s replication system provides protection against failures and data loss, while certain mechanisms such as commit log and consumer offsets help control the message processing and ensure data processing consistency.

Apache Kafka has become the standard for real-time data exchange across many industries and applications. This powerful and flexible message processing system continues to evolve and improve, offering new capabilities to process and analyze large amounts of data.

K2 Cloud supports the following versions of Apache Kafka:

  • 3.6.1;

  • 3.7.0.

RabbitMQ#

RabbitMQ is a popular open source message broker. It serves as an intermediary in the message transmission between information system components. Thanks to the broker, they no longer need to monitor message delivery status, resend messages or wait for the transmission to complete, since the broker solves all these tasks. Thus, you can improve overall system performance by reducing the number of jobs done by other system components and time spent on their execution.

RabbitMQ offers rich functionality for customizing and optimizing the messaging process according to the system needs. For example, you can use different protocols to send and receive messages, set up routing rules, specify message handling priorities, and more.

Thanks to a lot of messaging protocols supported, RabbitMQ can be used in multi-protocol systems. In particular, it supports AMQP (Advanced Message Queuing Protocol), MQTT (Message Queuing Telemetry Transport), STOMP (Streaming Text Oriented Messaging Protocol), and XMPP (Extensible Messaging and Presence Protocol).

In addition, RabbitMQ features high degree of reliability and availability. Built-in replication and clustering mechanisms allow you to create message-intensive HA systems.

K2 Cloud supports the following RabbitMQ versions:

  • 3.8;

  • 3.9;

  • 3.10.

Before you begin#

To be able to work with the broker service, a user needs to have PaaSServicePolicy project grants. For instance, such grants are available to administrators in CloudAdministrators group. If necessary, you can create a separate user, add the user to the project, and either attach the PaaSServicePolicy policy to the user or add the user to the group of cloud administrators in this project.

In addition, the project should have the following resources:

We also advise to read the recommendations on how to work with the service in the cloud.

Launching a message broker service#

To launch the service, go to the Service store or Running services subsection, select the service in the Message brokers tab and click Create.

The service launch procedure comprises the following stages:

  1. Set the required network parameters:

    • The configuration of the cluster in which the service will be deployed. The following options are available for selection:

      • single-node (non-high-availability service);

      • three notes in one availability zone;

      • three nodes in three availability zones.

      • six nodes in a single availability zone (3 brokers and 3 coordinators);

      • six nodes in three availability zones (3 brokers and 3 coordinators).

    • VPC where the service will be deployed.

    • Security groups to control traffic through interfaces of the instances on which the service will run.

    • Subnets where the message broker service will be deployed, or network interfaces through which cluster nodes will connect to subnets.

    Note

    To run the service in the selected VPC, you must first create a subnet in the preferred availability zone (in a configuration with one zone), or one subnet in each availability zone (in a configuration with three zones). In addition, the same volume types must be supported in the availability zones used.

    Note

    The ability to attach network interfaces may be useful, for example, when you need to recreate the cluster where the message broker has been deployed. If you delete a service, but do not delete attached network interfaces, you will be able to reuse them for connecting nodes of a new cluster to subnets, where the new cluster will be deployed. Thus, you can keep previous network settings, such as private IP addresses and security groups, rather than configure them again.

    • (Only for RabbitMQ) Internal and/or internet-facing load balancers when the three-node cluster configuration is selected (for details, see Load balancer management).

  2. Set the configuration of one or more instances on which the message broker service will run.

    Select the configuration of the instance where the master node (in case of RabbitMQ) or the broker (in case of Apache Kafka) will be deployed:

    • instance type;

    • system volume parameters: type, size and IOPS (if available for the type);

    • data volume parameters: type, size and IOPS (if available for the type).

    For Apache Kafka service, select also the configuration of the instance where the coordinator will be deployed.

    In addition, you can specify an SSH key. In this case, after automatic service configuration, you will have SSH access to the respective instances.

    Attention

    You can connect to instances using an SSH key while the message brokder service is in a beta testing stage. This feature may be disabled in the future.

  3. Set the main service parameters:

    • Service name – any unique name for the caching service.

    • Version.

    • Monitoring agent installation option. For centralized monitoring of a message broker service, deploy the Prometheus-based monitoring service first.

      • Monitoring service – The selection of a monitoring service is only available when the Enable monitoring checkbox is checked.

      • Monitoring labels — Optionally, you can set monitoring labels, which the installed monitoring agents will assign to the collected metrics (for details, see Using labels). Labels can only be set when the Enable monitoring checkbox is selected.

    • Logging agent installation option – For centralized PaaS service logging, first create a logging service. Once a PaaS service is created, you will be able to install logging agents manually only.

      • Logging service – The selection of a logging service is only available when the Enable logging checkbox is checked.

      • Logging tags – Tags are assigned to logs when the latter are imported by the logging service. They simplify search for the required logs. Tags can only be set when the Enable logging checkbox is checked.

    • User password (for RabbitMQ only). It can be set manually or generated automatically.

  4. Set the advanced settings, if necessary. Click advanced settings and enter the settings and their values.

    Important

    The specified settings will be a part of the service configuration and, therefore, will affect its operation. Add only the settings you really need.

  5. Click Create.

    Note

    The service launching process usually takes 5 to 15 minutes.

Managing Kafka topics#

For an Apache Kafka service, you can create and set up topics.

Creating a topic#

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Topics tab and click Create.

  4. In the window that opens, set the following required parameters:

    • Topic name.

    • Partitions (not more than that of brokers).

    • Replication factor (not more than that of brokers).

  5. If you need to set other parameters, proceed to the next step by clicking Additional parameters. The list of supported parameters and their valid values can be found on the page Kafka topic parameters.

  6. After setting all the required parameters, click Create.

Editing a topic#

To change the topic parameters, follow these steps:

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Topic tab, select the topic in the resource table and click on:bdg-light:Modify.

  4. You can modify the following parameters of the topic:

    • Partitions (can only be increased).

    • Number of the partition replicas.

  5. If you need to modify the values of other parameters, proceed to the next step by clicking Additional parameters. The list of supported parameters and their valid values can be found on the page Kafka topic parameters.

  6. After setting all the required parameters, click Save.

Deleting a topic#

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Topics tab, select the topic in the resource table and click Delete.

  4. In the dialog window, confirm the action.

Load balancer management#

For RabbitMQ service in a three-node high-availability configuration, you can create an internal and/or internet-facing load balancer (for details, see load balancers). They automatically distribute incoming requests among cluster nodes.

Load balancers are created automatically; their parameters and associated resources cannot be changed. Information about the created load balancers, in particular their DNS names, can be found in the Load balancers tab on the service page.

Attention

A load balancer running together with a PaaS service can be deleted only on the page of this service.

Important

To run the internet-facing load balancer, give external access to HAProxy ports, which is denied by default. Add enabling rule for tcp/5000 port to the security group that was specified when creating the service.

Create a load balancer#

A load balancer for a message broker service can only be created when the service is in Running status. You can create one internal and one internet-facing load balancer per service.

Important

You can create an internal load balancer only if route propagation is enabled in VPC.

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Create.

  4. In the window that opens, select a balancer you want to create. If none has been created yet, you can create both internal and internet-facing balancers at once.

  5. Click Create to complete the action.

Delete load balancer#

Load balancer associated with the service can only be deleted when the service is in Running status.

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Delete.

  4. In the window that opens, select a load balancer you want to delete. If two load balancers have been created for the service, you can delete both at the same time.

  5. Click Delete to confirm the action.

Configuring a message broker service#

If you have not enabled monitoring and/or logging when creating the message broker service, or you want to disable them, you can do it when the service is in the Ready state. You can also change the password and advanced settings.

Note

To enable monitoring and logging, first deploy Prometheus-based monitoring service and ELK-based logging service.

Important

If an attempt to modify some settings fails, then the service will be reset to default ones.

To set up the message broker service:

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Parameters tab and click Modify.

  4. In the window that opens, you can generate or set a new password and configure monitoring and logging (or disable them if they are already enabled).

  5. To change the advanced settings, click Additional parameters. Then click Advanced settings and set the new settings and their values.

  6. To save settings, click Modify.

Environment upgrade#

PaaS services are updated regularly. If you want the already deployed message broker service to support new features, then update its environment. For the current environment version, see the Information tab on the service page.

Note

All services with environment version 3_6 and higher support environment update. It is also available for some previously deployed services with environment version 3_5. To check if you can update them, use the API method DescribeService: the response should contain the common:update_environment value in the SupportedFeatures list.

Important

If an attempt to update the environment fails, the service will be reset to the default one.

To update the environment version:

  1. Go to PaaS Running services and open the Message brokers tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. In the Information tab, click Update environment version.

  4. In the window that opens, select the version from the list, to which you want to upgrade the current environment.

  5. Click Update to change the version.

Deleting a message broker service#

Deleting a message broker service deletes all instances and volumes created with it.

You can delete the service using one of the following methods.

  1. Go to the section PaaS Running services.

  2. Open the Message brokers tab.

  3. Find the service in the table and click on the icon .

  4. If you want to use network interfaces in the future, for example, to recreate a service on the cluster with the same network parameters, then, in the window that opens, disable the Delete associated network interfaces option.

  5. Confirm the action.

  1. Go to the section PaaS Running services.

  2. Open the Message brokers tab.

  3. Find the service in the table and go to the service page.

  4. Click :Delete in the Information tab.

  5. If you want to use network interfaces in the future, for example, to recreate a service on the cluster with the same network parameters, then, in the window that opens, disable the Delete associated network interfaces option.

  6. Confirm the action.