Search and analytics#

General information#

Elasticsearch automated service in K2 Cloud allows you to quickly deploy and configure search, analysis, and monitoring of all data types.

Elasticsearch is a distributed search and analytics system that provides a unified data storage for Elastic Stack based solutions: Elasticsearch, Elastic Agent, Beats, Logstash and Kibana. To learn more about Elasticsearch, visit the official website.

Along with Elasticsearch, K2 Cloud supports Kibana service for convenient data management and visualization.

K2 Cloud supports the following Elasticsearch versions:

  • 7.11.2;

  • 7.12.1;

  • 7.13.1;

  • 7.14.2;

  • 7.15.2;

  • 7.16.3;

  • 7.17.4;

  • 8.0.1;

  • 8.1.3.

Before you begin#

To be able to work with the search and analysis service, a user needs to have PaaSServicePolicy project grants. For instance, such grants are available to administrators in CloudAdministrators group. If necessary, you can create a separate user, add the user to the project, and either attach the PaaSServicePolicy policy to the user or add the user to the group of cloud administrators in this project.

In addition, the project should have the following resources:

Мы советуем также ознакомиться с рекомендациями по работе с сервисом в облаке.

Starting the Elasticsearch service#

To start the service, go to the Services store or Running Services subsection, select the Elasticsearch service in the Search and analytics tab and click Create.

The service launch procedure comprises the following stages:

  1. Set the required network parameters for an Elasticsearch service:

    • The configuration of the cluster in which the service will be deployed. The following options are available for selection:

      • single-node (non-high-availability service);

      • three notes in one availability zone;

      • three nodes in three availability zones;

    • VPC where the service will be deployed.

    • Security groups to control traffic through interfaces of the instances on which the Elasticsearch service will run.

    • Subnets to which instances with the running service will be attached, or network interfaces through which cluster nodes will be attached to subnets.

    Note

    To run the service in the selected VPC, you must first create a subnet in the preferred availability zone (in a configuration with one zone), or one subnet in each availability zone (in a configuration with three zones). In addition, the same volume types must be supported in the availability zones used.

    Note

    The ability to attach network interfaces may be useful, for example, when you need to recreate a search and analytics service. If you delete the service, but do not delete attached network interfaces, you will be able to reuse them for connecting nodes of the new cluster to subnets, where the new cluster will be deployed. Thus, you can keep previous network settings, such as private IP addresses and security groups, rather than configure them again.

    • Internal and/or internet-facing load balancers when the three-node cluster configuration is selected (for details, see Load balancer management).

  2. Set the configuration of one or more instances on which the data search and analysis service will run.

    Select the instance type and parameters of its volumes: type, size and IOPS (if available for a type).

    Note

    The Elasticsearch service performance depends on the node components. We recommend using the Memory Optimized instance type.

    In addition, you can specify an SSH key. In this case, after automatic service configuration, you will have SSH access to the respective instances.

    Attention

    We provide the option to connect to instances using an SSH key while the new Elasticsearch service is beta testing. This feature may be disabled in the future.

    To enable a configuration with cluster arbitrator, tick the Use arbitrator checkbox.

  3. Set the main service parameters:

    • Service name – any unique name for the caching service.

    • Version.

    • Option to install Kibana, a visualization and reporting tool.

    • Monitoring agent installation option — For centralized monitoring of the search and analysis service, deploy the Prometheus-based monitoring service first.

      • Monitoring service – The selection of a monitoring service is only available when the Enable monitoring checkbox is checked.

      • Monitoring labels — Optionally, you can set monitoring labels, which the installed monitoring agents will assign to the collected metrics (for details, see Using labels). Labels can only be set when the Enable monitoring checkbox is selected.

    • Logging agent installation option – For centralized PaaS service logging, first create a logging service. Once a PaaS service is created, you will be able to install logging agents manually only.

      • Logging service – The selection of a logging service is only available when the Enable logging checkbox is checked.

      • Logging tags – Tags are assigned to logs when the latter are imported by the logging service. They simplify search for the required logs. Tags can only be set when the Enable logging checkbox is checked.

    • Elasticsearch user password. You can set it manually or generate automatically. If a password is set, authentication is required to log in to Kibana.

  4. Set the advanced settings, if necessary. Click advanced settings and enter the settings and their values.

    Important

    The specified settings will be a part of the service configuration and, therefore, will affect its operation. Add only the settings you really need.

  5. Click Create.

    Note

    The service launching process usually takes 5 to 15 minutes.

Load balancer management#

For Elasticsearch service in a high-availability configuration, you can create an internal and/or internet-facing load balancer (for details, see load balancers). They automatically distribute incoming requests among fully functional cluster nodes (requests are not sent to the arbitrator if there is any).

Load balancers are created automatically; their parameters and associated resources cannot be changed. Information about the created load balancers, in particular their DNS names, can be found in the Load balancers tab on the service page.

Attention

Load balancer running together with PaaS service can be deleted only on the page of this service.

Important

To run the external load balancer, give external access to service ports, which is denied by default. Add enabling rule for the corresponding ports to the security group that was specified when creating the service. The ports listened by Elasticsearch service can be found in the Information tab on its page.

Create a load balancer#

Database service load balancer can only be created when the service is in Running status. You can create one internal and one external load balancer per service.

Important

You can create an internal load balancer only if route propagation is enabled in VPC.

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Create.

  4. In the window that opens, select a balancer you want to create. If none has been created yet, you can create both internal and internet-facing balancers at once.

  5. Click Create to complete the action.

Delete load balancer#

Load balancer associated with the service can only be deleted when the service is in Running status.

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Delete.

  4. In the window that opens, select a load balancer you want to delete. If two load balancers have been created for the service, you can delete both at the same time.

  5. Click Delete to confirm the action.

Elasticsearch service configuration#

If you have not enabled monitoring and/or logging when creating the search and analysis service, or you want to disable them, you can do it when the service is in the Ready state. You can also change the advanced settings.

Note

To enable monitoring and logging, first deploy Prometheus-based monitoring service and ELK-based logging service.

Important

If an attempt to modify some settings fails, then the service will be reset to default ones.

To set up the Elasticsearch service parameters:

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Parameters tab and click Modify.

  4. In the window that opens, you can configure monitoring and logging (or disable them if they are already enabled).

  5. To change the advanced settings, click Additional parameters. Then click Advanced settings and set the new settings and their values.

  6. To save settings, click Modify.

Environment upgrade#

PaaS services are updated regularly. If you want an already deployed Elasticsearch service to support new features, then update its environment. For the current environment version, see the Information tab on the service page.

Note

All services with environment version 3_6 and higher support environment update. It is also available for some previously deployed services with environment version 3_5. To check if you can update them, use the API method DescribeService: the response should contain the common:update_environment value in the SupportedFeatures list.

Important

If an attempt to update the environment fails, the service will be reset to the default one.

To update the environment version:

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. In the Information tab, click Update environment version.

  4. In the window that opens, select the version from the list, to which you want to upgrade the current environment.

  5. Click Update to change the version.

Connecting snapshot repository#

Elasticsearch built-in snapshot creation mechanism can backup indexes, data flows, and cluster state. The resultant snapshots can be used to recover data after failure or migrate data between Elasticsearch services. K2 Cloud automates the creation and registration of the snapshot repository for an Elasticsearch service.

Snapshots are stored in object storage buckets. Along with a repository, a directory is created in the bucket and mounted on each node of the Elasticsearch cluster as an s3fs file system. Thus, you can use the same bucket to store snapshots of different services and types. Storing snapshots as files simplifies snapshot sharing: users can upload their own snapshots for service recovery or download existing ones.

Cloud tools can be used only to connect the snapshot repository, so Elasticsearch built-in tools should be used to manage the snapshots. Snapshots can be created using Elasticsearch own API or Snapshot Lifecycle Management (SLM). To be able to recover the index and other data, the index, the snapshot and the cluster version should be compatible with each other.

Add a snapshot repository#

You can create a new snapshot repository or connect an existing one of another service. To add the repository, follow two steps: first create/connect it and then register in Elasticsearch.

The ability to connect the existing repository can be useful for recovering the service or migrating data from one service to another. You can also reuse an existing repository if it is not registered to another service.

Note

If the repository is registered to another service, it can be connected as read-only. Only one repository can be connected to a service, so disconnect the repository after data recovery/migration and create a new one if you want to back up the service.

Add a new repository#

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, set the following parameters (the Existing repository option should be disabled):

    • The bucket where the snapshot repository will be created.

    • User with PaaSBackupUser privileges; backups will be written to the bucket under this user.

    • The directory that will be mounted as the repository.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Add a registered repository#

You can connect a registered repository to another service only as a read-only repository

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, select the Existing repository checkbox and in the Recovery service field select the service, whose storage you want to connect.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Add an unregistered repository#

The process of connecting an unregistered repository is similar to creating a new one. The difference is that in this case, you have to specify an existing directory previously used as the snapshot repository.

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, set the following parameters (the Existing repository option should be disabled):

    • The bucket where the snapshot repository will be created.

    • User with PaaSBackupUser privileges; backups will be written to the bucket under this user.

    • The directory previously used as the repository.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Disconnect a snapshot repository#

Disconnect the repository in the reverse order: first unregister the repository in the service, and then remove it. The file system is unmounted, but the snapshot directory is not deleted.

  1. Go to PaaS Running services and open the Search and analytics tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and disable Snapshot registration

  4. Wait until the service status changes to Running and click Delete.

  5. In the dialog window, confirm the deletion.

Deleting an Elasticsearch service#

Deleting the Elasticsearch service deletes all instances and volumes created with it.

You can delete the service using one of the following methods.

  1. Go to the section PaaS Running services.

  2. Open the Search and analytics tab.

  3. Find the service in the table and click on the icon .

  1. Go to the section PaaS Running services.

  2. Open the Search and analytics tab.

  3. Find the service in the table and go to the service page.

  4. Click :Delete in the Information tab.