Logging#

General information#

ELK cloud service allows for the centralized and automated data collection from PaaS service logs. In addition, it can be used to extract data from logs of other systems. The service is based on Elastic Stack consisting of:

  • Elasticsearch to store and index logs;

  • Logstash to filter and process logs;

  • Kibana to visualize retrieved data.

Activity logs are integral to the monitoring of the overall system health. When running, applications and services log various debugging information, save error messages and warnings, and log actions and operations. This helps quickly identify service problems and analyze any abnormal situation, be it a failure or performance degradation. In addition, based on the data stored, you can make forecasts and take measures to prevent problems in the future.

Analyzing a problem can take a lot of time and effort if you manually scrutinize log files of different components from multiple servers and try to find correlations. Moreover, programs often log a lot of excessive data that is only needed under certain circumstances. In such cases, it is critical to be able to quickly find and filter data to extract the required information from different sources.

Kibana’s advanced capabilities allow you to perform a full-text search across all logs, select only desired time intervals, show/hide individual message fields, and count the number of events. This helps quickly and easily filter the required information, highlight it in search results, and, if necessary, visualize as charts and graphs.

ELK service is easy-to-use and scalable. It helps timely monitor the health of various services and receive application performance data in a convenient form for analysis and decision-making in order to optimize application performance and ensure the service uptime through quick troubleshooting.

Billing#

The PaaS logging service ELK is free of charge. Payment is charged for actually consumed billable resources: instances, volumes, Elastic IP addresses, and other resources.

Before you begin#

We advise to preliminarily read the recommendations on how to work with the service in the cloud.

Required privileges#

To be able to work with the logging service, a user needs to have PaaSFullAccess project grants. For instance, such grants are available to administrators in the CloudAdministrators group. If necessary, you can create a separate user, add the user to the project, and either attach the PaaSFullAccess policy to the user or add the user to the group of cloud administrators in this project.

The PaaSFullAccess project grants provide the user with full permissions to all PaaS services. To allow only particular actions with the logging service for a user, create a policy with the appropriate grants and assign it to the user. For details, see the IAM documentation on how to create a policy and assign it to the user.

Required resources#

There should be the following resources in the project:

ELK service launch#

To launch the service, go to the Service store or Running services subsection, select ELK service in the Logging tab and click Create.

Important

The logging service can be launched only in those subnets that have Internet access. To configure such access, you might need to attach an internet gateway to a VPC and create a default route in a route table.

Note

Creating a high-availability service also automatically creates placement groups for every three nodes. For example, in case of a five-node cluster in one availability zone, two placement groups are created, containing three and two nodes, respectively. Should it be a five-node cluster across three availability zones, one placement group is created in each zone, with two groups containing two nodes each, and one group containing only one node. Instances from the same group are deployed on different physical servers, which improves the service availability.

The service launch procedure comprises the following stages:

  1. Set the network parameters required for ELK service:

    Note

    To run the service in the selected VPC, you must first create a subnet in the preferred availability zone (in a configuration with one zone), or one subnet in each availability zone (in a configuration with three zones). In addition, the same volume types must be supported in the availability zones used.

    Note

    The ability to attach network interfaces may be useful, for example, when you need to recreate the cluster where the logging service has been deployed. If you delete a service, but do not delete attached network interfaces, you will be able to reuse them for connecting nodes of a new cluster to subnets, where the new cluster will be deployed. Thus, you can retain previous network settings, such as private IP addresses and security groups, rather than configure them again.

    Important

    It is not recommended to create a four-node cluster as otherwise any connectivity loss may lead to quorum loss due to split brain.

    • Cluster configuration where the service will be deployed.

      You can select only configurations with the following number of nodes:

      • one master node (the service without high availability);

      • two master nodes and one arbitrator node;

      • from three to nine master nodes 1.

      1

      We recommend selecting this configuration if the cluster scalability may be required in the future.

    • VPC where the logging service will be deployed.

    • Security groups to control traffic through interfaces of the instances on which the logging service will run.

    • Subnets to which instances with the running service will be attached, or network interfaces through which cluster nodes will be attached to subnets.

      Note

      When selecting subnets, you may specify either one or three subnets in different availability zones. In the former case, the service will be deployed in the same availability zone as the selected subnet.

      When selecting network interfaces for a multi-node configuration, you must specify as many interfaces as there are nodes in the cluster.

      Network interfaces should be evenly distributed across all availability zones where the service will be deployed, if possible. For example, if you plan to create a cluster with five master nodes across three availability zones, you should specify two interfaces in each of two availability zones and one interface in the remaining zone.

    • Internal and/or internet-facing load balancers (for details, see Load balancer management).

      In the DNS name field, you can specify an arbitrary subdomain name in the paas.<region>.web.c2.croc.ru DNS service zone for an Internet-facing load balancer and in the paas.<vpc-id>.internal zone for an internal load balancer. The name may contain Latin letters, digits, and hyphens.

      If the DNS name field is left blank, the name will be specified automatically.

    Click Next to proceed to the next step.

  2. If the cluster configuration three to nine master nodes was selected in the previous step, you can set the number of master nodes in the cluster. However, if network interfaces were specified, the number of instances is fixed and equal to the number of selected network interfaces.

    В блоке Мастер-узел задайте конфигурацию экземпляров, которые будут использоваться в качестве мастер-узлов. Выберите тип экземпляра.

    Note

    The ELK service performance depends on the node components. We recommend using the Memory Optimized instance type.

    Если на предыдущем шаге была выбрана конфигурация с кластерным арбитром, то в блоке Арбитр задайте конфигурацию экземпляра, который будет использоваться в качестве арбитра. Выберите тип экземпляра.

    Click Next to proceed to the next step.

  3. Set the main service parameters:

    • Name tag — Optional arbitrary description of the service.

    • Service name – any unique name for the caching service.

    • Elasticsearch version.

    • Elasticsearch superuser password. It can be set manually or generated automatically.

      Note

      To configure anonymous access to Kibana, set a password first.

      Anonymous access to Kibana is allowed by default if the password is not set. Otherwise, you should authenticate with Kibana by default. You can modify this setting and allow anonymous access. In this case, select the role for anonymous access with viewing (viewer) or editing (editor) rights.

    • Enable monitoring option. For centralized monitoring of a logging service, first deploy the Prometheus-based monitoring service.

      When selecting this option, also select the monitoring service you want to use. Optionally, you can set monitoring labels, which the installed monitoring agents will assign to collected metrics (for details, see labels).

    Click Next to proceed to the next step.

  4. Set the maintenance window (see подробнее). If scheduled maintenance can be performed at any time, leave the Maintenance window option set to Custom. If there are time restrictions when maintenance can be performed, select the interval that is convenient for you. To do so, specify the following parameters:

    • the Scheduled value for the Maintenance window option;

    • a day of the week for the maintenance to take place;

    • hour interval for the maintenance window.

    To set additional parameters or tags for the service, click the respective button to move to the required step — Additional parameters or Add tags. If neither need to be set, click Create to launch the service.

  5. Set the advanced settings, if necessary. Click advanced settings and enter the settings and their values.

    Important

    The specified settings will be a part of the service configuration and, therefore, will affect its operation. Add only the settings you really need.

    To set tags for the service, click Add tags to go to the next step. If no tags are required, click Create to launch the service.

  6. To add a tag, click Add tag and enter tag key and value.

    Note

    If no tags have been set yet, you can add the Name tag by clicking Add Name tag and specifying its value.

    When done, click Create to launch the service.

Important

The service launching process usually takes 5 to 15 minutes.

Modifying service node options#

Add nodes for the service#

If the cluster configuration three to nine master nodes was selected at the ELK service creation and the cluster has fewer than nine nodes, you can scale it out by adding extra nodes. Scaling is only possible when the service is in the Running state.

Important

For the logging service created before 30.01.2026, cluster scaling is not supported.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Information tab and click Add instances.

  4. In the window that opens, specify the number of instances to be added. If network interfaces were specified at the service creation, you can either create interfaces automatically by pushing Create automatically or specifying the existing ones by pushing Select manually and selecting the interfaces in the specified availability zones.

  5. Click Add to complete the operation.

Modify the instance type for the node#

You can modify the instance type for cluster nodes running the logging service. You can do it only when the service is in the Ready state.

Important

Instance type cannot be modified for nodes with the arbitrator role.

To modify the instance type:

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. In the Information tab, edit the Instance type field in the configuration section of master node. To do this, click the edit icon and selecting the instance type.

  4. Click to save changes.

Modify data volume parameters#

You can only change the data volume parameters when the service is in the Ready state.

You can configure the data volume size and its performance in IOPS right from the cloud web interface.

Important

You can change IOPS only for a data volume of the type io2.

To change the volume parameters, follow these steps:

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. In the Information tab, edit the required fields (Data volume size and/or Data volume IOPS) by clicking the edit icon .

    Important

    The new data volume size must exceed the current one.

  4. Click to save changes.

Load balancer management#

For ELK service, you can create an internal and/or internet-facing load balancer (for details, see load balancers). They automatically distribute incoming requests among fully functional cluster nodes (requests are not sent to the arbitrator if there is any).

Load balancers are created automatically; their parameters and associated resources cannot be modified. To modify the DNS name of an existing load balancer, delete it first and then create a new one. For information about the created load balancers and, in particular, the service addresses, open the Load balancers tab on the service page.

Attention

Load balancer running together with PaaS service can be deleted only on the page of this service.

Important

To run the external load balancer, give external access to service ports, which is denied by default. Add enabling rule for the corresponding ports to the security group that was specified when creating the service. The ports listened by ELK service can be found in the Information tab on its page.

Create a load balancer#

A logging service load balancer can only be created when the service is in Running status. You can create one internal and one external load balancer per service.

Important

You can create an internal load balancer only if route propagation is enabled in VPC.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Create.

  4. In the window that opens, select a balancer you want to create. If none has been created yet, you can create both internal and internet-facing balancers at once.

    In the DNS name field, you can specify an arbitrary subdomain name in the paas.<region>.web.c2.croc.ru DNS service zone for an Internet-facing load balancer and in the paas.<vpc-id>.internal zone for an internal load balancer. The name may contain Latin letters, digits, and hyphens.

    If the DNS name field is left blank, the name will be specified automatically.

  5. Click Create to complete the action.

Delete load balancer#

Load balancer associated with the service can only be deleted when the service is in Running status.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Load balancers tab and click Delete.

  4. In the window that opens, select a load balancer you want to delete. If two load balancers have been created for the service, you can delete both at the same time.

  5. Click Delete to confirm the action.

Processing the event logs#

Logstash component allows you to filter, aggregate, and modify event logs before they are sent to Elasticsearch. You can configure a pipeline to automatically pre-process logs. The pipeline defines the data inflow processing rules and can contain input, filter, output, and codec blocks. Plug-ins from the input block generate events, filter ensures data pre-processing, and output forwards data to the final destination. The codec changes data representation and can be applied to input and/or output to decode/encode data.

To create a pipeline, describe its configuration, namely, the plugins to be used along with their settings. For details about Logstash features, available plugins, and their configuration options, see the official documentation.

ELK service uses a preconfigured pipeline to automatically connect and process event logs from other PaaS services. It receives input data from filebeat agents and, depending on the settings and the version of Elasticsearch used, writes the data either to an index or to a data stream named <beatname>-<version>. The output plugin is configured to log to the Elasticsearch repository all events that have an index field in the @metadata.

You can use a preconfigured output plugin to log events from your own systems to ELK, even if such events are not generated by -beat agents (filebeat, metricbeat, auditbeat), as long as their metadata contains the index field. Alternatively, you can add your own plugins to output, but remember that each new plugin needs a new client connection for logging to Elasticsearch. To use the ELK service more efficiently, it is advisable to minimize the number of individual connections, so it is better to reuse existing output plugins instead of creating new ones.

Create a pipeline#

To set a pipeline configuration:

  • Go to the section PaaS Running services.

  • Open the Logging tab and click the ELK service name to go to its page.

  • Open the Pipelines tab and click Create.

  • In the window that opens, specify the pipeline name and configuration.

  • Click Create.

Attention

The beats-to-elasticsearch pipeline name is reserved by the system. It is impossible to create a pipeline under this name.

Modify a pipeline#

To modify a pipeline configuration:

  • Go to the section PaaS Running services.

  • Open the Logging tab and click the ELK service name to go to its page.

  • Open the Pipelines tab, select the pipeline from the list and click Modify.

  • Edit the pipeline configuration.

  • Click Save.

Delete a pipeline#

To delete a pipeline:

  • Go to the section PaaS Running services.

  • Open the Logging tab and click the ELK service name to go to its page.

  • Open the Pipelines tab, select a pipeline from the list, and click Delete. You can select multiple pipelines for deletion at the same time.

  • In the dialog window, confirm the action.

ELK service configuration#

If you have not enabled monitoring when creating the logging service, or you want to disable it, you can do it when the service is in the Ready state.

Note

To enable monitoring, deploy the Prometheus-based monitoring service first.

Important

If an attempt to modify monitoring settings fails, then the service will be reset to default ones.

To enable monitoring for the logging service:

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Parameters tab and click Modify.

  4. In the window that opens, you can configure monitoring (or disable it if it’s already enabled).

  5. To save settings, click Modify.

Connecting snapshot repository#

Elasticsearch built-in snapshot creation mechanism can backup indexes, data flows, and Elasticsearch cluster state. The resultant snapshots can be used to recover data after failure or migrate Elasticsearch data between ELK services. K2 Cloud automates the creation and registration of the snapshot repository for an ELK service.

Snapshots are stored in object storage buckets. Along with a repository, a directory is created in the bucket and mounted on each node of the Elasticsearch cluster as an s3fs file system. Thus, you can use the same bucket to store snapshots of different services and types. Storing snapshots as files simplifies snapshot sharing: users can upload their own snapshots for service recovery or download existing ones.

Cloud tools can be used only to connect a snapshot repository, so Elasticsearch built-in tools should be used to manage the snapshots. Snapshots can be created using Elasticsearch own API or Snapshot Lifecycle Management (SLM).

Add a snapshot repository#

You can create a new snapshot repository or connect an existing one of another service. To add the repository, follow two steps: first create/connect it and then register in Elasticsearch.

The ability to connect an existing repository can be useful for recovering Elasticsearch data or migrating it from one ELK service to another. You can also reuse an existing repository if it is not registered to another service.

Note

If the repository is registered to another service, it can be connected as read-only. Only one repository can be connected to a service, so disconnect the repository after data recovery/migration and create a new one if you want to back up the service.

Add a new repository#

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, set the following parameters (the Existing repository option should be disabled):

    • The bucket where the snapshot repository will be created.

    • User with PaaSBackupUser privileges; backups will be written to the bucket under this user.

    • The directory that will be mounted as the repository.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Add a registered repository#

You can connect a registered repository to another service only as a read-only repository.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, select the Existing repository checkbox and in the Recovery service field select the service, whose storage you want to connect.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Add an unregistered repository#

The process of connecting an unregistered repository is similar to creating a new one. The difference is that in this case, you have to specify an existing directory previously used as the snapshot repository.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and click Create.

  4. In the window that opens, set the following parameters (the Existing repository option should be disabled):

    • The bucket where the snapshot repository will be created.

    • User with PaaSBackupUser privileges; backups will be written to the bucket under this user.

    • The directory previously used as the repository.

  5. Click Add to create the repository.

  6. Wait until the service status changes to Running and enable Repository registration.

Disconnect a snapshot repository#

Disconnect the repository in the reverse order: first unregister the repository in the service, and then remove it. The file system is unmounted, but the snapshot directory is not deleted.

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Snapshot repositories tab and disable Repository registration.

  4. Wait until the service status changes to Running and click Delete.

  5. In the window that opens, confirm the deletion.

Assigning tags#

Important

Tags are only supported for PaaS services with an fm-cluster-xxxxxxxxxxx ID. If you want to add tag support for the installed services with a UUID as an ID, please contact the support service.

Tags allow you to classify PaaS services by various attributes, such as purpose, owner, etc. Their usage simplifies the management of many similar resources.

To add, modify or delete tag for the logging service:

  1. Go to PaaS Running services and open the Logging tab.

  2. In the resource table, find the service for which you want to assign tags, and click on its ID to go to the service page.

  3. Open the Tags tab.

  4. To add a tag, click Add tag and specify the Key and Value fields.

    To modify a tag, edit the required fields (Value and/or Key) of the respective tag.

    To delete a tag, click the icon next to the tag you no longer need.

    Note

    If no tags have been set earlier, you can add the Name tag by clicking Add Name tag.

  5. Click Apply to save the changes.

Service start and stop#

You can stop the logging service, if you don’t need it, and start it when necessary, thus saving money on computing resources, because only attached volumes are billed when the service is stopped.

You can also restart the service in case of an error, and synchronize the service status with those of the instances where it is installed.

Stop service#

The service can be stopped when it is in the Running or Error state.

  1. Go to PaaS Running services and open the Logging tab.

  2. In the resource table, find the service you want to stop and click the shutdown icon in the Actions field. Alternatively, click the service ID to go to its page and then click Stop in the Information tab.

  3. In the dialog window, confirm the action.

Restart the service#

Note

You can only restart services with the environment version 3.6 or higher. You can find out the current version of the environment in the Information tab on the service page.

The service can be restarted when it is in the Running or Error state.

  1. Go to PaaS Running services and open the Logging tab.

  2. In the resource table, find the service you want to restart and click the restart icon in the Actions field. Alternatively, click the service ID to go to its page and then click Restart in the Information tab.

  3. In the dialog window, confirm the action.

Synchronize the service with instances#

Synchronization allows you to match the states of the service and the instances on which it is installed, when instances were manually started after the service was stopped. It can also be used to match the states if an error occurred when stopping, starting, or restarting the service and instances had to be manually stopped/started.

Attention

Synchronization is not possible if the error cannot be eliminated (the error status can be viewed, for example, using the DescribeService method).

  1. Go to PaaS Running services and open the Logging tab.

  2. In the resource table, find the service you want to synchronize and click the synchronization icon in the Actions field. Alternatively, click the service ID to go to its page and then click Synchronize with confirmation in the Information tab.

  3. In the dialog window, confirm the action.

Start the service#

You can only start the service, when it is in the Stopped state.

  1. Go to PaaS Running services and open the Logging tab.

  2. In the resource table, find the service you want to start and click the start icon in the Actions field. Alternatively, click the service ID to go to its page and then click Start in the Information tab.

  3. In the dialog window, confirm the action.

Configuring the maintenance window#

During scheduled maintenance, critical security patches and other important updates are installed.

Attention

During maintenance, operations with the service, including its deletion, are not available, but the service itself will continue to operate. In some cases, it may be necessary to stop and restart the cluster nodes.

Maintenance is carried out strictly during a preset time period (if a maintenance window is specified). All actions of K2 Cloud specialists during maintenance are logged.

Note

You can modify the maintenance window settings when the service is in the Running state only.

To modify the maintenance window settings:

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. In the Maintenance tab, click Set up.

  4. In the window that opens, specify the maintenance interval that is convenient for you. To do so, select the following parameters:

    • the Scheduled value for the Maintenance window option;

    • a day of the week for the maintenance to take place;

    • hour interval for the maintenance window.

    Or select the Custom value for the Maintenance window option, if maintenance can be performed at any time.

  5. Click Save to modify the maintenance window.

Viewing metrics#

For the logging service, the following metrics are available:

Metrics

Description

logstash_jvm_mem_heap_used_percent

Share of JVM heap memory in use

logstash_node_pipeline_queue_size_in_bytes

Current size of data pipeline queue

logstash_pipeline_queue_events_count

Number of events in data pipeline queue

logstash_process_cpu_percent

Share of CPU utilization by Logstash

logstash_up

Indicates whether the Logstash server is up

To view the metrics of a specific service:

  1. Go to PaaS Running services and open the Logging tab.

  2. Find the desired service in the table and click the service ID to go to its page.

  3. Open the Monitoring tab and select:

    • metrics,

    • instance to collect metric from.

    Note

    Instances available for a particular metric depend on their role. For certain metrics, there can be additional selectors apart from instance choice.

    Go to Display parameters to set the period the metric graph covers, statistics, and metric calculation interval. To automatically refresh the graph, enable Auto-refresh; to refresh it manually, click Refresh.

Deleting ELK service#

Deleting ELK service deletes all instance, volumes and placement groups created with it.

You can delete the service using one of the following methods.

  1. Go to the section PaaS Running services.

  2. Open the Logging tab.

  3. Find the service in the table and click on the icon .

  4. If you want to use network interfaces in the future, for example, to recreate a service on the cluster with the same network parameters, then, in the window that opens, disable the Delete associated network interfaces option.

  5. Confirm the action.

  1. Go to the section PaaS Running services.

  2. Open the Logging tab.

  3. Find the service in the table and go to the service page.

  4. Click :Delete in the Information tab.

  5. If you want to use network interfaces in the future, for example, to recreate a service on the cluster with the same network parameters, then, in the window that opens, disable the Delete associated network interfaces option.

  6. Confirm the action.