Tutorials#

How to prepare OS for supporting consistent backup#

To ensure snapshot consistency during backup, install and configure QEMU Guest Agent used by the backup service to freeze I/O in the guest OS. In case of Linux, additional preparation of applications is required to provide application data consistency.

Windows preparation#

To ensure consistency of backups of Windows instances, it is enough to install QEMU Guest Agent. It calls Volume Shadow Copy Service (VSS) to freeze the file system and applications such as Windows SQL Server. Thus, no additional setup is required, since VSS ensures the necessary preparation of compatible applications.

To install QEMU Guest Agent on a guest Windows:

  1. Download the image with the qemu-guest-agent package and drivers.

  2. Mount the downloaded ISO image within a VM.

  3. Install the Virtioserial driver.

    ../../_images/virtioserial.png

  4. Install qemu-guest-agent.

    ../../_images/qemu-guest-agent.png

  5. Reboot the operating system.

Linux preparation#

To ensure file system consistency during backup, install QEMU Guest Agent on an instance. To ensure application data consistency, also download hooks for the corresponding applications. The choice of hooks depends on the installed applications.

Hooks are executable utilities/scripts. Actions performed by the hook depend on the application whose data consistency needs to be ensured. By default, K2 Cloud images do not have pre-installed hooks.

Note

Hook execution results are not known to the cloud, so data consistency is maintained at the file system level only (for details, see the Backup consistency). The consistency status is displayed in the resource table in the Recovery points section.

Note

This instruction assumes the use of Ubuntu 20.04 or CentOS 8.2.

QEMU Guest Agent installation#

  1. To install QEMU Guest Agent on a guest OS, run the following commands:

    apt install qemu-guest-agent
systemctl enable qemu-guest-agent

    yum install qemu-guest-agent
systemctl enable qemu-guest-agent --now

  2. To freeze applications, the unit file of the service must be configured to start a script, which in turn starts the necessary hooks:

    [Service]
ExecStart=-/usr/sbin/qemu-ga -F/etc/qemu/fsfreeze-hook

    [Service]
ExecStart=-/usr/bin/qemu-ga -F/etc/qemu-ga/fsfreeze-hook

  3. Restart the service:

    systemctl daemon-reload
systemctl restart qemu-guest-agent

Hook preparation#

The required executable files of the hooks should be stored in the following directories:

/etc/qemu/fsfreeze-hook.d/

/etc/qemu-ga/fsfreeze-hook.d/

Let’s take a hook to provide consistency for MariaDB in CentOS as an example.

Hook file: /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh#
#!/bin/sh

case "$1" in
 freeze)
echo "Stopping MariaDB for consistent snapshot"
systemctl stop mariadb
echo "MariaDB service status: $(systemctl is-active mariadb)"
;;

 thaw)
echo "Starting MariaDB after snapshot"
systemctl start mariadb
echo "MariaDB service status: $(systemctl is-active mariadb)"
     ;;

 *)
     exit 1
     ;;
esac

Depending on the passed argument (freeze or thaw), one of the two code blocks will be executed when the hook is called. You can use different logic for each of them and make it display the messages you need (for details, see Agent and hook operation diagnostics).

Make the hook file executable:

chmod u+x /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh

Freeze/thaw process#

Before making a volume snapshot, the backup service sends the freeze command to QEMU Guest Agent to stop I/O. Having received the command, the agent calls the script specified in the settings (see step 2 of QEMU Guest Agent installation), which in turn starts hooks from fsfreeze-hook.d directory. The freeze value is passed to each hook as the first argument.

Note

Hook execution results in no way impact file system freezing. The master script simply writes the hook exit code status to the log (see the next section).

Hooks are executed first, then fsfreeze command runs, which saves the file system cache to a volume and freezes I/O. Thawing uses the reversed sequence: the file system is thawed first, and then the hooks (with the thaw argument) are started.

Agent and hook operation diagnostics#

Note

The cloud does not check whether hooks operate correctly. To make sure they do, we recommend adding debugging messages to scripts. This will help you better understand the situation using log entries of errors and messages.

The agent saves its messages to the system log. For example, if you use CentOS, you can find them in /var/log/messaging:

Nov 23 13:38:05 ip-10-200-78-63 qemu-ga[10106]: info: guest-ping called
Nov 23 13:38:05 ip-10-200-78-63 qemu-ga[10106]: info: guest-fsfreeze called
Nov 23 13:38:05 ip-10-200-78-63 qemu-ga[10106]: info: executing fsfreeze hook with arg 'freeze'
Nov 23 13:38:07 ip-10-200-78-63 qemu-ga[10106]: info: executing fsfreeze hook with arg 'thaw'

In CentOS, user hook messages and execution status are saved to /var/log/qga-fsfreeze-hook.log:

Thu Nov 23 13:38:05 MSK 2023: execute /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh freeze
Stopping MariaDB for consistent snapshot
MariaDB service status: inactive
Thu Nov 23 13:38:06 MSK 2023: /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh finished with status=0
Thu Nov 23 13:38:07 MSK 2023: execute /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh thaw
Starting MariaDB after snapshot
MariaDB service status: active
Thu Nov 23 13:38:07 MSK 2023: /etc/qemu-ga/fsfreeze-hook.d/example-hook.sh finished with status=0

Log entries show which scripts were executed, the messages displayed, and their exit code status.

Note

See documentation of your distribution package how to properly configure and diagnose the agent, as the process varies for each package and even version.

Setting up alerts on backup failure#

Alert on failure of any backup job in the project#

If you want to be informed of any backup failures in your project, then set up an alarm for the NumberOfBackupJobsFailed project metric.

  1. Go to the section Monitoring Alarms.

  2. Click Create.

  3. In the window that opens, set the metrics.

    • In the Namespace field, select AWS/Backup.

    • In the Resource type field, leave the default value (Project "<your-project-name>").

    • In the metrics field, select NumberOfBackupJobsFailed.

    Click Next to proceed to the next step.

  4. Create an alarm:

    • In the Name field, set an arbitrary alarm name.

    • (Optional) In the Description field, enter an arbitrary description of the alarm.

    • In the Statistics field, select Max.

    • Leave the Metric fields unchanged (NumberOfBackupJobsFailed, >, 0).

    • In the Metric check field set, specify the period between checks. If you want to receive notifications on a backup failure promptly, then leave the default values (1, 10). The minimum period between checks is 10 minutes. If you need notifications of the failures for the last 24 hours only, then change the last value to 1440. The value in the first field (the number of periods) should remain unchanged.

    Click Set actions to proceed to the next step.

  5. Configure email notifications:

    • Click to send an email notification.

    • In the Alarm state field, select Alarm.

    • Leave the Action field unchanged (Send notification).

    • In the Email field, enter the email address, to which a notification is to be sent.

  6. Click Create to complete the alarm creation.

Alert in case of a backup job failure for a particular resource#

Note

The example describes alarm setup for the instance metric. The alarm for the volume metric is set up similarly.

If you want to receive notifications in case of a backup job error for a particular instance, then set up an alarm for the NumberOfBackupJobsFailed instance metric.

  1. Go to the section Monitoring Alarms.

  2. Click Create.

  3. In the window that opens, set the metrics.

    • In the Namespace field, select AWS/Backup

    • In the Resource type field, select Instance.

    • In the Resource field, selet the instance by its ID.

    • In the metrics field, select NumberOfBackupJobsFailed.

    Click Next to proceed to the next step.

  4. Create an alarm:

    • In the Name field, set an arbitrary alarm name.

    • (Optional) In the Description field, enter an arbitrary description of the alarm.

    • In the Statistics field, select Max.

    • Leave the Metric fields unchanged (NumberOfBackupJobsFailed, >, 0).

    • In the Metric check field set, specify the period between checks. If you want to receive notifications on a backup failure promptly, then leave the default values (1, 10). The minimum period between checks is 10 minutes. If you need notifications of the failures for the last 24 hours only, then change the last value to 1440. The value in the first field (the number of periods) should remain unchanged.

    Click Set actions to proceed to the next step.

  5. Configure email notifications:

    • Click to send an email notification.

    • In the Alarm state field, select Alarm.

    • Leave the Action field unchanged (Send notification).

    • In the Email field, enter the email address, to which a notification is to be sent.

  6. Click Create to complete the alarm creation.