KitsNet Operations:Services:Backup

Stuff about backups

1 References

Veeam Agent for Linux User Guide

Veeam Agent for Linux 6.0 Release Notes

2 Recover Backup Volume from USB Backup

Backups from all systems are ultimately written to the /backup mountpoint on the //bkup server (currently KitsNet Operations:Builds:Linux:hennessy). If the active mountpoint becomes damaged or lost, it will be necessary to restore it from a backup written to the KitsNet-pbd2 encrypted external drive. Doing this from the context of the KVM host will have the lowest overhead and generally be the simplest to execute.. The procedure is a variation of the monthly /root/backup_guestlv-V3buv054_backups job. Execute as root the following:

1. Mount the USB drive that will be the restore source: /root/vcmount-KitsNet /dev/sdx
2. Remount the restore source read-only: mount -o remount,ro /mnt/KitsNet-pbd2
3. Create guest-level device map on host from virtual disk LV: /sbin/kpartx -avs /dev/T3b/uv054-disk2
4. Add the guest-level device map to those that LVM will reference: /sbin/lvmdevices --adddev /dev/mapper/T3b-uv054--disk2p1
5. Activate guest-level volume group exposed to host: /sbin/vgchange --activate y --activationmode=complete --verbose V3buv054
6. Mount guest-level LV that will be the restore target: mount /dev/V3buv054/backups /mnt/nas_backup
7. Use rsync to restore the volume from the USB backup. Note: Trailing slash on the source specification prevents creating that directory level again at the destination.

   root@wort:~[root@wort ~]# rsync -aAXh --stats /mnt/KitsNet-pbd2/nas_backup/ /mnt/nas_backup
   
   Number of files: 651,695 (reg: 636,184, dir: 15,510)
   Number of created files: 651,695 (reg: 636,184, dir: 15,510)
   Number of deleted files: 0
   Number of regular files transferred: 636,184
   Total file size: 6.64T bytes
   Total transferred file size: 6.64T bytes
   Literal data: 6.64T bytes
   Matched data: 0 bytes
   File list size: 217.88M
   File list generation time: 0.001 seconds
   File list transfer time: 0.000 seconds
   Total bytes sent: 6.64T
   Total bytes received: 15.48M
   
   sent 6.64T bytes  received 15.48M bytes  131.04M bytes/sec
   total size is 6.64T  speedup is 1.00
   

8. Dismount the restore target: umount /mnt/nas_backup
9. De-activate guest-level volume group exposed to host: /sbin/vgchange --activate n --verbose V3buv054
10. Delete the guest-level device map from those that LVM will reference: /sbin/lvmdevices --deldev /dev/mapper/T3b-uv054--disk2p1
11. Remove guest-level device map on host from virtual disk LV: /sbin/kpartx -dv /dev/T3b/uv054-disk2
12. Dismount the USB drive: /root/vcumount-KitsNet 2

3 Updates

3.1 2/16/2023

Veeam version 12 was released 2/14 and Veeam Agent for Linux v6 along with it. This version was supposed to address the snapshot kernel mod problem and allow us all to advance to the newer version of RHEL compatible kernels. Unfortunately, once the auto-upgrade was applied, backups failed. LSS, the Veeam backup was refusing to load the snapshot module at the start of the backup when executing the python script /usr/sbin/veeamsnap-loader . It turns out that this scrpt is very picky about the data it pulls up from /etc/os-release

NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

The problem python code is in two places:

def dist_os_release():
    d = {}
    with open("/etc/os-release") as file:
        for line in file:
            stripped = line.rstrip()
            if stripped:
                key, val = stripped.split("=")
                d[key] = val
    return d["ID"].strip("\""), d["VERSION_ID"].strip("\"")

and one of the sections that uses this information in the main module:

        if not distName in ["oracle", "ol", "redhat", "rhel", "centos"]:
            raise RuntimeError("Found unsupported distribution [{0}]".format(distName) )

One way to hack around this is to add to the list on the distName check to allow "rocky". However, you'd have to add "alma" and others if you wanted to go down this path. Arguably a better option would be to use the ID_LIKE value, which identifies the release family, instead. That way, no explicit coding for specific distros in this family need be maintained. That said, whether this is a bug or a feature depends on your point of view (and which side of the vendor relationship you sit).

So I cam up with a patch that updates the logic to work off of ID_KEY. The patch file that generated the final file is on wort int eh KitsNet repo, along with the resulting file with the name veeamsnap-loader.distro_family To deploy across KitsNet, use the following sequence:

cd /usr/sbin
ls -al veeamsnap*
mv veeamsnap-loader veeamsnap-loader.AS_SHIPPED
wget http://wort/KitsNet/veeamsnap-loader.distro_family -O  veeamsnap-loader
chmod u+x veeamsnap-loader

3.2 7/21/2023

Veeam pushed a code upgrade on RHEL v8 variant systems which caused backups to fail on loading the snap module (again!) From the dnf logs:

/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: kmod-veeamsnap-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: veeam-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:03-0400 SUBDEBUG Upgraded: veeam-6.0.2.1168-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:09-0400 SUBDEBUG Upgraded: kmod-veeamsnap-6.0.2.1168-1.el8.x86_64

Backup job logs would include an error report like: [21.07.2023 10:06:42.522] <140035981797120> prtcl    | ERR |Failed to load module [veeamsnap].

The solution is to reapply the patch to veeamsnap-loader. I have done this against ballantine, campari, camus, chivas, cristal, fireball, hendrick, okeefe, quadsec, ripple, and zoco.

3.3 11/24/2023

Update of the kernel with the upgrade to Rocky Linux version 8.9 triggered failures of Veeam. Since Veeam has a 90 day SLA on new kernel versions (!!!), there is no choice but to downgrade the default boot version in grub. For this, we turn to the grubby utility again with this command sequence:

 sudo grubby --info=ALL | grep ^kernel
 sudo grubby --grub2 --default-title
 sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
 sudo grubby --grub2 --default-title

Which ends up looking like this when run:

[psmode@fireball ~]$ sudo grubby --info=ALL | grep ^kernel
kernel="/boot/vmlinuz-4.18.0-513.5.1.el8_9.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.21.1.el8_8.x86_64"
kernel="/boot/vmlinuz-0-rescue-2d6c7acf2eb8483587e389b0d9726286"
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-513.5.1.el8_9.x86_64) 8.9 (Green Obsidian)
[psmode@fireball ~]$ sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
The default is /boot/loader/entries/2d6c7acf2eb8483587e389b0d9726286-4.18.0-477.27.1.el8_8.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-477.27.1.el8_8.x86_64) 8.8 (Green Obsidian)

As of this date, this correction was run and validated on Campari, Martell and Hendrick

3.4 12/6/2023

Veeam came out with support for Linux Kernels derived from RedHat 8.9 and there was a patch along the way to Rocky Linux taking the kernel version up another notch. When Veeam updated the client, they made further updates to veeamsnap-loader, meaning that I had to create a new patch file and updated script, all based on the same changes as before. Updates to the Veeam client also require interactively accepting the license again. I tested on camus to get all this right. To repeat for all systems:

1. Check for and remove any exclusions on kernel updates (exclude=kernel*) from /etc/yum.conf
2. Set default kernel with grubby to 4.18.0-513.9.1
3. Copy in updated veeamsnap-loader
4. Reboot
5. Run Veeam and accept license and set version to Server
6. Verify with successful backup run.

Run with these commands:

grep exclude /etc/yum.conf
cd /usr/sbin
ls -al veeamsnap*
sudo mv veeamsnap-loader veeamsnap-loader.AS_SHIPPED2
sudo wget http://wort/KitsNet/veeamsnap-loader.distro_family2 -O  veeamsnap-loader
sudo chmod u+x veeamsnap-loader
sudo grubby --set-default 0

Executed updates on chivas, cristal, fireball, ripple, campari, quadsec, ballantine (OS won't update because of /usr/share/mysql/charsets conflicts - need to fix elsewhere), okeefe, zoco,

Screech and corzo required acknowledgement of the license, but no code changes.

3.5 5/20/2024

Veeam is screwing up with Rocky Linux 9.4 kernel 5.14.0-427.16.1.el9_4.x86_64 on corzo and alphakronik, but not on ciroc! Once again, it is the snapshots that are failing in the job. To address this, I downgraded the kernel version we are booting on corzo and alphakronik. This is all I have time to do for now.

3.6 5/21/2024

Looking over alphakronik and comparing with ciroc, there is a difference in the blksnap modules on the two systems. While dnf reports both to the same version, find gives different results. For ciroc (where is works):

[root@ciroc ~]# find / -name "*blksnap*"
/dev/veeamblksnap
/sys/kernel/debug/printk/index/veeamblksnap
/sys/class/veeamblksnap
/sys/class/veeamblksnap/veeamblksnap
/sys/devices/virtual/veeamblksnap
/sys/devices/virtual/veeamblksnap/veeamblksnap
/sys/module/veeamblksnap
/sys/module/bdevfilter/holders/veeamblksnap
/var/log/veeam/blksnap.log
/var/lib/dkms/blksnap
/var/lib/dkms/blksnap/6.1.0.1498/5.14.0-362.24.1.el9_3.x86_64/x86_64/module/veeamblksnap.ko
/var/lib/dkms/blksnap/6.1.0.1498/5.14.0-362.24.1.el9_3.0.1.x86_64/x86_64/module/veeamblksnap.ko.xz
/var/lib/dkms/blksnap/6.1.0.1498/5.14.0-427.16.1.el9_4.x86_64/x86_64/module/veeamblksnap.ko.xz
/usr/lib/modules/5.14.0-362.24.1.el9_3.x86_64/extra/veeamblksnap.ko
/usr/lib/modules/5.14.0-362.24.1.el9_3.0.1.x86_64/weak-updates/veeamblksnap.ko
/usr/lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/veeamblksnap.ko.xz
/usr/src/blksnap-6.1.0.1498
[root@ciroc ~]#  dnf list "*blksnap*"
Last metadata expiration check: 1:51:29 ago on Tue 21 May 2024 06:06:22 AM EDT.
Installed Packages
blksnap.noarch                                                                                           6.1.0.1498-1                                                                                       @veeam
Available Packages
blksnap-ueficert.noarch                                                                                  6.1.0.1498-1                                                                                       veeam
kmod-blksnap.x86_64                                                                                      6.1.0.1498-1.el9                                                                                   veeam
[root@ciroc ~]# uname -a
Linux ciroc.lan.kitsnet.us 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 8 17:48:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
[root@ciroc blksnap]#
[root@ciroc blksnap]# ls -al
total 0
drwxr-xr-x. 3 root root 124 May 11 07:02 .
drwxr-xr-x. 3 root root  51 Mar 29 14:04 ..
drwxr-xr-x. 5 root root 132 May 11 07:02 6.1.0.1498
lrwxrwxrwx. 1 root root  46 Mar 29 14:04 kernel-5.14.0-362.24.1.el9_3.x86_64-x86_64 -> 6.1.0.1498/5.14.0-362.24.1.el9_3.x86_64/x86_64
lrwxrwxrwx. 1 root root  46 May 11 07:02 kernel-5.14.0-427.16.1.el9_4.x86_64-x86_64 -> 6.1.0.1498/5.14.0-427.16.1.el9_4.x86_64/x86_64

For alphakronik (where it fails):

[psmode@alphakronik ~]$ uname -a
Linux alphakronik.lan.kitsnet.us 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 8 17:48:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
[psmode@alphakronik ~]$ sudo veeam
[psmode@alphakronik ~]$ sudo find / -name "*blksnap*"
/usr/lib/modules/5.14.0-284.18.1.el9_2.x86_64/extra/blksnap.ko.xz
/usr/lib/modules/5.14.0-362.8.1.el9_3.x86_64/extra/blksnap.ko.xz
/usr/lib/modules/5.14.0-362.8.1.el9_3.x86_64/extra/veeamblksnap.ko.xz
/usr/lib/modules/5.14.0-362.24.1.el9_3.x86_64/extra/veeamblksnap.ko.xz
/usr/lib/modules/5.14.0-362.24.1.el9_3.0.1.x86_64/weak-updates/veeamblksnap.ko.xz
/usr/src/blksnap-6.0.2.1168
/usr/src/blksnap-6.0.3.1221
/usr/src/blksnap-6.1.0.1498
/var/log/veeam/blksnap.log
/var/lib/dkms/blksnap
/var/lib/dkms/blksnap/6.0.3.1221/5.14.0-284.18.1.el9_2.x86_64/x86_64/module/blksnap.ko.xz
/var/lib/dkms/blksnap/6.1.0.1498/5.14.0-362.24.1.el9_3.x86_64/x86_64/module/veeamblksnap.ko.xz
[psmode@alphakronik ~]$ sudo dnf list "*blksnap*"
Last metadata expiration check: 0:46:06 ago on Tue 21 May 2024 07:20:13 AM EDT.
Installed Packages
blksnap.noarch                                                                6.1.0.1498-1                                                           @veeam
Available Packages
blksnap-ueficert.noarch                                                       6.1.0.1498-1                                                           veeam
kmod-blksnap.x86_64                                                           6.1.0.1498-1.el9                                                       veeam
[psmode@alphakronik ~]$  cd  /var/lib/dkms/blksnap
[psmode@alphakronik blksnap]$ sudo ls -al
total 0
drwxr-xr-x. 3 root root 74 May 21 08:06 .
drwxr-xr-x. 3 root root 51 Mar  6 12:01 ..
drwxr-xr-x. 3 root root 56 Dec  6 06:46 6.0.3.1221
lrwxrwxrwx. 1 root root 46 Jul 21  2023 kernel-5.14.0-284.18.1.el9_2.x86_64-x86_64 -> 6.0.3.1221/5.14.0-284.18.1.el9_2.x86_64/x86_64

Reinstalling did not help:

[psmode@alphakronik ~]$ sudo dnf reinstall blksnap
Last metadata expiration check: 0:46:20 ago on Tue 21 May 2024 07:20:13 AM EDT.
Dependencies resolved.
===========================================================================================================================================================
 Package                             Architecture                       Version                                    Repository                         Size
===========================================================================================================================================================
Reinstalling:
 blksnap                             noarch                             6.1.0.1498-1                               veeam                              58 k

Transaction Summary
===========================================================================================================================================================

Total download size: 58 k
Installed size: 231 k
Is this ok [y/N]: y
Downloading Packages:
blksnap-6.1.0.1498-1.noarch.rpm                                                                                            208 kB/s |  58 kB     00:00
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                      207 kB/s |  58 kB     00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                   1/1
  Reinstalling     : blksnap-6.1.0.1498-1.noarch                                                                                                       1/2
  Running scriptlet: blksnap-6.1.0.1498-1.noarch                                                                                                       1/2
Removing old blksnap-6.1.0.1498 DKMS files...
Module blksnap-6.1.0.1498 for kernel 5.14.0-362.24.1.el9_3.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

veeamblksnap.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/5.14.0-362.24.1.el9_3.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

bdevfilter.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/5.14.0-362.24.1.el9_3.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod....
Deleting module blksnap-6.1.0.1498 completely from the DKMS tree.
Loading new blksnap-6.1.0.1498 DKMS files...
Building for 5.14.0-427.16.1.el9_4.x86_64
Building initial module for 5.14.0-427.16.1.el9_4.x86_64
Done.

veeamblksnap.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/

bdevfilter.ko.xz:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/
depmod....

  Running scriptlet: blksnap-6.1.0.1498-1.noarch                                                                                                       2/2
Module blksnap-6.1.0.1498 for kernel 5.14.0-427.16.1.el9_4.x86_64 (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

veeamblksnap.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

bdevfilter.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.
depmod....
Deleting module blksnap-6.1.0.1498 completely from the DKMS tree.

  Cleanup          : blksnap-6.1.0.1498-1.noarch                                                                                                       2/2
  Running scriptlet: blksnap-6.1.0.1498-1.noarch                                                                                                       2/2
  Verifying        : blksnap-6.1.0.1498-1.noarch                                                                                                       1/2
  Verifying        : blksnap-6.1.0.1498-1.noarch                                                                                                       2/2

Reinstalled:
  blksnap-6.1.0.1498-1.noarch

Complete!
[psmode@alphakronik ~]$ sudo find / -name "*blksnap*"
/usr/lib/modules/5.14.0-284.18.1.el9_2.x86_64/extra/blksnap.ko.xz
/usr/lib/modules/5.14.0-362.8.1.el9_3.x86_64/extra/blksnap.ko.xz
/usr/lib/modules/5.14.0-362.8.1.el9_3.x86_64/extra/veeamblksnap.ko.xz
/usr/lib/modules/5.14.0-362.24.1.el9_3.0.1.x86_64/weak-updates/veeamblksnap.ko.xz
/usr/src/blksnap-6.0.2.1168
/usr/src/blksnap-6.0.3.1221
/usr/src/blksnap-6.1.0.1498
/var/log/veeam/blksnap.log
/var/lib/dkms/blksnap
/var/lib/dkms/blksnap/6.0.3.1221/5.14.0-284.18.1.el9_2.x86_64/x86_64/module/blksnap.ko.xz

Solution is to uninstall blksnap (which removes KNveeam and veeam), then reinstall blksnap by itself before reinstalling KNveeam and veeam. This will result in the loss of the backup job definition and schedule, so capturing these before uninstall is a good idea.

I fixed this on alphakronik and corzo.