KitsNet Operations:Services:Backup

Stuff about backups

1 References

Veeam Agent for Linux User Guide

Veeam Agent for Linux 6.0 Release Notes

2 Updates

2.1 2/16/2023

Veeam version 12 was released 2/14 and Veeam Agent for Linux v6 along with it. This version was supposed to address the snapshot kernel mod problem and allow us all to advance to the newer version of RHEL compatible kernels. Unfortunately, once the auto-upgrade was applied, backups failed. LSS, the Veeam backup was refusing to load the snapshot module at the start of the backup when executing the python script /usr/sbin/veeamsnap-loader . It turns out that this scrpt is very picky about the data it pulls up from /etc/os-release

NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

The problem python code is in two places:

def dist_os_release():
    d = {}
    with open("/etc/os-release") as file:
        for line in file:
            stripped = line.rstrip()
            if stripped:
                key, val = stripped.split("=")
                d[key] = val
    return d["ID"].strip("\""), d["VERSION_ID"].strip("\"")

and one of the sections that uses this information in the main module:

        if not distName in ["oracle", "ol", "redhat", "rhel", "centos"]:
            raise RuntimeError("Found unsupported distribution [{0}]".format(distName) )

One way to hack around this is to add to the list on the distName check to allow "rocky". However, you'd have to add "alma" and others if you wanted to go down this path. Arguably a better option would be to use the ID_LIKE value, which identifies the release family, instead. That way, no explicit coding for specific distros in this family need be maintained. That said, whether this is a bug or a feature depends on your point of view (and which side of the vendor relationship you sit).

So I cam up with a patch that updates the logic to work off of ID_KEY. The patch file that generated the final file is on wort int eh KitsNet repo, along with the resulting file with the name veeamsnap-loader.distro_family To deploy across KitsNet, use the following sequence:

cd /usr/sbin
ls -al veeamsnap*
mv veeamsnap-loader veeamsnap-loader.AS_SHIPPED
wget http://wort/KitsNet/veeamsnap-loader.distro_family -O  veeamsnap-loader
chmod u+x veeamsnap-loader

2.2 7/21/2023

Veeam pushed a code upgrade on RHEL v8 variant systems which caused backups to fail on loading the snap module (again!) From the dnf logs:

/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: kmod-veeamsnap-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: veeam-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:03-0400 SUBDEBUG Upgraded: veeam-6.0.2.1168-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:09-0400 SUBDEBUG Upgraded: kmod-veeamsnap-6.0.2.1168-1.el8.x86_64

Backup job logs would include an error report like: [21.07.2023 10:06:42.522] <140035981797120> prtcl    | ERR |Failed to load module [veeamsnap].

The solution is to reapply the patch to veeamsnap-loader. I have done this against ballantine, campari, camus, chivas, cristal, fireball, hendrick, okeefe, quadsec, ripple, and zoco.

2.3 11/24/2023

Update of the kernel with the upgrade to Rocky Linux version 8.9 triggered failures of Veeam. Since Veeam has a 90 day SLA on new kernel versions (!!!), there is no choice but to downgrade the default boot version in grub. For this, we turn to the grubby utility again with this command sequence:

 sudo grubby --info=ALL | grep ^kernel
 sudo grubby --grub2 --default-title
 sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
 sudo grubby --grub2 --default-title

Which ends up looking like this when run:

[psmode@fireball ~]$ sudo grubby --info=ALL | grep ^kernel
kernel="/boot/vmlinuz-4.18.0-513.5.1.el8_9.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.21.1.el8_8.x86_64"
kernel="/boot/vmlinuz-0-rescue-2d6c7acf2eb8483587e389b0d9726286"
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-513.5.1.el8_9.x86_64) 8.9 (Green Obsidian)
[psmode@fireball ~]$ sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
The default is /boot/loader/entries/2d6c7acf2eb8483587e389b0d9726286-4.18.0-477.27.1.el8_8.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-477.27.1.el8_8.x86_64) 8.8 (Green Obsidian)

As of this date, this correction was run and validated on Campari, Martell and Hendrick

2.4 12/6/2023

Veeam came out with support for Linux Kernels derived from RedHat 8.9 and there was a patch along the way to Rocky Linux taking the kernel version up another notch. When Veeam updated the client, they made further updates to veeamsnap-loader, meaning that I had to create a new patch file and updated script, all based on the same changes as before. Updates to the Veeam client also require interactively accepting the license again. I tested on camus to get all this right. To repeat for all systems:

1. Check for and remove any exclusions on kernel updates (exclude=kernel*) from /etc/yum.conf
2. Set default kernel with grubby to 4.18.0-513.9.1
3. Copy in updated veeamsnap-loader
4. Reboot
5. Run Veeam and accept license and set version to Server
6. Verify with successful backup run.

Run with these commands:

grep exclude /etc/yum.conf
cd /usr/sbin
ls -al veeamsnap*
sudo mv veeamsnap-loader veeamsnap-loader.AS_SHIPPED2
sudo wget http://wort/KitsNet/veeamsnap-loader.distro_family2 -O  veeamsnap-loader
sudo chmod u+x veeamsnap-loader
sudo grubby --set-default "/boot/vmlinuz-4.18.0-513.9.1.el8_9.x86_64"