KitsNet Operations:Services:Backup

Stuff about backups

1 References

Veeam Agent for Linux User Guide

Veeam Agent for Linux 6.0 Release Notes

2 Updates

2.1 2/16/2023

Veeam version 12 was released 2/14 and Veeam Agent for Linux v6 along with it. This version was supposed to address the snapshot kernel mod problem and allow us all to advance to the newer version of RHEL compatible kernels. Unfortunately, once the auto-upgrade was applied, backups failed. LSS, the Veeam backup was refusing to load the snapshot module at the start of the backup when executing the python script /usr/sbin/veeamsnap-loader . It turns out that this scrpt is very picky about the data it pulls up from /etc/os-release

NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"

The problem python code is in two places:

def dist_os_release():
    d = {}
    with open("/etc/os-release") as file:
        for line in file:
            stripped = line.rstrip()
            if stripped:
                key, val = stripped.split("=")
                d[key] = val
    return d["ID"].strip("\""), d["VERSION_ID"].strip("\"")

and one of the sections that uses this information in the main module:

        if not distName in ["oracle", "ol", "redhat", "rhel", "centos"]:
            raise RuntimeError("Found unsupported distribution [{0}]".format(distName) )

One way to hack around this is to add to the list on the distName check to allow "rocky". However, you'd have to add "alma" and others if you wanted to go down this path. Arguably a better option would be to use the ID_LIKE value, which identifies the release family, instead. That way, no explicit coding for specific distros in this family need be maintained. That said, whether this is a bug or a feature depends on your point of view (and which side of the vendor relationship you sit).

So I cam up with a patch that updates the logic to work off of ID_KEY. The patch file that generated the final file is on wort int eh KitsNet repo, along with the resulting file with the name veeamsnap-loader.distro_family To deploy across KitsNet, use the following sequence:

cd /usr/sbin
ls -al veeamsnap*
mv veeamsnap-loader veeamsnap-loader.AS_SHIPPED
wget http://wort/KitsNet/veeamsnap-loader.distro_family -O  veeamsnap-loader
chmod u+x veeamsnap-loader

2.2 7/21/2023

Veeam pushed a code upgrade on RHEL v8 variant systems which caused backups to fail on loading the snap module (again!) From the dnf logs:

/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: kmod-veeamsnap-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:02-0400 SUBDEBUG Upgrade: veeam-6.0.3.1221-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:03-0400 SUBDEBUG Upgraded: veeam-6.0.2.1168-1.el8.x86_64
/var/log/dnf.rpm.log:2023-07-21T06:16:09-0400 SUBDEBUG Upgraded: kmod-veeamsnap-6.0.2.1168-1.el8.x86_64

Backup job logs would include an error report like: [21.07.2023 10:06:42.522] <140035981797120> prtcl    | ERR |Failed to load module [veeamsnap].

The solution is to reapply the patch to veeamsnap-loader. I have done this against ballantine, campari, camus, chivas, cristal, fireball, hendrick, okeefe, quadsec, ripple, and zoco.

2.3 11/24/2023

Update of the kernel with the upgrade to Rocky Linux version 8.9 triggered failures of Veeam. Since Veeam has a 90 day SLA on new kernel versions (!!!), there is no choice but to downgrade the default boot version in grub. For this, we turn to the grubby utility again with this command sequence:

 sudo grubby --info=ALL | grep ^kernel
 sudo grubby --grub2 --default-title
 sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
 sudo grubby --grub2 --default-title

Which ends up looking like this when run:

[psmode@fireball ~]$ sudo grubby --info=ALL | grep ^kernel
kernel="/boot/vmlinuz-4.18.0-513.5.1.el8_9.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
kernel="/boot/vmlinuz-4.18.0-477.21.1.el8_8.x86_64"
kernel="/boot/vmlinuz-0-rescue-2d6c7acf2eb8483587e389b0d9726286"
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-513.5.1.el8_9.x86_64) 8.9 (Green Obsidian)
[psmode@fireball ~]$ sudo grubby --set-default "/boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64"
The default is /boot/loader/entries/2d6c7acf2eb8483587e389b0d9726286-4.18.0-477.27.1.el8_8.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-477.27.1.el8_8.x86_64
[psmode@fireball ~]$ sudo grubby --grub2 --default-title
Rocky Linux (4.18.0-477.27.1.el8_8.x86_64) 8.8 (Green Obsidian)

As of this date, this correction was run and validated on Campari, Martell and Hendrick