View source for KVM:host:Wort Operations:2021 (CentOS 8)

Operational history of wort during its CentOS 8 incarnation

==Updates==
===12/31/2021===
====Start work to enable /tmp on a tmpfs filesystem====
I started the work to enable /tmp on a tmpfs filesystem and make it ephemeral. See https://unix.stackexchange.com/questions/352042/systemd-backed-tmpfs-how-to-specify-tmp-size-manually for details. As root, executed <code>systemctl edit tmp.mount</code> and entered:<syntaxhighlight lang="text">
[Mount]
Options=mode=1777,strictatime,nosuid,nodev,size=536870912
</syntaxhighlight>This created the supporting directory structure and the file <code>/etc/systemd/system/tmp.mount.d/override.conf</code> with this content. At this point, there is no change to <code>/tmp</code> which already contains files. What happens during reboot will be interesting.
====Allowed NTP through firewall====
<syntaxhighlight lang="shell-session">
[psmode@wort ~]$ sudo firewall-cmd --permanent --add-service=ntp
success
[psmode@wort ~]$ sudo firewall-cmd --reload
success
[psmode@wort ~]$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: bond0 eno1 enp4s0f0 enp5s0f1 enp6s0f0 enp6s0f1 kvm-bkup kvm-guests nm-bond
  sources:
  services: cockpit dhcpv6-client http https ntp ssh
  ports: 3071/tcp 7902-7939/tcp
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
        rule family="ipv4" source address="192.168.15.64/32" port port="10050" protocol="tcp" accept
        rule family="ipv4" source address="192.168.15.54/32" port port="10050" protocol="tcp" accept
</syntaxhighlight>
===01/02/2022===
====Fixed glances====
Fixed glances with code mod as per https://mangolassi.it/topic/19117/how-can-i-show-disk-io-in-glances/7 while the citation calls for <code>sed</code>, I did it manually in <code>vi</code>:<syntaxhighlight lang="shell-session">
# sed -i 's#fields_len == 14:#fields_len == 14 or fields_len == 18:#g' /usr/lib64/python3.6/site-packages/psutil/_pslinux.py
</syntaxhighlight>I also disabled ipv6 on bond0. It appears ipv6 on this interface is hopeless for now.
===1/7/2002===
====Get sensors to work for Zabbix====
*Installed acpid
*Zabbix is picking up errors getting temperatures (wort metric ''Information about the lm-sensors''). Problem is with a temperature reading for the <code>Adapter: Virtual device</code>. (see https://github.com/nojhan/liquidprompt/issues/445) Solution to suppressing the error for the thing we do not care about is to suppress taking the reading. To do this, create a text file named <code>/etc/sensors.d/iwlwifi_1-virtual-0</code> and populate it with:<syntaxhighlight lang="text">
chip "iwlwifi-virtual-*"
    ignore temp1
</syntaxhighlight>
*Still not working. Edited <code>/etc/zabbix/zabbix_agent2.d/userparameter_sensors.conf</code> to call for read of specific chips<syntaxhighlight lang="text">
UserParameter=sensors,sensors -j -A nouveau-pci-0400 it8792-isa-0a60 k10temp-pci-00c3
</syntaxhighlight>
*After restart of the agent, the data going back looks good, but there is not yet creation of any new items or graphs.
====Installing support for Zabbix virbix====
*See (https://share.zabbix.com/virtualization/kvm/virbix and https://github.com/sergiotocalini/virbix)
*Installed ksh (required for deployment)
*did the git clone and updated <code>ZABBIX_INC="${ZABBIX_DIR}/zabbix_agentd.d</code> reference to <code>ZABBIX_INC="${ZABBIX_DIR}/zabbix_agentd2.d"</code>
*Deployed<syntaxhighlight lang="shell-session">
[root@wort src]# sudo ./virbix/deploy_zabbix.sh -u "qemu:///system"
'./virbix/virbix/virbix.sh' -> '/etc/zabbix/scripts/agentd/virbix/virbix.sh'
'./virbix/virbix/scripts' -> '/etc/zabbix/scripts/agentd/virbix/scripts'
'./virbix/virbix/scripts/domain_check.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/domain_check.sh'
'./virbix/virbix/scripts/domain_list.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/domain_list.sh'
'./virbix/virbix/scripts/functions.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/functions.sh'
'./virbix/virbix/scripts/net_check.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/net_check.sh'
'./virbix/virbix/scripts/net_list.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/net_list.sh'
'./virbix/virbix/scripts/pool_check.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/pool_check.sh'
'./virbix/virbix/scripts/pool_list.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/pool_list.sh'
'./virbix/virbix/scripts/report.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/report.sh'
'./virbix/virbix/scripts/report_domains.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/report_domains.sh'
'./virbix/virbix/scripts/report_nets.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/report_nets.sh'
'./virbix/virbix/scripts/report_node.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/report_node.sh'
'./virbix/virbix/scripts/report_pools.sh' -> '/etc/zabbix/scripts/agentd/virbix/scripts/report_pools.sh'
'./virbix/virbix/virbix.conf.example' -> '/etc/zabbix/scripts/agentd/virbix/virbix.conf.new'
'./virbix/virbix/zabbix_agentd.conf' -> '/etc/zabbix/zabbix_agentd2.d/virbix.conf.new'
</syntaxhighlight>
*Test<syntaxhighlight lang="shell-session">
[root@wort src]# /etc/zabbix/scripts/agentd/virbix/virbix.sh -s domain_list -j DOMID:DOMNAME:DOMUUID:DOMTYPE:DOMSTATE
{
   "data":[
      { "{#DOMID}":"63", "{#DOMNAME}":"uv038", "{#DOMUUID}":"02160641-ed2a-4b3a-a3d2-8db8c3d55551", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"71", "{#DOMNAME}":"wv902", "{#DOMUUID}":"0d0ec200-b0da-3fc7-df68-eb17840f0291", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"70", "{#DOMNAME}":"uv037", "{#DOMUUID}":"15b657d2-1e7f-44c1-a9e3-09110ff7320a", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"-", "{#DOMNAME}":"uv019", "{#DOMUUID}":"1899841b-2cbd-a480-d5d8-aa2d9f281ec1", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"shut off" },
      { "{#DOMID}":"67", "{#DOMNAME}":"uv034", "{#DOMUUID}":"2580bf9c-7d91-41f7-b204-b58b1acee4b2", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"66", "{#DOMNAME}":"uv017", "{#DOMUUID}":"266b61f2-bfab-1813-6c00-7dc5c780f928", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"60", "{#DOMNAME}":"uv040", "{#DOMUUID}":"2d6c7acf-2eb8-4835-87e3-89b0d9726286", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"68", "{#DOMNAME}":"uv039", "{#DOMUUID}":"376d4633-2e70-4554-9ab2-32bb26f39a71", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"72", "{#DOMNAME}":"uv033", "{#DOMUUID}":"4acb2cb0-5489-40dc-a662-4a08f73c45c3", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"62", "{#DOMNAME}":"uv032", "{#DOMUUID}":"61e9a7b1-4ba8-4292-b1fe-8b9d897c0b44", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"64", "{#DOMNAME}":"uv029", "{#DOMUUID}":"6aeb3ad2-1a5b-452a-bd38-b982b21256fa", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"73", "{#DOMNAME}":"uv031", "{#DOMUUID}":"6e8cdc83-3284-4866-97b6-deccf2c14f65", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"65", "{#DOMNAME}":"uv023", "{#DOMUUID}":"6faae374-e311-f965-4f57-e6f2ecc22a3f", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"-", "{#DOMNAME}":"uv020", "{#DOMUUID}":"9838fbec-6c80-8c51-809d-3f3079b96124", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"shut off" },
      { "{#DOMID}":"-", "{#DOMNAME}":"uv026", "{#DOMUUID}":"9bf46d15-22d2-eb2c-4d25-7cf6966534ba", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"shut off" },
      { "{#DOMID}":"69", "{#DOMNAME}":"uv025", "{#DOMUUID}":"b7b4795b-9334-ac91-ed77-b3ffcfd13594", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" },
      { "{#DOMID}":"61", "{#DOMNAME}":"uv035", "{#DOMUUID}":"fb62047b-d9a7-4203-af3a-43fb993c21b3", "{#DOMTYPE}":"hvm", "{#DOMSTATE}":"running" }
   ]
}
</syntaxhighlight>
*Found mistaken directory reference Z<code>ABBIX_INC="${ZABBIX_DIR}/zabbix_agentd2.d"</code> instead of <code>ZABBIX_INC="${ZABBIX_DIR}/zabbix_agent2.d"</code>. I moved the config file, removed the directory and then went back to fix the deployment script.
====Installing support for zabbix-kvm-res====
*see https://share.zabbix.com/virtualization/kvm/kvm-monitoring and https://github.com/bushvin/zabbix-kvm-res
====Fixing sudo====
*Fixed <code>/etc/sudoers.d/zabbix</code> file with <code>zabbix  ALL=(ALL) NOPASSWD: /sbin/smartctl, /usr/bin/virsh</code>
===1/10/2022===
====Unable to find AMD Northbridge id====
*the system message log is filling up with these messages and causing the logwatch report to grow so large that i cannot be emailed.
*This appears to be an issue in the kernel with the k10temp code. As per https://gitmemory.cn/repo/groeck/k10temp/activity, i will disable the module with <code>modprobe -r k10temp</code> to make the errors stop. When the module is fixed, it will need to be reenabled.
===9/11/2022===
Trying to setup new 8TB Veracrypt drives turned into near disaster.
====New 8TB kitsnet-pbd Drives====
I inserted both new drives in the USB dock (Western Digital Gold S/N VR0DR3BK and VYG3DZ3R). Partitioned each with parted using a gpt disk label<syntaxhighlight lang="shell-session">
[root@wort ~]# parted /dev/sde print
Model: ASMT USB 3.0 Destop H (scsi)
Disk /dev/sde: 8002GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  8002GB  8002GB
</syntaxhighlight>With the devices partitioned, I started two screen sessions and a script logging session within each for the execution of the truecrypt encryption activity, which was going to take over seven hours.<syntaxhighlight lang="shell-session">
[root@wort ~]# veracrypt -t -c --volume-type=normal /dev/sde1 --encryption=AES-Twofish-Serpent --hash=sha-512 --filesystem=none --non-interactive --pim=0 --verbose -k=truecrypt.keyfile -p=""
</syntaxhighlight>Both drives managed to run up to 100% usage according to diskmon.

In the morning, I was not able to log into wort. It would respond to ping, but bringing up the login prompt was very, very slow. Due to an error in selecting the correct console video output, I was not able to attempt local console login. In the end, I tried resets, but did not realize that the reboot was underway and later did a power cycle to force the system down. From here, things got worse
====Restart Failure====
Boot would fail quite early on. The system would show a grub boot menu with valid choices, but attempting to boot any of the known OSes would fail with a kernel panic and an error message resembling ''end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)'' . None of the available boot options would get anywhere. I theorized that spontaneous changes to the TPM settings in BIOS could cause boot failures. This proved to not be an issue as it appeared that our starting position was with TPM features disabled.

I did locate a Rocky Linux v8.6 installation thumb drive and booted into recovery mode there. However, rescue mode still said that it could not find any Linux partitions. However, upon going down to the shell, all partitions on all permanent drives (even those in the RAID controller) were found and appeared valid. LVM had started and successfully mapped all the LVs. Running xfs_repair against the LVs on T0h was generally successful, though a couple of the LVs had to be mounted to flush pending operations before the xfs_repair was successful. This still did not address the main problem, but possibly would have been needed anyway.

The problem was very odd. Grub was complaining that it could not find Linux partitions, but they were clearly there and appeared fine. Becoming concerned about the grub configuration, I looked for <code>grub.cfg</code> and found two of them. Doing a diff on the two returned:<syntaxhighlight lang="shell-session">
# diff /boot/efi/EFI/rocky/grub.cfg /boot/grub2/grub.cfg
140c140
<   set kernelopts="root=/dev/mapper/T0h-root ro crashkernel=auto resume=/dev/mapper/T0h-swap rd.lvm.lv=T0h/root rd.lvm.lv=T0h/swap rhgb quiet "
---
>   set kernelopts="root=/dev/mapper/T0h-root ro crashkernel=auto resume=/dev/mapper/T0h-swap rd.lvm.lv=T0h/root rd.lvm.lv=T0h/swap rhgb usbcore.autosuspend=-1 acpi_enforce_resources=lax mem_encrypt=on kvm_amd.sev=1 "
</syntaxhighlight>I looked up the meaning of the kernel options that were unique to <code>/boot/grub2/grub.cfg</code> (the one actually used for booting). Most all of the options were really required, but [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-kernel_parameters_changes mem_encrypt=on] did not seem to absolutely required. This setting is part of [https://www.kernel.org/doc/Documentation/x86/amd-memory-encryption.txt Secure Memory Encryption (SME)] . SME and Secure Encrypted Virtualization (SEV) are features found on AMD processors. Doing some googling on this turned up more than a couple articles complaining about stability issues with this. In particular, [https://forums.whonix.org/t/enable-secure-memory-encryption-sme-kernel-parameter-mem-encrypt-by-default/10393 Enable Secure Memory Encryption (SME) - kernel parameter mem_encrypt - by default?] argued that the feature was very buggy and that <code>mem_encrypt</code> should be set to off lest there be booting problems.

Next boot, I edited the kernel options conversationally to omit the <code>mem_encrypt=on</code> clause. '''''With this change, booting was successful.'''''

After this, I updated the <code>GRUB_CMDLINE_LINUX</code> line in <code>/etc/default/grub</code> to change the '''<code>mem_encrypt=on</code>''' clause to '''<code>mem_encrypt=off</code>''' Next was to build a new grub configuration with <code>grub2-mkconfig -o /boot/grub2/grub.cfg</code> This new boot configuration will need to be tested after the completion of the encryption operation against the first drive.
====New 8TB kitsnet-pbd Drives - Take 2====
With the system back in operation, time to restart the drive encryption. For this attempt, only one drive will be done at a time. Restarting the session for <code>/dev/sde1</code> was successful. Perhaps unsurprisingly, throughput may be faster now. Prior attempt saw 195 MB/s sustained writes, while this attempt is running at around 230 MB/s. Zabbix data confirms that IO operations are up from 395 writes/s to 495 writes/s. As of 15:15, truecrypt estimated six hours remaining.
===2/6/2023===
====Improve monthly backup efficiency====
The monthly backup of guest virtual disks has never been fully efficient when it comes to large volumes. The code is setup to evaluate the size of the LV implementing the virtual disk and will run it through a gzip process for really large disks. The idea is that compressing out these disk images, especially for largely empty volumes or volumes with highly compressible data, will cut down on backup time overall since a much smaller number of blocks need be encrypted and written to the single output volume. The problem is that throughput of these operations is constrained by the single threaded throughput of gzip. It turns out there is a multi-threaded implementation available, [http://www.zlib.net/pigz/ pigz]. While the option to build from source is available, I will start off by using the packaged version available from the Anaconda repo that wort already references. While this may be a few versions behind, automated maintenance would be a very good thing. So good in fact, that the utility is already installed!

The default in version 2.4 of pigz is to use eight processors; later versions increased the default to all online processors. This may be overridden with the <code>--processes</code> argument.

The reference to gzip is hardcoded in <code>/usr/local/sbin/make_guest_bck_script.awk</code>, which is distributed as part of the KNkvmHostBackup package on KitsNet. This awk script is called in <code>/usr/local/sbin/KVMbackup</code> to dynamically generate the script which will then be used to snap and backup all the LVs implementing virtual disks for all the guests on the host. I am prototyping an update to the code so that the awk script will conditionally replace the default gzip command with the command passed to it via a variable named <code>ZipCmd</code>. The calling shell script, <code>KVMbackup</code>, will look for the pigz command on the path and will define the <code>ZipCmd</code> variable to the awk script with the multithreaded pigz command. By default, the variable will be am empty string, which will cause the awk script to use the single-threaded <code>gzip</code> command by default.

In the end, an uncontested test run completed in 3.5 hours instead of the normal 6 hours. Systemwide idle times would range from 70% during execution of the the regular <code>dd</code> command to as low as 20% when <code>pigz</code> was running up to over 800% CPU utilization. Over the execution of the entire job, CPU Nice time (as recorded by Zabbix) ranged from 15% to 56%.

With the prototype in place, I adjusted the start time of the monthly KVMbackup from from 5:45 to 4:45. I will evaluate system behavior on the next monthly run to see if reduction in the overall elpased time of the monthly backup can be achieved by moving this start time up further.
===6/11/2023===
====Planned downtime for 128 GB RAM upgrade and top up of radiator.====
In preparation for RAM expansion, I expanded the <code>T0h/kvm_save</code> volume from 64 GB to 128 GB.

In order to eliminate co-incidental issues complicating the restart of wort after the memory upgrade, i performed a simple reboot of wort from the physical console. I discovered the wireless keyboard batteries were dead and replaced these. Switching the left monitor to input HDMI1 allowed me to see the console display. For additional boot prep, I ensured that the USB backup drives were dismounted and the dock powered down. I also shutdown the es40 emulator within alphakronik, since it was spinning the CPUs assigned to it. I also noticed that the clock on the instance was way off showing 5/31 (probably not worth running this). All other instances were left running, including nested simulators.

Wort came back up relatively smoothly. However, DONQ appears to have crashed. It appeared that the remote console was hosed as well, so I restarted it within ConsoleWorks, though this may not have actually been necessary. The initial phases of simulator startup were very, very slow, but gained speed once the simulator instance advanced to the VMS startup sequence. All other instances (including porter running x86 VMS) came back nicely, excepting for time. Many of the guests had their time out of sync by as much as five minutes.

I did discover that wort was configured to use its own time as a source in <code>/etc/chronyd.conf</code> which lead it to mistakenly promote itself to stratum 0. For now, I dropped the time sevrer (CNAME for wort) as a source and replaced it with courvoisier.

There are settings defined in <code>/etc/sysconfig/libvirt-guests</code> that can improve the performance of shutdown/startup of libvirt and guests on the system as described in the RHEL Virtualization Administration Guide section [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_administration_guide/sub-sect-shutting_down_rebooting_and_force_shutdown_of_a_guest_virtual_machine-manipulating_the_libvirt_guests_configuration_settings 14.9.3. Manipulating the libvirt-guests Configuration Settings] . I updated the <code>PARALLEL_SHUTDOWN</code> parameter from the default of zero (no parallelism) to 6. I also updated <code>SYNC_TIME</code> to 1 so that libvirt will try to sync guest time on domain resume. To test the efficacy of these settings, a second and third reboot test was carried out, with DONQ crashing and requiring console restart each time.

To prepare for the "real" shutdown, I used <code>/usr/local/sbin/able-vm-startup.sh disable</code> to make the host safe for multiple reboots.

During the downtime, I cleaned the dust from the case and fans, topped up the radiator with distilled water and ran the system with the motherboard power bypassed so the pump would run. With the coolant circulating, i was able to verify the reservoir full after tilting it back and forth and restarting it a few times. I did have a probllem with the SATA power connector to the AOC assembly getting disconnected while i was working on it, resulting in no power to the pump.

Reboot was a hassle because the memory settings in BIOS has the XMP profile of the 64 GB kit still active, so the system would not event get to the POST screen. I removed all the new member, put in one stick of the old and got into BIOS. I then disabled XMP, saved and exited. I was than able to install the new RAM and get into BIOS to re-enable XMP with the profile of the new RAM. I had to manually set the RAM voltage to 1.35V. With that, the system booted clean and I was able to re-enable libvirtd and reboot to bring the guests back up. They loaded just fine from the suspend files.
===8/22/2023===
====Fix MegaRAID====
It turns out that MegaRAID was never properly installed, so there was no local CLI support nor any remote GUI support to check on the status of the RAID controller and the devices connected to it. I downloaded updated versions of the MSM software and updated the report on wort. For more information, see [http://wiki.kitsnet.us/index.php%3Ftitle=KitsNet_Operations:External_Packages:Linux:MegaRAID_Storage_Manager KitsNet Operations:External Packages:Linux:MegaRAID Storage Manager]. All packages associated with MSM (MegaRAID_Storage_Manager, sas_snmp and sas_ir_snmp) appeared to have most of their files missing despite showing as installed in rpm and dnf. I ended up removing all these three products in order to facilitate a clean install of the updated versions. With this version of MSM, it was also necessary to install a Java environment. AS pre the recommendation of the MSM readme file, I installed the older <u>java-1.8.0-openjdk</u> package via dnf. once all this was in place, I then executed the <code>install.csh</code> script for a complete install.

With the install completed, both the CLI and GUI versions of MS wer operational. However, the lcoation of the storcli64 image is now in the path at <code>/usr/local/sbin/storcli64</code>