→Check all domains’ statuses:: fixed reporting commands |
→Enable/start all domains:: Fixed broken command |
||
| Line 74: | Line 74: | ||
===== Enable/start all domains:===== | ===== Enable/start all domains:===== | ||
<code>sudo systemctl restart 'socat-kvm@*.service'</code> | <code>sudo systemctl restart $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')</code> | ||
===== Check all domains’ statuses: ===== | ===== Check all domains’ statuses: ===== | ||
Revision as of 09:37, 20 March 2026
The KitsNet Console Management environment is hosted on the server conswrks. This is a deployment of ConsoleWorks by TDi Technologies.
Socat is used on the KVM host to provide a connection point for ConsoleWorks to connect to the serial console of each Linux Guest. ConsoleWorks also runs syslog receivers Linux systems and network devices. It also makes console connections to the virtual VAX and Alpha systems in the environment.
1 Local Code and Customizations[edit | edit source]
1.1 socat-kvm[edit | edit source]
ConsoleWorks is used in KitsNet to connect to, capture and manage access to the consoles of KVM virtual machines. The underlying mechanism used is socat, making a connection to an IP port associated with the KVM guest at its creation on the KVM host. ConsoleWorks then reaches across the network to make telnet connections to these consoles. The trick has been to create and maintain the socat connections, while dealing with exception conditions like KVM host restart, creation of new KVM guests or shutdown/startup of one or more KVM guests. This infrastructure is now implemented on the KVM host OS on Rocky Linux 9 and up with a system unit file that automatically scales to support all the KVM guests.
To make this work, we start with a directory named /kvm/socat-kvm that has one file per VM with name corresponding to the KVM VM. These files have a single line with the IP port number on which socat should setup a listener that ConsoleWorks will connect in order to get to the console for the KVM guest. It will be necessary to use firewall-cmd to enable access to these ports (preferably limited by source IP of the ConsoleWorks server).
There is a single native systemd unit file that will create one kvm_console process per file in /kvm/socat-kvm. This provides granular control, with us able to start/stop/restart/query a single socat listener at a time. At the same time, we don’t have a buch of separate unit files to maintain because of the coding approach used. Here is /etc/systemd/system/socat-kvm@.service
[Unit]
Description=Socat KVM Console Listener for %i
After=network.target libvirtd.service
Requires=libvirtd.service
[Service]
Type=forking
User=root
Group=root
# Create /run/socat-kvm automatically
RuntimeDirectory=socat-kvm
PIDFile=/run/socat-kvm/socat-%i.pid
# Start the kvm_console for the given domain (%i = domain name)
ExecStart=/bin/bash -c '/usr/local/sbin/kvm_console %i $(cat /kvm/socat-kvm/%i) /run/socat-kvm/socat-%i.pid'
ExecStop=/bin/bash -c 'if [ -f /run/socat-kvm/socat-%i.pid ]; then kill $(cat /run/socat-kvm/socat-%i.pid); fi'
# Automatically restart if it fails
Restart=on-failure
RestartSec=2s
[Install]
WantedBy=multi-user.target
It relies on /usr/local/sbin/kvm_console
#!/bin/sh
#
KVM_DOMAIN=$1
TTYCONSOLE=$(virsh -c qemu:///system ttyconsole $KVM_DOMAIN)
IP_PORT=$2
PIDFILE=$3
if [ -z "$IP_PORT" ]
then
echo "** BAD IP_PORT SPECIFIED: $IP_PORT **"
exit 1
fi
if [ -z "$TTYCONSOLE" ]
then
echo "** NO CONSOLE FOR KVM DOMAIN $KVM_DOMAIN **"
exit 1
else
echo -n "Socat console listener on $IP_PORT for KVM domain $KVM_DOMAIN"
/usr/bin/socat -lm TCP-L:$IP_PORT,reuseaddr,fork file:$TTYCONSOLE,nonblock,echo=0,raw &
SOCAT_PID=$!
if [ -n "$PIDFILE" ]
then
echo $SOCAT_PID > $PIDFILE
fi
fi
1.1.1 Management commands[edit | edit source]
1.1.1.1 Enable/start one domain[edit | edit source]
sudo systemctl enable --now socat-kvm@KVMguest.service
- Where KVMguest is the name of the KVM guest domain
1.1.1.2 Enable/start all domains:[edit | edit source]
sudo systemctl restart $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')
1.1.1.3 Check all domains’ statuses:[edit | edit source]
systemctl status --no-pager -l $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')
[psmode@wort ~]$ systemctl status --no-pager -l $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')
● socat-kvm@uv043.service - Socat KVM Console Listener for uv043
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116638 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv043 $(cat /kvm/socat-kvm/uv043) /run/socat-kvm/socat-uv043.pid (code=exited, status=0/SUCCESS)
Main PID: 1116815 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 23ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv043.service
├─1116815 /usr/bin/socat -lm TCP-L:7943,reuseaddr,fork file:/dev/pts/16,nonblock,echo=0,raw
└─1116946 /usr/bin/socat -lm TCP-L:7943,reuseaddr,fork file:/dev/pts/16,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv043...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv043.
● socat-kvm@uv045.service - Socat KVM Console Listener for uv045
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116645 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv045 $(cat /kvm/socat-kvm/uv045) /run/socat-kvm/socat-uv045.pid (code=exited, status=0/SUCCESS)
Main PID: 1116813 (socat)
Tasks: 1 (limit: 820180)
Memory: 920.0K (peak: 3.2M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv045.service
└─1116813 /usr/bin/socat -lm TCP-L:7945,reuseaddr,fork file:/dev/pts/20,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv045...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv045.
● socat-kvm@uv046.service - Socat KVM Console Listener for uv046
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116646 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv046 $(cat /kvm/socat-kvm/uv046) /run/socat-kvm/socat-uv046.pid (code=exited, status=0/SUCCESS)
Main PID: 1116809 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv046.service
├─1116809 /usr/bin/socat -lm TCP-L:7946,reuseaddr,fork file:/dev/pts/7,nonblock,echo=0,raw
└─1117079 /usr/bin/socat -lm TCP-L:7946,reuseaddr,fork file:/dev/pts/7,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv046...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv046.
● socat-kvm@uv047.service - Socat KVM Console Listener for uv047
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116650 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv047 $(cat /kvm/socat-kvm/uv047) /run/socat-kvm/socat-uv047.pid (code=exited, status=0/SUCCESS)
Main PID: 1116810 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 19ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv047.service
├─1116810 /usr/bin/socat -lm TCP-L:7947,reuseaddr,fork file:/dev/pts/15,nonblock,echo=0,raw
└─1116839 /usr/bin/socat -lm TCP-L:7947,reuseaddr,fork file:/dev/pts/15,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv047...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv047.
● socat-kvm@uv048.service - Socat KVM Console Listener for uv048
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116652 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv048 $(cat /kvm/socat-kvm/uv048) /run/socat-kvm/socat-uv048.pid (code=exited, status=0/SUCCESS)
Main PID: 1116816 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 21ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv048.service
├─1116816 /usr/bin/socat -lm TCP-L:7948,reuseaddr,fork file:/dev/pts/10,nonblock,echo=0,raw
└─1117077 /usr/bin/socat -lm TCP-L:7948,reuseaddr,fork file:/dev/pts/10,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv048...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv048.
● socat-kvm@uv049.service - Socat KVM Console Listener for uv049
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116656 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv049 $(cat /kvm/socat-kvm/uv049) /run/socat-kvm/socat-uv049.pid (code=exited, status=0/SUCCESS)
Main PID: 1116803 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 22ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv049.service
├─1116803 /usr/bin/socat -lm TCP-L:7949,reuseaddr,fork file:/dev/pts/21,nonblock,echo=0,raw
└─1116836 /usr/bin/socat -lm TCP-L:7949,reuseaddr,fork file:/dev/pts/21,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv049...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv049.
● socat-kvm@uv050.service - Socat KVM Console Listener for uv050
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116660 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv050 $(cat /kvm/socat-kvm/uv050) /run/socat-kvm/socat-uv050.pid (code=exited, status=0/SUCCESS)
Main PID: 1116818 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv050.service
├─1116818 /usr/bin/socat -lm TCP-L:7950,reuseaddr,fork file:/dev/pts/5,nonblock,echo=0,raw
└─1117081 /usr/bin/socat -lm TCP-L:7950,reuseaddr,fork file:/dev/pts/5,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv050...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv050.
● socat-kvm@uv051.service - Socat KVM Console Listener for uv051
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116663 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv051 $(cat /kvm/socat-kvm/uv051) /run/socat-kvm/socat-uv051.pid (code=exited, status=0/SUCCESS)
Main PID: 1116808 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 22ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv051.service
├─1116808 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw
└─1116838 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv051...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv051.
● socat-kvm@uv052.service - Socat KVM Console Listener for uv052
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116667 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv052 $(cat /kvm/socat-kvm/uv052) /run/socat-kvm/socat-uv052.pid (code=exited, status=0/SUCCESS)
Main PID: 1116807 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 21ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv052.service
├─1116807 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/8,nonblock,echo=0,raw
└─1117078 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/8,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv052...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv052.
● socat-kvm@uv053.service - Socat KVM Console Listener for uv053
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116670 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv053 $(cat /kvm/socat-kvm/uv053) /run/socat-kvm/socat-uv053.pid (code=exited, status=0/SUCCESS)
Main PID: 1116805 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.1M (peak: 3.3M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv053.service
├─1116805 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/9,nonblock,echo=0,raw
└─1116841 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/9,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv053...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv053.
● socat-kvm@uv054.service - Socat KVM Console Listener for uv054
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116674 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv054 $(cat /kvm/socat-kvm/uv054) /run/socat-kvm/socat-uv054.pid (code=exited, status=0/SUCCESS)
Main PID: 1116817 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 18ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv054.service
├─1116817 /usr/bin/socat -lm TCP-L:7954,reuseaddr,fork file:/dev/pts/4,nonblock,echo=0,raw
└─1117082 /usr/bin/socat -lm TCP-L:7954,reuseaddr,fork file:/dev/pts/4,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv054...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv054.
● socat-kvm@uv055.service - Socat KVM Console Listener for uv055
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116676 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv055 $(cat /kvm/socat-kvm/uv055) /run/socat-kvm/socat-uv055.pid (code=exited, status=0/SUCCESS)
Main PID: 1116812 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 21ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv055.service
├─1116812 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/6,nonblock,echo=0,raw
└─1116840 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/6,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv055...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv055.
● socat-kvm@uv056.service - Socat KVM Console Listener for uv056
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116683 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv056 $(cat /kvm/socat-kvm/uv056) /run/socat-kvm/socat-uv056.pid (code=exited, status=0/SUCCESS)
Main PID: 1116806 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv056.service
├─1116806 /usr/bin/socat -lm TCP-L:7956,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw
└─1117080 /usr/bin/socat -lm TCP-L:7956,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv056...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv056.
● socat-kvm@uv057.service - Socat KVM Console Listener for uv057
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116686 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv057 $(cat /kvm/socat-kvm/uv057) /run/socat-kvm/socat-uv057.pid (code=exited, status=0/SUCCESS)
Main PID: 1116804 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.3M (peak: 3.2M)
CPU: 21ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv057.service
├─1116804 /usr/bin/socat -lm TCP-L:7957,reuseaddr,fork file:/dev/pts/14,nonblock,echo=0,raw
└─1116837 /usr/bin/socat -lm TCP-L:7957,reuseaddr,fork file:/dev/pts/14,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv057...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv057.
● socat-kvm@uv058.service - Socat KVM Console Listener for uv058
Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
Process: 1116689 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv058 $(cat /kvm/socat-kvm/uv058) /run/socat-kvm/socat-uv058.pid (code=exited, status=0/SUCCESS)
Main PID: 1116814 (socat)
Tasks: 2 (limit: 820180)
Memory: 1.0M (peak: 3.2M)
CPU: 20ms
CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv058.service
├─1116814 /usr/bin/socat -lm TCP-L:7958,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw
└─1117076 /usr/bin/socat -lm TCP-L:7958,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw
Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv058...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv058.
1.1.1.4 Check all domains’ statuses (concise list)[edit | edit source]
systemctl list-units --all 'socat-kvm@*.service'
[psmode@wort ~]$ systemctl list-units --all 'socat-kvm@*.service'
UNIT LOAD ACTIVE SUB DESCRIPTION
socat-kvm@uv043.service loaded active running Socat KVM Console Listener for uv043
socat-kvm@uv045.service loaded active running Socat KVM Console Listener for uv045
socat-kvm@uv046.service loaded active running Socat KVM Console Listener for uv046
socat-kvm@uv047.service loaded active running Socat KVM Console Listener for uv047
socat-kvm@uv048.service loaded active running Socat KVM Console Listener for uv048
socat-kvm@uv049.service loaded active running Socat KVM Console Listener for uv049
socat-kvm@uv050.service loaded active running Socat KVM Console Listener for uv050
socat-kvm@uv051.service loaded active running Socat KVM Console Listener for uv051
socat-kvm@uv052.service loaded active running Socat KVM Console Listener for uv052
socat-kvm@uv053.service loaded active running Socat KVM Console Listener for uv053
socat-kvm@uv054.service loaded active running Socat KVM Console Listener for uv054
socat-kvm@uv055.service loaded active running Socat KVM Console Listener for uv055
socat-kvm@uv056.service loaded active running Socat KVM Console Listener for uv056
socat-kvm@uv057.service loaded active running Socat KVM Console Listener for uv057
socat-kvm@uv058.service loaded active running Socat KVM Console Listener for uv058
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
15 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
1.2 KitsNet-ConditionalCancel.sh[edit | edit source]
Some of the event definitons for KitsNet are based on broad patterns instead of highly specific messages. This modified "catchall" approach make maintenance easier and prevents missing significant issues, but it can fall into the trap of generating a lot of alert noise. What was needed was a way to let the more generalized patterns remain in effect, but for those more specific noise events, Acknowledge and Cancel them automatically so that we reduce operational toil. For this, the conditional cancellation automatic action was developed. The basis is a redis table with one or more entries per event. Each entry corresponds to a text pattern that, if matched, will cause us to conditionally cancel the event that fired. Entries in the table may be further qualified by console name so that the exception is more specific. Here is an example command used to add an entry to the table:
$ valkey-cli LPUSH KNLINUXCA-WARNNOIOCTL "SYSLOG_CATCHALL:.*the system time has been pushed back, adjusting active check schedule"
In this case SYSLOG_CATCHALL is the name of the specific console to have the conditional cancel to apply. The ‘:’ character is a delimiter. It will be necessary in most cases to use the .* sequence before the text pattern, but not always. There are some exceptionally clever things you can do in there (conditionally cancel before noon or perhaps only in August?).
For a full example, lets assume we are having trouble with is KNLINUXCA-WARNNOIOCTL. This is describe as Catchall Warning without deprecated SCSI ioctl. It is going to pick up most all rsyslog with Warning severity. The pattern is: ^\<(4|20|28|36|44|52|60|68|76|84|92|100|108|116|124)\>(?:(?!SG_IO).)*$
While this is rather handy, the system is getting swamped with line from a home router with a lousy clock crystal:
<28>Apr 3 13:47:36 billw.lan.kitsnet.us zabbix_agentd[2056]: the system time has been pushed back, adjusting active check schedule
So to make this work, associate the script /opt/ConsoleWorks/KitsNet/actions/event/KitsNet-ConditionalCancel.sh with an Auto Action and associate the Auto Action with the KNLINUXCA-WARNNOIOCTL event and the console(s) on which conditional cancellation should happen. The script borrows heavily from the auto_cancel.sh script provided by TDI.
#!/bin/bash
# $1 = Console
# $2 = Event
# $3 = Sequence Number
# $4 = Event Context File - delete before this script ends
# $5 = Contact
# $6 = User parameter
# $7 = Conwrks username that Acked/Purged event
#
CW_TERM=/opt/ConsoleWorks/bin/cwterm
LOGGER="#/usr/bin/logger"
#REDIS=/usr/bin/redis-cli
REDIS=/usr/bin/valkey-cli
REDIS_HOST="localhost"
REDIS_PORT="6379"
#
# Lookup this event and console to see if there are any conditional cancels.
#
CCpattern=$($REDIS -h $REDIS_HOST -p $REDIS_PORT LRANGE $2 0 -1)
if [ -n "$CCpattern" ]; then
$LOGGER "CCpattern: $CCpattern"
#
# Dig out log text that triggered Event
#
SERVER=CWserver
PORT=CWport
AUTHORIZATION=AUTH
CtxBlk=""
InContextBlock=0
while read input_line
do
if [ "$InContextBlock" -eq 1 ]
then
if [ "$input_line" = "CONTEXT_END:" ]
then
InContextBlock=0
else
CtxBlk="$CtxBlk\n$input_line"
fi
else
if [ "$input_line" = "CONTEXT_BEGIN:" ]
then
InContextBlock=1
else
for param in SERVER PORT AUTHORIZATION
do
let len=${#param}+2
beginning_of_line=$(echo "$input_line" | cut -c1-$len)
if [ "$param"": " = "$beginning_of_line" ]
then
let pos=len+1
val=$(echo "$input_line" | cut -c$pos-${#input_line})
eval tempvar=\$$param
eval $tempvar=$val
fi
done
fi
fi
done < $4
#
# See if console:text pattern matches one of the Conditional Cancels from Redis
#
EVT_INSTANCE="$1:$CtxBlk"
$LOGGER "EVT_INSTANCE: $EVT_INSTANCE"
while IFS= read -r pattern; do
$LOGGER "pattern: $pattern"
if [[ "$EVT_INSTANCE" =~ $pattern ]]; then
$CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport ack event -console=$1 -seq=$3 -comment="\"Autocancel $6\"" >/dev/null 2>&1
$CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport purge event -console=$1 -seq=$3 >/dev/null 2>&1
break
fi
done <<< "$CCpattern"
fi
rm $4
