KitsNet Operations:Monitoring:ConsoleWorks: Difference between revisions

Back to version history
← Older edit
VisualWikitext
Peter A. Smode (talk | contribs)
administrator, Bureaucrats, family, kitsnet_adm, Administrators
2,612 edits
→Enable/start all domains:: Fixed broken command
Peter A. Smode (talk | contribs)
administrator, Bureaucrats, family, kitsnet_adm, Administrators
2,612 edits
→Local Code and Customizations: recover console for av002
 
Line 330: Line 330:
To show all installed unit files use 'systemctl list-unit-files'.
To show all installed unit files use 'systemctl list-unit-files'.
</syntaxhighlight>
</syntaxhighlight>
===== Recover av002 Console for dusse =====
There have been times during a cold shutdown/startup of dusse that the console connection did no re-establish cleanly. It appears that this failure was a '''stale serial-console/getty state after the VM shutdown/start''', not a socat or libvirt configuration problem. To recover from this, execute this command on dusse to restat the serial console there: <code>sudo systemctl restart serial-getty@ttyS0.service</code>


===KitsNet-ConditionalCancel.sh===
===KitsNet-ConditionalCancel.sh===

Latest revision as of 11:44, 4 September 2026

The KitsNet Console Management environment is hosted on the server conswrks. This is a deployment of ConsoleWorks by TDi Technologies.

Socat is used on the KVM host to provide a connection point for ConsoleWorks to connect to the serial console of each Linux Guest. ConsoleWorks also runs syslog receivers Linux systems and network devices. It also makes console connections to the virtual VAX and Alpha systems in the environment.

1 Local Code and Customizations[edit | edit source]

1.1 socat-kvm[edit | edit source]

ConsoleWorks is used in KitsNet to connect to, capture and manage access to the consoles of KVM virtual machines. The underlying mechanism used is socat, making a connection to an IP port associated with the KVM guest at its creation on the KVM host. ConsoleWorks then reaches across the network to make telnet connections to these consoles. The trick has been to create and maintain the socat connections, while dealing with exception conditions like KVM host restart, creation of new KVM guests or shutdown/startup of one or more KVM guests. This infrastructure is now implemented on the KVM host OS on Rocky Linux 9 and up with a system unit file that automatically scales to support all the KVM guests.

To make this work, we start with a directory named /kvm/socat-kvm that has one file per VM with name corresponding to the KVM VM. These files have a single line with the IP port number on which socat should setup a listener that ConsoleWorks will connect in order to get to the console for the KVM guest. It will be necessary to use firewall-cmd to enable access to these ports (preferably limited by source IP of the ConsoleWorks server).

There is a single native systemd unit file that will create one kvm_console process per file in /kvm/socat-kvm. This provides granular control, with us able to start/stop/restart/query a single socat listener at a time. At the same time, we don’t have a buch of separate unit files to maintain because of the coding approach used. Here is /etc/systemd/system/socat-kvm@.service

[Unit]
Description=Socat KVM Console Listener for %i
After=network.target libvirtd.service
Requires=libvirtd.service

[Service]
Type=forking
User=root
Group=root

# Create /run/socat-kvm automatically
RuntimeDirectory=socat-kvm
PIDFile=/run/socat-kvm/socat-%i.pid

# Start the kvm_console for the given domain (%i = domain name)
ExecStart=/bin/bash -c '/usr/local/sbin/kvm_console %i $(cat /kvm/socat-kvm/%i) /run/socat-kvm/socat-%i.pid'
ExecStop=/bin/bash -c 'if [ -f /run/socat-kvm/socat-%i.pid ]; then kill $(cat /run/socat-kvm/socat-%i.pid); fi'


# Automatically restart if it fails
Restart=on-failure
RestartSec=2s

[Install]
WantedBy=multi-user.target

It relies on /usr/local/sbin/kvm_console

#!/bin/sh
#

KVM_DOMAIN=$1
TTYCONSOLE=$(virsh -c qemu:///system ttyconsole $KVM_DOMAIN)
IP_PORT=$2
PIDFILE=$3

if [ -z "$IP_PORT" ]
then
        echo "** BAD IP_PORT SPECIFIED: $IP_PORT **"
        exit 1
fi

if [ -z "$TTYCONSOLE" ]
then
        echo "** NO CONSOLE FOR KVM DOMAIN $KVM_DOMAIN **"
        exit 1
else
        echo -n "Socat console listener on $IP_PORT for KVM domain $KVM_DOMAIN"
        /usr/bin/socat -lm TCP-L:$IP_PORT,reuseaddr,fork file:$TTYCONSOLE,nonblock,echo=0,raw &
        SOCAT_PID=$!
        if [ -n "$PIDFILE" ]
        then
           echo $SOCAT_PID > $PIDFILE
        fi
fi

1.1.1 Management commands[edit | edit source]

1.1.1.1 Enable/start one domain[edit | edit source]

sudo systemctl enable --now socat-kvm@KVMguest.service

  • Where KVMguest is the name of the KVM guest domain
1.1.1.2 Enable/start all domains:[edit | edit source]

sudo systemctl restart $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')

1.1.1.3 Check all domains’ statuses:[edit | edit source]

systemctl status --no-pager -l $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')

[psmode@wort ~]$ systemctl status --no-pager -l $(systemctl list-units --all --type=service 'socat-kvm@*.service' --no-legend --plain | awk '{print $1}')
● socat-kvm@uv043.service - Socat KVM Console Listener for uv043
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116638 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv043 $(cat /kvm/socat-kvm/uv043) /run/socat-kvm/socat-uv043.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116815 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 23ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv043.service
             ├─1116815 /usr/bin/socat -lm TCP-L:7943,reuseaddr,fork file:/dev/pts/16,nonblock,echo=0,raw
             └─1116946 /usr/bin/socat -lm TCP-L:7943,reuseaddr,fork file:/dev/pts/16,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv043...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv043.

● socat-kvm@uv045.service - Socat KVM Console Listener for uv045
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116645 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv045 $(cat /kvm/socat-kvm/uv045) /run/socat-kvm/socat-uv045.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116813 (socat)
      Tasks: 1 (limit: 820180)
     Memory: 920.0K (peak: 3.2M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv045.service
             └─1116813 /usr/bin/socat -lm TCP-L:7945,reuseaddr,fork file:/dev/pts/20,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv045...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv045.

● socat-kvm@uv046.service - Socat KVM Console Listener for uv046
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116646 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv046 $(cat /kvm/socat-kvm/uv046) /run/socat-kvm/socat-uv046.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116809 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv046.service
             ├─1116809 /usr/bin/socat -lm TCP-L:7946,reuseaddr,fork file:/dev/pts/7,nonblock,echo=0,raw
             └─1117079 /usr/bin/socat -lm TCP-L:7946,reuseaddr,fork file:/dev/pts/7,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv046...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv046.

● socat-kvm@uv047.service - Socat KVM Console Listener for uv047
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116650 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv047 $(cat /kvm/socat-kvm/uv047) /run/socat-kvm/socat-uv047.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116810 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 19ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv047.service
             ├─1116810 /usr/bin/socat -lm TCP-L:7947,reuseaddr,fork file:/dev/pts/15,nonblock,echo=0,raw
             └─1116839 /usr/bin/socat -lm TCP-L:7947,reuseaddr,fork file:/dev/pts/15,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv047...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv047.

● socat-kvm@uv048.service - Socat KVM Console Listener for uv048
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116652 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv048 $(cat /kvm/socat-kvm/uv048) /run/socat-kvm/socat-uv048.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116816 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 21ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv048.service
             ├─1116816 /usr/bin/socat -lm TCP-L:7948,reuseaddr,fork file:/dev/pts/10,nonblock,echo=0,raw
             └─1117077 /usr/bin/socat -lm TCP-L:7948,reuseaddr,fork file:/dev/pts/10,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv048...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv048.

● socat-kvm@uv049.service - Socat KVM Console Listener for uv049
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116656 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv049 $(cat /kvm/socat-kvm/uv049) /run/socat-kvm/socat-uv049.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116803 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 22ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv049.service
             ├─1116803 /usr/bin/socat -lm TCP-L:7949,reuseaddr,fork file:/dev/pts/21,nonblock,echo=0,raw
             └─1116836 /usr/bin/socat -lm TCP-L:7949,reuseaddr,fork file:/dev/pts/21,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv049...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv049.

● socat-kvm@uv050.service - Socat KVM Console Listener for uv050
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116660 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv050 $(cat /kvm/socat-kvm/uv050) /run/socat-kvm/socat-uv050.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116818 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv050.service
             ├─1116818 /usr/bin/socat -lm TCP-L:7950,reuseaddr,fork file:/dev/pts/5,nonblock,echo=0,raw
             └─1117081 /usr/bin/socat -lm TCP-L:7950,reuseaddr,fork file:/dev/pts/5,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv050...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv050.

● socat-kvm@uv051.service - Socat KVM Console Listener for uv051
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116663 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv051 $(cat /kvm/socat-kvm/uv051) /run/socat-kvm/socat-uv051.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116808 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 22ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv051.service
             ├─1116808 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw
             └─1116838 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv051...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv051.

● socat-kvm@uv052.service - Socat KVM Console Listener for uv052
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116667 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv052 $(cat /kvm/socat-kvm/uv052) /run/socat-kvm/socat-uv052.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116807 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 21ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv052.service
             ├─1116807 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/8,nonblock,echo=0,raw
             └─1117078 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/8,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv052...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv052.

● socat-kvm@uv053.service - Socat KVM Console Listener for uv053
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116670 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv053 $(cat /kvm/socat-kvm/uv053) /run/socat-kvm/socat-uv053.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116805 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.1M (peak: 3.3M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv053.service
             ├─1116805 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/9,nonblock,echo=0,raw
             └─1116841 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/9,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv053...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv053.

● socat-kvm@uv054.service - Socat KVM Console Listener for uv054
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116674 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv054 $(cat /kvm/socat-kvm/uv054) /run/socat-kvm/socat-uv054.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116817 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 18ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv054.service
             ├─1116817 /usr/bin/socat -lm TCP-L:7954,reuseaddr,fork file:/dev/pts/4,nonblock,echo=0,raw
             └─1117082 /usr/bin/socat -lm TCP-L:7954,reuseaddr,fork file:/dev/pts/4,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv054...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv054.

● socat-kvm@uv055.service - Socat KVM Console Listener for uv055
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116676 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv055 $(cat /kvm/socat-kvm/uv055) /run/socat-kvm/socat-uv055.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116812 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 21ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv055.service
             ├─1116812 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/6,nonblock,echo=0,raw
             └─1116840 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/6,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv055...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv055.

● socat-kvm@uv056.service - Socat KVM Console Listener for uv056
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116683 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv056 $(cat /kvm/socat-kvm/uv056) /run/socat-kvm/socat-uv056.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116806 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv056.service
             ├─1116806 /usr/bin/socat -lm TCP-L:7956,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw
             └─1117080 /usr/bin/socat -lm TCP-L:7956,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv056...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv056.

● socat-kvm@uv057.service - Socat KVM Console Listener for uv057
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116686 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv057 $(cat /kvm/socat-kvm/uv057) /run/socat-kvm/socat-uv057.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116804 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.3M (peak: 3.2M)
        CPU: 21ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv057.service
             ├─1116804 /usr/bin/socat -lm TCP-L:7957,reuseaddr,fork file:/dev/pts/14,nonblock,echo=0,raw
             └─1116837 /usr/bin/socat -lm TCP-L:7957,reuseaddr,fork file:/dev/pts/14,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv057...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv057.

● socat-kvm@uv058.service - Socat KVM Console Listener for uv058
     Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
     Active: active (running) since Tue 2026-02-10 20:23:12 EST; 25min ago
    Process: 1116689 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv058 $(cat /kvm/socat-kvm/uv058) /run/socat-kvm/socat-uv058.pid (code=exited, status=0/SUCCESS)
   Main PID: 1116814 (socat)
      Tasks: 2 (limit: 820180)
     Memory: 1.0M (peak: 3.2M)
        CPU: 20ms
     CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv058.service
             ├─1116814 /usr/bin/socat -lm TCP-L:7958,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw
             └─1117076 /usr/bin/socat -lm TCP-L:7958,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw

Feb 10 20:23:11 wort.lan.kitsnet.us systemd[1]: Starting Socat KVM Console Listener for uv058...
Feb 10 20:23:12 wort.lan.kitsnet.us systemd[1]: Started Socat KVM Console Listener for uv058.
1.1.1.4 Check all domains’ statuses (concise list)[edit | edit source]

systemctl list-units --all 'socat-kvm@*.service'

[psmode@wort ~]$ systemctl list-units --all 'socat-kvm@*.service'
  UNIT                    LOAD   ACTIVE SUB     DESCRIPTION
  socat-kvm@uv043.service loaded active running Socat KVM Console Listener for uv043
  socat-kvm@uv045.service loaded active running Socat KVM Console Listener for uv045
  socat-kvm@uv046.service loaded active running Socat KVM Console Listener for uv046
  socat-kvm@uv047.service loaded active running Socat KVM Console Listener for uv047
  socat-kvm@uv048.service loaded active running Socat KVM Console Listener for uv048
  socat-kvm@uv049.service loaded active running Socat KVM Console Listener for uv049
  socat-kvm@uv050.service loaded active running Socat KVM Console Listener for uv050
  socat-kvm@uv051.service loaded active running Socat KVM Console Listener for uv051
  socat-kvm@uv052.service loaded active running Socat KVM Console Listener for uv052
  socat-kvm@uv053.service loaded active running Socat KVM Console Listener for uv053
  socat-kvm@uv054.service loaded active running Socat KVM Console Listener for uv054
  socat-kvm@uv055.service loaded active running Socat KVM Console Listener for uv055
  socat-kvm@uv056.service loaded active running Socat KVM Console Listener for uv056
  socat-kvm@uv057.service loaded active running Socat KVM Console Listener for uv057
  socat-kvm@uv058.service loaded active running Socat KVM Console Listener for uv058

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
15 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
1.1.1.5 Recover av002 Console for dusse[edit | edit source]

There have been times during a cold shutdown/startup of dusse that the console connection did no re-establish cleanly. It appears that this failure was a stale serial-console/getty state after the VM shutdown/start, not a socat or libvirt configuration problem. To recover from this, execute this command on dusse to restat the serial console there: sudo systemctl restart serial-getty@ttyS0.service

1.2 KitsNet-ConditionalCancel.sh[edit | edit source]

Some of the event definitons for KitsNet are based on broad patterns instead of highly specific messages. This modified "catchall" approach make maintenance easier and prevents missing significant issues, but it can fall into the trap of generating a lot of alert noise. What was needed was a way to let the more generalized patterns remain in effect, but for those more specific noise events, Acknowledge and Cancel them automatically so that we reduce operational toil. For this, the conditional cancellation automatic action was developed. The basis is a redis table with one or more entries per event. Each entry corresponds to a text pattern that, if matched, will cause us to conditionally cancel the event that fired. Entries in the table may be further qualified by console name so that the exception is more specific. Here is an example command used to add an entry to the table: 

$ valkey-cli LPUSH KNLINUXCA-WARNNOIOCTL "SYSLOG_CATCHALL:.*the system time has been pushed back, adjusting active check schedule"

In this case SYSLOG_CATCHALL is the name of the specific console to have the conditional cancel to apply. The ‘:’ character is a delimiter. It will be necessary in most cases to use the .* sequence before the text pattern, but not always. There are some exceptionally clever things you can do in there (conditionally cancel before noon or perhaps only in August?).

For a full example, lets assume we are having trouble with is KNLINUXCA-WARNNOIOCTL. This is describe as Catchall Warning without deprecated SCSI ioctl. It is going to pick up most all rsyslog with Warning severity. The pattern is: ^\<(4|20|28|36|44|52|60|68|76|84|92|100|108|116|124)\>(?:(?!SG_IO).)*$

While this is rather handy, the system is getting swamped with line from a home router with a lousy clock crystal:

<28>Apr 3 13:47:36 billw.lan.kitsnet.us zabbix_agentd[2056]: the system time has been pushed back, adjusting active check schedule

So to make this work, associate the script /opt/ConsoleWorks/KitsNet/actions/event/KitsNet-ConditionalCancel.sh with an Auto Action and associate the Auto Action with the KNLINUXCA-WARNNOIOCTL event and the console(s) on which conditional cancellation should happen. The script borrows heavily from the auto_cancel.sh script provided by TDI.

#!/bin/bash

#             $1 = Console
#             $2 = Event
#             $3 = Sequence Number
#             $4 = Event Context File - delete before this script ends
#             $5 = Contact
#             $6 = User parameter
#             $7 = Conwrks username that Acked/Purged event
#

CW_TERM=/opt/ConsoleWorks/bin/cwterm
LOGGER="#/usr/bin/logger"
#REDIS=/usr/bin/redis-cli
REDIS=/usr/bin/valkey-cli

REDIS_HOST="localhost"
REDIS_PORT="6379"


        #
        # Lookup this event and console to see if there are any conditional cancels.
        #
        CCpattern=$($REDIS -h $REDIS_HOST -p $REDIS_PORT LRANGE $2 0 -1)
        if [ -n "$CCpattern" ]; then
 $LOGGER "CCpattern: $CCpattern"
                #
                # Dig out log text that triggered Event
                #
                SERVER=CWserver
                PORT=CWport
                AUTHORIZATION=AUTH

                CtxBlk=""
                InContextBlock=0
                while read input_line
                do
                        if  [ "$InContextBlock" -eq 1 ]
                        then
                                if [ "$input_line" = "CONTEXT_END:" ]
                                then
                                        InContextBlock=0
                                else
                                        CtxBlk="$CtxBlk\n$input_line"
                                fi
                        else
                                if [ "$input_line" = "CONTEXT_BEGIN:" ]
                                then
                                        InContextBlock=1
                                else
                                        for param in SERVER PORT AUTHORIZATION
                                        do
                                                let len=${#param}+2
                                                beginning_of_line=$(echo "$input_line" | cut -c1-$len)
                                                if [ "$param"": " = "$beginning_of_line" ]
                                                then
                                                        let pos=len+1
                                                        val=$(echo "$input_line" | cut -c$pos-${#input_line})
                                                        eval tempvar=\$$param
                                                        eval $tempvar=$val
                                                fi
                                        done
                                fi
                        fi
                done < $4

                #
                # See if console:text pattern matches one of the Conditional Cancels from Redis
                #
                EVT_INSTANCE="$1:$CtxBlk"
 $LOGGER "EVT_INSTANCE: $EVT_INSTANCE"
                while IFS= read -r pattern; do
 $LOGGER "pattern: $pattern"
                    if [[ "$EVT_INSTANCE" =~ $pattern ]]; then
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport ack event -console=$1 -seq=$3 -comment="\"Autocancel $6\"" >/dev/null 2>&1
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport purge event -console=$1 -seq=$3 >/dev/null 2>&1
                        break
                    fi
                done <<< "$CCpattern"
        fi
        rm $4
Retrieved from "https://wiki.kitsnet.us/w/index.php?title=KitsNet_Operations:Monitoring:ConsoleWorks&oldid=2499"