KitsNet Operations:Monitoring:ConsoleWorks

The KitsNet Console Management environment is hosted on the server conswrks. This is a deployment of ConsoleWorks by TDi Technologies.

Socat is used on the KVM host to provide a connection point for ConsoleWorks to connect to the serial console of each Linux Guest. ConsoleWorks also runs syslog receivers Linux systems and network devices. It also makes console connections to the virtual VAX and Alpha systems in the environment.

1 Local Code and Customizations

1.1 socat-kvm

ConsoleWorks is used in KitsNet to connect to, capture and manage access to the consoles of KVM virtual machines. The underlying mechanism used is socat, making a connection to an IP port associated with the KVM guest at its creation on the KVM host. ConsoleWorks then reaches across the network to make telnet connections to these consoles. The trick has been to create and maintain the socat connections, while dealing with exception conditions like KVM host restart, creation of new KVM guests or shutdown/startup of one or more KVM guests. This infrastructure is now implemented on the KVM host OS on Rocky Linux 9 and up with a system unit file that automatically scales to support all the KVM guests.

To make this work, we start with a directory named /kvm/socat-kvm that has one file per VM with name corresponding to the KVM VM. These files have a single line with the IP port number on which socat should setup a listener that ConsoleWorks will connect in order to get to the console for the KVM guest. It will be necessary to use firewall-cmd to enable access to these ports (preferably limited by source IP of the ConsoleWorks server).

There is a single native systemd unit file that will create one kvm_console process per file in /kvm/socat-kvm. This provides granular control, with us able to start/stop/restart/query a single socat listener at a time. At the same time, we don’t have a buch of separate unit files to maintain because of the coding approach used. Here is /etc/systemd/system/socat-kvm@.service

[Unit]
Description=Socat KVM Console Listener for %i
After=network.target libvirtd.service
Requires=libvirtd.service

[Service]
Type=forking
User=root
Group=root

# Create /run/socat-kvm automatically
RuntimeDirectory=socat-kvm
PIDFile=/run/socat-kvm/socat-%i.pid

# Start the kvm_console for the given domain (%i = domain name)
ExecStart=/bin/bash -c '/usr/local/sbin/kvm_console %i $(cat /kvm/socat-kvm/%i) /run/socat-kvm/socat-%i.pid'
ExecStop=/bin/bash -c 'if [ -f /run/socat-kvm/socat-%i.pid ]; then kill $(cat /run/socat-kvm/socat-%i.pid); fi'


# Automatically restart if it fails
Restart=on-failure
RestartSec=2s

[Install]
WantedBy=multi-user.target

It relies on /usr/local/sbin/kvm_console

#!/bin/sh
#

KVM_DOMAIN=$1
TTYCONSOLE=$(virsh -c qemu:///system ttyconsole $KVM_DOMAIN)
IP_PORT=$2
PIDFILE=$3

if [ -z "$IP_PORT" ]
then
        echo "** BAD IP_PORT SPECIFIED: $IP_PORT **"
        exit 1
fi

if [ -z "$TTYCONSOLE" ]
then
        echo "** NO CONSOLE FOR KVM DOMAIN $KVM_DOMAIN **"
        exit 1
else
        echo -n "Socat console listener on $IP_PORT for KVM domain $KVM_DOMAIN"
        /usr/bin/socat -lm TCP-L:$IP_PORT,reuseaddr,fork file:$TTYCONSOLE,nonblock,echo=0,raw &
        SOCAT_PID=$!
        if [ -n "$PIDFILE" ]
        then
           echo $SOCAT_PID > $PIDFILE
        fi
fi

1.1.1 Management commands

1.1.1.1 Enable/start one domain

sudo systemctl enable --now socat-kvm@KVMguest.service

• Where KVMguest is the name of the KVM guest domain

1.1.1.2 Enable/start all domains:

• sudo systemctl restart 'socat-kvm@*.service'

1.1.1.3 Check all domains’ statuses:

• systemctl status 'socat-kvm@*.service'

  [root@wort kvm]# systemctl status 'socat-kvm@*.service' 
  ● socat-kvm@uv053.service - Socat KVM Console Listener for uv053
       Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
       Active: active (running) since Sat 2025-08-16 13:29:44 EDT; 2 months 11 days ago
     Main PID: 104080 (socat)
        Tasks: 2 (limit: 820187)
       Memory: 972.0K
          CPU: 7.431s
       CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv053.service
               ├─ 104080 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/17,nonblock,echo=0,raw
               └─2610517 /usr/bin/socat -lm TCP-L:7953,reuseaddr,fork file:/dev/pts/17,nonblock,echo=0,raw
  
  Notice: journal has been rotated since unit was started, output may be incomplete.
  
  ● socat-kvm@uv052.service - Socat KVM Console Listener for uv052
       Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
       Active: active (running) since Sat 2025-08-16 13:29:41 EDT; 2 months 11 days ago
     Main PID: 104023 (socat)
        Tasks: 2 (limit: 820187)
       Memory: 976.0K
          CPU: 7.398s
       CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv052.service
               ├─ 104023 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw
               └─2610516 /usr/bin/socat -lm TCP-L:7952,reuseaddr,fork file:/dev/pts/2,nonblock,echo=0,raw
  
  Notice: journal has been rotated since unit was started, output may be incomplete.
  
  ● socat-kvm@uv055.service - Socat KVM Console Listener for uv055
       Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
       Active: active (running) since Sun 2025-08-24 12:49:02 EDT; 2 months 3 days ago
     Main PID: 4148712 (socat)
        Tasks: 2 (limit: 820187)
       Memory: 972.0K
          CPU: 7.911s
       CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv055.service
               ├─2610515 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw
               └─4148712 /usr/bin/socat -lm TCP-L:7955,reuseaddr,fork file:/dev/pts/13,nonblock,echo=0,raw
  
  Notice: journal has been rotated since unit was started, output may be incomplete.
  
  ● socat-kvm@uv051.service - Socat KVM Console Listener for uv051
       Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
       Active: active (running) since Sat 2025-08-16 13:29:41 EDT; 2 months 11 days ago
     Main PID: 104021 (socat)
        Tasks: 2 (limit: 820187)
       Memory: 972.0K
          CPU: 7.104s
       CGroup: /system.slice/system-socat\x2dkvm.slice/socat-kvm@uv051.service
               ├─ 104021 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw
               └─2610513 /usr/bin/socat -lm TCP-L:7951,reuseaddr,fork file:/dev/pts/0,nonblock,echo=0,raw
  
  Notice: journal has been rotated since unit was started, output may be incomplete.
  
  ● socat-kvm@uv043.service - Socat KVM Console Listener for uv043
       Loaded: loaded (/etc/systemd/system/socat-kvm@.service; enabled; preset: disabled)
       Active: active (running) since Fri 2025-08-29 08:52:47 EDT; 1 month 28 days ago
      Process: 3384497 ExecStart=/bin/bash -c /usr/local/sbin/kvm_console uv043 $(cat /kvm/socat-kvm/uv043) /run/socat-kvm/socat-uv043.pid (code=exited, status=0/SUCCESS)
     Main PID: 3384501 (socat)
        Tasks: 2 (limit: 820187)
       Memory: 972.0K
          CPU: 11.218s
  lines 1-60
  

1.1.1.4 Check all domains’ statuses (concise list)

• systemctl list-units --type=service | grep socat-kvm@

  [root@wort ~]# systemctl list-units --type=service | grep socat-kvm@
    socat-kvm@uv043.service                               loaded active running Socat KVM Console Listener for uv043
    socat-kvm@uv045.service                               loaded active running Socat KVM Console Listener for uv045
    socat-kvm@uv046.service                               loaded active running Socat KVM Console Listener for uv046
    socat-kvm@uv047.service                               loaded active running Socat KVM Console Listener for uv047
    socat-kvm@uv048.service                               loaded active running Socat KVM Console Listener for uv048
    socat-kvm@uv049.service                               loaded active running Socat KVM Console Listener for uv049
    socat-kvm@uv050.service                               loaded active running Socat KVM Console Listener for uv050
    socat-kvm@uv051.service                               loaded active running Socat KVM Console Listener for uv051
    socat-kvm@uv052.service                               loaded active running Socat KVM Console Listener for uv052
    socat-kvm@uv053.service                               loaded active running Socat KVM Console Listener for uv053
    socat-kvm@uv054.service                               loaded active running Socat KVM Console Listener for uv054
    socat-kvm@uv055.service                               loaded active running Socat KVM Console Listener for uv055
    socat-kvm@uv056.service                               loaded active running Socat KVM Console Listener for uv056
    socat-kvm@uv057.service                               loaded active running Socat KVM Console Listener for uv057
    socat-kvm@uv058.service                               loaded active running Socat KVM Console Listener for uv058
  

1.2 KitsNet-ConditionalCancel.sh

Some of the event definitons for KitsNet are based on broad patterns instead of highly specific messages. This modified "catchall" approach make maintenance easier and prevents missing significant issues, but it can fall into the trap of generating a lot of alert noise. What was needed was a way to let the more generalized patterns remain in effect, but for those more specific noise events, Acknowledge and Cancel them automatically so that we reduce operational toil. For this, the conditional cancellation automatic action was developed. The basis is a redis table with one or more entries per event. Each entry corresponds to a text pattern that, if matched, will cause us to conditionally cancel the event that fired. Entries in the table may be further qualified by console name so that the exception is more specific. Here is an example command used to add an entry to the table:

$ valkey-cli LPUSH KNLINUXCA-WARNNOIOCTL "SYSLOG_CATCHALL:.*the system time has been pushed back, adjusting active check schedule"

In this case SYSLOG_CATCHALL is the name of the specific console to have the conditional cancel to apply. The ‘:’ character is a delimiter. It will be necessary in most cases to use the .* sequence before the text pattern, but not always. There are some exceptionally clever things you can do in there (conditionally cancel before noon or perhaps only in August?).

For a full example, lets assume we are having trouble with is KNLINUXCA-WARNNOIOCTL. This is describe as Catchall Warning without deprecated SCSI ioctl. It is going to pick up most all rsyslog with Warning severity. The pattern is: ^\<(4|20|28|36|44|52|60|68|76|84|92|100|108|116|124)\>(?:(?!SG_IO).)*$

While this is rather handy, the system is getting swamped with line from a home router with a lousy clock crystal:

<28>Apr 3 13:47:36 billw.lan.kitsnet.us zabbix_agentd[2056]: the system time has been pushed back, adjusting active check schedule

So to make this work, associate the script /opt/ConsoleWorks/KitsNet/actions/event/KitsNet-ConditionalCancel.sh with an Auto Action and associate the Auto Action with the KNLINUXCA-WARNNOIOCTL event and the console(s) on which conditional cancellation should happen. The script borrows heavily from the auto_cancel.sh script provided by TDI.

#!/bin/bash

#             $1 = Console
#             $2 = Event
#             $3 = Sequence Number
#             $4 = Event Context File - delete before this script ends
#             $5 = Contact
#             $6 = User parameter
#             $7 = Conwrks username that Acked/Purged event
#

CW_TERM=/opt/ConsoleWorks/bin/cwterm
LOGGER="#/usr/bin/logger"
#REDIS=/usr/bin/redis-cli
REDIS=/usr/bin/valkey-cli

REDIS_HOST="localhost"
REDIS_PORT="6379"


        #
        # Lookup this event and console to see if there are any conditional cancels.
        #
        CCpattern=$($REDIS -h $REDIS_HOST -p $REDIS_PORT LRANGE $2 0 -1)
        if [ -n "$CCpattern" ]; then
 $LOGGER "CCpattern: $CCpattern"
                #
                # Dig out log text that triggered Event
                #
                SERVER=CWserver
                PORT=CWport
                AUTHORIZATION=AUTH

                CtxBlk=""
                InContextBlock=0
                while read input_line
                do
                        if  [ "$InContextBlock" -eq 1 ]
                        then
                                if [ "$input_line" = "CONTEXT_END:" ]
                                then
                                        InContextBlock=0
                                else
                                        CtxBlk="$CtxBlk\n$input_line"
                                fi
                        else
                                if [ "$input_line" = "CONTEXT_BEGIN:" ]
                                then
                                        InContextBlock=1
                                else
                                        for param in SERVER PORT AUTHORIZATION
                                        do
                                                let len=${#param}+2
                                                beginning_of_line=$(echo "$input_line" | cut -c1-$len)
                                                if [ "$param"": " = "$beginning_of_line" ]
                                                then
                                                        let pos=len+1
                                                        val=$(echo "$input_line" | cut -c$pos-${#input_line})
                                                        eval tempvar=\$$param
                                                        eval $tempvar=$val
                                                fi
                                        done
                                fi
                        fi
                done < $4

                #
                # See if console:text pattern matches one of the Conditional Cancels from Redis
                #
                EVT_INSTANCE="$1:$CtxBlk"
 $LOGGER "EVT_INSTANCE: $EVT_INSTANCE"
                while IFS= read -r pattern; do
 $LOGGER "pattern: $pattern"
                    if [[ "$EVT_INSTANCE" =~ $pattern ]]; then
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport ack event -console=$1 -seq=$3 -comment="\"Autocancel $6\"" >/dev/null 2>&1
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport purge event -console=$1 -seq=$3 >/dev/null 2>&1
                        break
                    fi
                done <<< "$CCpattern"
        fi
        rm $4