KitsNet Operations:Monitoring:ConsoleWorks

The KitsNet Console Management environment is hosted on the server conswrks. This is a deployment of ConsoleWorks by TDi Technologies.

Socat is used on the KVM host to provide a connection point for ConsoleWorks to connect to the serial console of each Linux Guest. ConsoleWorks also runs syslog receivers Linux systems and network devices. It also makes console connections to the virtual VAX and Alpha systems in the environment.

1 Local Code and Customizations

1.2 KitsNet-ConditionalCancel.sh

Some of the event definitons for KitsNet are based on broad patterns instead of highly specific messages. This modified "catchall" approach make maintenance easier and prevents missing significant issues, but it can fall into the trap of generating a lot of alert noise. What was needed was a way to let the more generalized patterns remain in effect, but for those more specific noise events, Acknowledge and Cancel them automatically so that we reduce operational toil. For this, the conditional cancellation automatic action was developed. The basis is a redis table with one or more entries per event. Each entry corresponds to a text pattern that, if matched, will cause us to conditionally cancel the event that fired. Entries in the table may be further qualified by console name so that the exception is more specific. Here is an example command used to add an entry to the table:

$ valkey-cli LPUSH KNLINUXCA-WARNNOIOCTL "SYSLOG_CATCHALL:.*the system time has been pushed back, adjusting active check schedule"

In this case SYSLOG_CATCHALL is the name of the specific console to have the conditional cancel to apply. The ‘:’ character is a delimiter. It will be necessary in most cases to use the .* sequence before the text pattern, but not always. There are some exceptionally clever things you can do in there (conditionally cancel before noon or perhaps only in August?).

For a full example, lets assume we are having trouble with is KNLINUXCA-WARNNOIOCTL. This is describe as Catchall Warning without deprecated SCSI ioctl. It is going to pick up most all rsyslog with Warning severity. The pattern is: ^\<(4|20|28|36|44|52|60|68|76|84|92|100|108|116|124)\>(?:(?!SG_IO).)*$

While this is rather handy, the system is getting swamped with line from a home router with a lousy clock crystal:

<28>Apr 3 13:47:36 billw.lan.kitsnet.us zabbix_agentd[2056]: the system time has been pushed back, adjusting active check schedule

So to make this work, associate the script /opt/ConsoleWorks/KitsNet/actions/event/KitsNet-ConditionalCancel.sh with an Auto Action and associate the Auto Action with the KNLINUXCA-WARNNOIOCTL event and the console(s) on which conditional cancellation should happen. The script borrows heavily from the auto_cancel.sh script provided by TDI.

#!/bin/bash

#             $1 = Console
#             $2 = Event
#             $3 = Sequence Number
#             $4 = Event Context File - delete before this script ends
#             $5 = Contact
#             $6 = User parameter
#             $7 = Conwrks username that Acked/Purged event
#

CW_TERM=/opt/ConsoleWorks/bin/cwterm
LOGGER="#/usr/bin/logger"
#REDIS=/usr/bin/redis-cli
REDIS=/usr/bin/valkey-cli

REDIS_HOST="localhost"
REDIS_PORT="6379"


        #
        # Lookup this event and console to see if there are any conditional cancels.
        #
        CCpattern=$($REDIS -h $REDIS_HOST -p $REDIS_PORT LRANGE $2 0 -1)
        if [ -n "$CCpattern" ]; then
 $LOGGER "CCpattern: $CCpattern"
                #
                # Dig out log text that triggered Event
                #
                SERVER=CWserver
                PORT=CWport
                AUTHORIZATION=AUTH

                CtxBlk=""
                InContextBlock=0
                while read input_line
                do
                        if  [ "$InContextBlock" -eq 1 ]
                        then
                                if [ "$input_line" = "CONTEXT_END:" ]
                                then
                                        InContextBlock=0
                                else
                                        CtxBlk="$CtxBlk\n$input_line"
                                fi
                        else
                                if [ "$input_line" = "CONTEXT_BEGIN:" ]
                                then
                                        InContextBlock=1
                                else
                                        for param in SERVER PORT AUTHORIZATION
                                        do
                                                let len=${#param}+2
                                                beginning_of_line=$(echo "$input_line" | cut -c1-$len)
                                                if [ "$param"": " = "$beginning_of_line" ]
                                                then
                                                        let pos=len+1
                                                        val=$(echo "$input_line" | cut -c$pos-${#input_line})
                                                        eval tempvar=\$$param
                                                        eval $tempvar=$val
                                                fi
                                        done
                                fi
                        fi
                done < $4

                #
                # See if console:text pattern matches one of the Conditional Cancels from Redis
                #
                EVT_INSTANCE="$1:$CtxBlk"
 $LOGGER "EVT_INSTANCE: $EVT_INSTANCE"
                while IFS= read -r pattern; do
 $LOGGER "pattern: $pattern"
                    if [[ "$EVT_INSTANCE" =~ $pattern ]]; then
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport ack event -console=$1 -seq=$3 -comment="\"Autocancel $6\"" >/dev/null 2>&1
                        $CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport purge event -console=$1 -seq=$3 >/dev/null 2>&1
                        break
                    fi
                done <<< "$CCpattern"
        fi
        rm $4