The KitsNet Console Management environment is hosted on the server conswrks. This is a deployment of ConsoleWorks by TDi Technologies.
Socat is used on the KVM host to provide a connection point for ConsoleWorks to connect to the serial console of each Linux Guest. ConsoleWorks also runs syslog receivers Linux systems and network devices. It also makes console connections to the virtual VAX and Alpha systems in the environment.
1 Local Code and Customizations[edit | edit source]
1.1 socat-kvm[edit | edit source]
1.2 KitsNet-ConditionalCancel.sh[edit | edit source]
Some of the event definitons for KitsNet are based on broad patterns instead of highly specific messages. This modified "catchall" approach make maintenance easier and prevents missing significant issues, but it can fall into the trap of generating a lot of alert noise. What was needed was a way to let the more generalized patterns remain in effect, but for those more specific noise events, Acknowledge and Cancel them automatically so that we reduce operational toil. For this, the conditional cancellation automatic action was developed. The basis is a redis table with one or more entries per event. Each entry corresponds to a text pattern that, if matched, will cause us to conditionally cancel the event that fired. Entries in the table may be further qualified by console name so that the exception is more specific. Here is an example command used to add an entry to the table:
$ valkey-cli LPUSH KNLINUXCA-WARNNOIOCTL "SYSLOG_CATCHALL:.*the system time has been pushed back, adjusting active check schedule"
In this case SYSLOG_CATCHALL is the name of the specific console to have the conditional cancel to apply. The ‘:’ character is a delimiter. It will be necessary in most cases to use the .* sequence before the text pattern, but not always. There are some exceptionally clever things you can do in there (conditionally cancel before noon or perhaps only in August?).
For a full example, lets assume we are having trouble with is KNLINUXCA-WARNNOIOCTL. This is describe as Catchall Warning without deprecated SCSI ioctl. It is going to pick up most all rsyslog with Warning severity. The pattern is: ^\<(4|20|28|36|44|52|60|68|76|84|92|100|108|116|124)\>(?:(?!SG_IO).)*$
While this is rather handy, the system is getting swamped with line from a home router with a lousy clock crystal:
<28>Apr 3 13:47:36 billw.lan.kitsnet.us zabbix_agentd[2056]: the system time has been pushed back, adjusting active check schedule
So to make this work, associate the script /opt/ConsoleWorks/KitsNet/actions/event/KitsNet-ConditionalCancel.sh with an Auto Action and associate the Auto Action with the KNLINUXCA-WARNNOIOCTL event and the console(s) on which conditional cancellation should happen. The script borrows heavily from the auto_cancel.sh script provided by TDI.
#!/bin/bash
# $1 = Console
# $2 = Event
# $3 = Sequence Number
# $4 = Event Context File - delete before this script ends
# $5 = Contact
# $6 = User parameter
# $7 = Conwrks username that Acked/Purged event
#
CW_TERM=/opt/ConsoleWorks/bin/cwterm
LOGGER="#/usr/bin/logger"
#REDIS=/usr/bin/redis-cli
REDIS=/usr/bin/valkey-cli
REDIS_HOST="localhost"
REDIS_PORT="6379"
#
# Lookup this event and console to see if there are any conditional cancels.
#
CCpattern=$($REDIS -h $REDIS_HOST -p $REDIS_PORT LRANGE $2 0 -1)
if [ -n "$CCpattern" ]; then
$LOGGER "CCpattern: $CCpattern"
#
# Dig out log text that triggered Event
#
SERVER=CWserver
PORT=CWport
AUTHORIZATION=AUTH
CtxBlk=""
InContextBlock=0
while read input_line
do
if [ "$InContextBlock" -eq 1 ]
then
if [ "$input_line" = "CONTEXT_END:" ]
then
InContextBlock=0
else
CtxBlk="$CtxBlk\n$input_line"
fi
else
if [ "$input_line" = "CONTEXT_BEGIN:" ]
then
InContextBlock=1
else
for param in SERVER PORT AUTHORIZATION
do
let len=${#param}+2
beginning_of_line=$(echo "$input_line" | cut -c1-$len)
if [ "$param"": " = "$beginning_of_line" ]
then
let pos=len+1
val=$(echo "$input_line" | cut -c$pos-${#input_line})
eval tempvar=\$$param
eval $tempvar=$val
fi
done
fi
fi
done < $4
#
# See if console:text pattern matches one of the Conditional Cancels from Redis
#
EVT_INSTANCE="$1:$CtxBlk"
$LOGGER "EVT_INSTANCE: $EVT_INSTANCE"
while IFS= read -r pattern; do
$LOGGER "pattern: $pattern"
if [[ "$EVT_INSTANCE" =~ $pattern ]]; then
$CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport ack event -console=$1 -seq=$3 -comment="\"Autocancel $6\"" >/dev/null 2>&1
$CW_TERM -user=console_manager -auth=$AUTH -server=$CWserver -port=$CWport purge event -console=$1 -seq=$3 >/dev/null 2>&1
break
fi
done <<< "$CCpattern"
fi
rm $4
