Linux:Tips and Tricks

Find files modified in past N days

find path -mtime -N -ls

[psmode@wort ~]$ find /kvm/ -mtime -3 -ls
102090127      4 drwxr-xr-x   7  root     root         4096 Jan  3 09:46 /kvm/
  2038539      4 drwxr-xr-x   2  root     kitsnet_adm     4096 Jan  3 09:13 /kvm/socat-kvm
  2038573      4 -rw-r--r--   1  root     root               5 Jan  3 09:13 /kvm/socat-kvm/uv040
101237691      8 -rw-r--r--   1  root     kitsnet_adm     6370 Jan  3 09:46 /kvm/uv034.xml
102163845     12 -rw-r--r--   1  root     root            9096 Jan  3 09:46 /kvm/uv039.xml
102090129     12 -rw-r--r--   1  root     root            8303 Jan  3 09:46 /kvm/uv040.xml

Search files modified in past N days

find path -mtime -N -type f -exec grep pattern  {} /dev/null \;

[psmode@ciroc ~]$ sudo find /etc -mtime -9 -type f -exec grep mailroom1  {} /dev/null \;
/etc/httpd/conf/httpd.conf:    DocumentRoot "/var/www/virt-html/mailroom1.kitsnet.us"
/etc/httpd/conf/httpd.conf:    ServerName mailroom1.kitsnet.us
/etc/httpd/conf/httpd.conf:#RewriteCond %{SERVER_NAME} =mailroom1.kitsnet.us

Shell script debugging modes

A good explanation of the debugging modes is given by How To Enable Shell Script Debugging Mode in Linux. Basically, there are three modes

• -v (short for verbose) – tells the shell to show all lines in a script while they are read, it activates verbose mode.
• -n (short for noexec or no execution) – instructs the shell read all the commands, however doesn’t execute them. This options activates syntax checking mode.
• -x (short for xtrace or execution trace) – tells the shell to display all commands and their arguments on the terminal while they are executed. This option enables shell tracing mode.

Which may be invoked by any of three different ways:

1. Modifying the first line of a shell script: #!/bin/sh option(s)
2. Invoking shell with debugging options: $ /bin/bash option(s) script_name argument1 ... argumentN   
3. Using set shell built-in command in the middle of the script: $ set -option to enable or $ set +option to disable

Network Diagnostics

See also Red Hat 7 Networking Guide 3.3. Configuring IP Networking with nmcli and RED HAT ENTERPRISE LINUX 8 Configuring and managing networking and Network Configuration - Rocky Linux 9 and How I migrated to NetworkManager keyfiles for configuration

# ip address
# ip link
# ip route
# ip -6 route
# firewall-cmd --list-all --zone=... (replace ... with whatever zones you are working with)
# nmcli connection show 
# nmcli connection show --active
# nmcli device status
# nmcli connection show <device>
# sysctl -a | grep -E 'forwarding'

How to use diff and patch

The diff utility can be used to generate a patch file. Basically, the differences are put togetehr with syntax from the patch utility so that the resulting file can be used to turn another copy of the original file into the desired final state file. A good reference article is available online The high level syntax is:

# diff -u file1.html file2.html > patchfile.patch
# patch file1.html patchfile.patch

To reverse the patch, use:

# patch -p0 -R -i patchfile.patch

Explore filesystem utilization

I have installed NCurses Disk Usage (ncdu ) on all KitsNet Linux systems.

Time coordination with chrony

yum install -y chrony                    # to install  
systemctl enable chronyd                 # to enable  
systemctl start chronyd                  # to start  
chronyc tracking                         # To get information about the main time reference
chronyc sources -v                       # equivalent information to the ntpq
ntpdate pool.ntp.org                     # To quickly synchronize a server
timedatectl                              # display time and synchronization status
chronyc -m 'burst 3/3' 'makestep 0.1 3'  # sync time using timesources NOW

If for some reason the clock is thrown off (e.g. after VM migration), you can force an immediate resync, but makestep just allows the clock to be corrected by step instead of slow slew. You should add the burst command to make a new set of measurements.

How to mount a multi-partition disk image in Linux

You can use kpartx or partx to create loop devices for the partitions on the image, and then mount them. So either:

$ sudo kpartx -v -a file.iso
 add map loop0p1 (253:17): 0 8382464 linear 7:1 2048
 $ mount /dev/mapper/loop0p1 ./mnt_point
 ...  do something with the partition  ...
 $ umount ./mnt_point
 $ kpartx -d -v file.iso
 del devmap : loop0p1
 loop deleted : /dev/loop0

or:

$ sudo partx -a -v file.iso
partition: none, disk: file.iso, lower: 0, upper: 0
Trying to use '/dev/loop0' for the loop device
/dev/loop0: partition table type 'dos' detected
range recount: max partno=1, lower=0, upper=0
/dev/loop0: partition #1 added
$ mount /dev/loop0p1 ./mnt_point
...  do something with the partition  ...
$ umount /dev/loop0p1 ./mnt_point
$ sudo partx -d -v /dev/loop0
partition: none, disk: /dev/loop0, lower: 0, upper: 0
/dev/loop0: partition #1 removed

DHCP Lease Renewal

Linux Force DHCP Client (dhclient) to Renew IP Address covers this in greater detail for a number of platforms and environments. For KitsNet, the two methods that work are based on dhclient or nmcli

dhclient

The -r flag explicitly releases the current lease, and once the lease has been released, the client exits. For example, open terminal application and type the command:

$ sudo dhclient -r

Now obtain fresh IP address using DHCP on Linux:

$ sudo dhclient

nmcli

The NetworkManager daemon attempts to make networking configuration and operation as painless and automatic as possible by managing the primary network connection and other network interfaces, like Ethernet, WiFi, and Mobile Broadband devices command-line tool for controlling NetworkManager. The nmcli is a command-line tool for controlling NetworkManager and getting its status. To renew IP address using nmcli for connection named ‘nixcraft_5G’ (use ‘nmcli con‘ command to get list of all connections):

nmcli con
nmcli con down id 'nixcraft_5G'
nmcli con up id 'nixcraft_5G'
# Attempt to update device with changes to the currently active 
# connection made since it was last applied and then try it again
nmcli device reapply 'nixcraft_5G'

GRUB2

the GNU GRUB manual - The GRand Unified Bootloader, version 2.12, originally downloaded from here

Booting into Rescue mode or Emergency Mode

How to Boot into Rescue Mode or Emergency Mode Through Systemd in CentOS/RHEL 7 and 8

Rescue mode is equivalent to single user mode and requires the root password. Rescue mode allows you to repair your system in situations when it is unable to complete a regular booting process. Rescue mode will try to mount all local file systems and start some important system services, but it does not activate network interfaces neither allow multiple users to be logged in. Emergency mode provides the most minimal environment possible and allows you to repair your system even in situations when the system is unable to enter rescue mode. In emergency mode, the system mounts the root file system as read-only, does not attempt to mount any other local file systems, does not activate network interfaces.

1. During bootup, when the GRUB2 menu shows up, press the e key for edit.
2. Add a parameter to the end of the linux16 line. For rescue mode add systemd.unit=rescue.target, for emergency mode add systemd.unit=emergency.target
3. Press Ctrl+x to boot the system with the parameter.

Change Default Boot Selection (or, how to downgrade the OS)

sudo grubby --info=ALL | grep ^kernel               #which versions are on the menu
sudo grubby --grub2 --default-title                 #which is the current default
sudo grubby --set-default "/boot/vmlinuz-5.14.0-362.13.1.el9_3.x86_64"  #Set the default to this by path
sudo grubby --set-default 0                         #reset the default to the top (most recent) choice)

MegaRAID

Copy of MegaRAID SAS Software User Guide (August 2014) from Avago Technologies

CLI

Handy CLI Commands

Task	Command
Show VD reconstruction/migration status	storcli64 /c0 /vall show migrate
Show VD rebuild status	storcli64 /c0 /eall /sall show rebuild or storcli64 -PDRbld -ShowProg -PhysDrv[252:3] -a0
Show PD detailed status	storcli64 -pdlist -aALL
Silence alarm	storcli64 /c0 set alarm=silence
Show VD detailed status	storcli64 -LDInfo -Lall -a0

it seems that either - or / may be used to introduce command switches.

Replacing a failed or failing disk

See REPLACING AN LSI RAID DISK WITH MEGACLI

1. Check is the status of the VD and of the PDs
2. Set the original disk offline if an error has not already cause the controller to set it offline: storcli64 -pdoffline -physdrv[252:3] -a0
3. Mark the failed disk as missing: storcli64 -pdmarkmissing -physdrv[252:3] -aAll
4. Mark the failed disk as prepared for removal: storcli64 -pdprprmv -physdrv[252:3] -a0
5. Replace the faulty disk (in other systems, pdlocate can be helpful)
6. If you don’t use hot spares you will need to add the disk to the array and start the rebuild manually: storcli64 -PdReplaceMissing -PhysDrv[252:3] -Array0 -row0 -a0 and storcli64 -PDRbld -Start -PhysDrv[252:3] -a0

Systemctl

Commands to show and manipulate services

List all the systemd service which are in state=active and sub=running	systemctl list-units --type=service --state=running
List all the systemd serice which are in state=active and sub either running or exited	systemctl list-units --type=service --state=active
To enable and start a service at the same time	systemctl enable --now service
Show enabled units	systemctl list-unit-files --state=enabled

Manual SMTP session

Not exactly an Linux tip, but close enough. I got tired of re-inventing this, so I asked ChatGPT. Additional information is available from SMTP Commands and Response Codes Guide

• Open a terminal and type: telnet <SMTP_server_address> 25
• Once connected, you'll see a response from the server. Typically, it will start with something like220 <SMTP_server_address> ESMTP
• Enter the following commands to send an email:

HELO <smtp_client_domain>
MAIL FROM:<sender_email_address>
RCPT TO:<recipient_email_address>
DATA
Subject: Your subject here
Your email body here.
.
QUIT

• After typing DATA, you'll be prompted to enter the body of your email. After composing your message, end it with a period . on a new line.
• Finally, type QUIT to close the connection.

Monitoring utilities on wort

Aside from systat, iostat, top and glances, wort has some additional utilities installed that can be used to capture information about disk IO performance amongst other things.

• htop is an interactive process viewer
• dstat has been forked to dool because RedHat hijacked the name
  • Try dstat -tcdD total,nvme0n1,sda,sdb,sdc,sdd,sde,sdf  --disk-wait --disk-util 30
• pcp (Performance Co-Pilot) supports storage performance analysis and other things

After installtion of microcode_ctl, initramfs hasn't been re-generated for all the installed kernel packages

Need to run dracut against all installed kernels to make sure they all get the updated microocde.

find  /lib/modules -name "*.dep" | awk -v FS=/ ' {system("sudo dracut -f --kver " $4)}'

Bash functions with parameters instead of alias

See https://askubuntu.com/questions/666130/bash-alias-with-parameters for background. Basically, aliases cannot part out parameters positionally as with a shell script; bash just expands the alias and anything else on the command line is parsed subsequently. So doing clever things like using a parameter twice in the exmpansion is not going to work with just an alias. If a script is not wanted, an alias is the way to go.

Here is one used to do XML dumps of VMs to a file named for the VM:

dxml() { virsh dumpxml $1 > $1.xml ;}

As with bash scripts, the $@ macro can be used to substiture for "rest of line"

Rescue physical disk

Once physical disks start throwing bad blocks consistently, it may be best to take an image of the disk and use that to restore to a new volume. However, traditional usage od the dd command will probably not work, since it will give up at the first read error. The best tool for the job is thus ddrescue . This tool implmenets direct, non-cached reads, multiple passes, selectable retries a logging mechaism, generation of a graphical map, detailed progress and more. Until i can make it more generic, here is the script used to rescue the SD card on the PictureFrame Raspberry Pi; the SD card was 12 years old by the time it failed.

#!/bin/bash

# CONFIGURATION
SOURCE_DEVICE="/dev/sdf"            # Replace with your source device
DEST_IMAGE="/mnt/rpi/pictureframe/rescue-sdf.img"             # Output disk image
LOGFILE="/mnt/rpi/pictureframe/rescue-sdf.log"                # Persistent log file
MAX_RETRIES=5                       # Number of retry passes
BLOCK_SIZE="512"                    # Use small blocks to isolate errors

# COLORS (optional for nice output)
GREEN="\e[32m"
YELLOW="\e[33m"
RED="\e[31m"
RESET="\e[0m"

# Check if running as root
if [ "$EUID" -ne 0 ]; then
  echo -e "${RED}Please run as root.${RESET}"
  exit 1
fi

# Confirm device
echo -e "${YELLOW}About to recover from: ${SOURCE_DEVICE}"
echo -e "Output image: ${DEST_IMAGE}"
echo -e "Log file: ${LOGFILE}${RESET}"
read -rp "Continue? [y/N]: " confirm
[[ "$confirm" != "y" && "$confirm" != "Y" ]] && exit 0

# Initial fast pass: skip errors for speed
echo -e "${GREEN}Starting initial fast copy pass...${RESET}"
ddrescue -f -n -b "$BLOCK_SIZE" "$SOURCE_DEVICE" "$DEST_IMAGE" "$LOGFILE"

# Retry pass: go over bad areas
echo -e "${YELLOW}Retrying failed sectors up to $MAX_RETRIES times...${RESET}"
ddrescue -d -r"$MAX_RETRIES" -b "$BLOCK_SIZE" "$SOURCE_DEVICE" "$DEST_IMAGE" "$LOGFILE"

# Done
echo -e "${GREEN}Recovery completed. Image saved to ${DEST_IMAGE}${RESET}"

The utility ddrescueview may be used to visualize the map generated in the log file from the run.

Visualization of rescued volume with some unrecoverable blocks