VMware rename CentOS 7 NIC names

CentOS 7 virtual machines on VMware will by default use predictable network device naming for network interfaces on the machine, causing their names to be in enoXXXXXXXX format.

This will cause issues when adding 10 or more additional IPs in WHM, as network interface name will be longer than the 15 characters.

Maximum length supported for network interface name on cPanel servers is 15 characters.

When starting ipaliases service, only first 9 additional IPs will be added, and for rest of the IPs error “RTNETLINK answers: Numerical result out of range” will be shown, and IPs will not be shown in ip addr, or ifconfig output.

[[email protected] ~]# /scripts/restartsrv_ipaliases
Waiting for "ipaliases" to stop ...finished.
Waiting for "ipaliases" to start ...finished.
Service Status
Startup Log
 Oct 03 20:29:20 server.example.com ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 server.example.com ipaliases[233833]: Bringing up eno33559296:cp14 RTNETLINK answers: Numerical
result out of range
 Oct 03 20:29:20 server.example.com ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 server.example.com ipaliases[233833]: Routing 204.93.248.69 RTNETLINK answers: Invalid argument
 Oct 03 20:29:20 server.example.com ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 server.example.com ipaliases[233833]: Bringing up eno33559296:cp15 RTNETLINK answers: Numerical
result out of range
 Oct 03 20:29:20 server.example.com ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 server.example.com ipaliases[233833]: Routing 204.93.248.70 RTNETLINK answers: Invalid argument
 Oct 03 20:29:20 server.example.com ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 server.example.com systemd[1]: Started cPanel IP aliases service.
Log Messages
 Oct 3 20:29:20 server ipaliases: [FAILED]
 Oct 3 20:29:20 server ipaliases: Routing x.x.x.x RTNETLINK answers: Invalid argument
 Oct 3 20:29:20 server ipaliases: [FAILED]
 Oct 3 20:29:20 server ipaliases: Bringing up eno33559296:cp15 RTNETLINK answers: Numerical result out of range

To resolve the issues, network devices can be renamed back to old ethX type of naming.

To rename network devices to old names following steps are needed.

  1. Edit /etc/sysconfig/grub
  2. Update GRUB configuration with new kernel parameters
  3. Rename network files
  4. Edit renamed network files
  5. Reboot the server

To rename devices do the following

Edit /etc/sysconfig/grub

Find a line containing “GRUB_CMDLINE_LINUX”, and append “net.ifnames=0 biosdevname=0“ on the line.

File should look something like this.

[[email protected] ~]# cat /etc/sysconfig/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=myvg/rootvol rhgb quiet net.ifnames=0 biosdevname=0"
GRUB_DISABLE_RECOVERY="true"

Update GRUB configuration with new kernel parameters, with following command:

grub2-mkconfig -o /boot/grub2/grub.cfg

Rename enoXXXXXXXX network files of all interfaces to ethX network file.

For example:

mv /etc/sysconfig/network-scripts/ifcfg-eno16777984 /etc/sysconfig/network-scripts/ifcfg-eth0
mv /etc/sysconfig/network-scripts/ifcfg-eno33557248 /etc/sysconfig/network-scripts/ifcfg-eth1

This will rename file ifcfg-eno16777984, to ifcfg-eth0, renaming interface eno16777984 to eth0, and will rename file ifcfg-eno33557248, to ifcfg-eth1, renaming interface eno33557248, to eth1.

Edit new ethX network files.

Replace value of both NAME and DEVICE field with new ethX names.

File should look something like this.

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=static
NAME=eth0
UUID=b02f4abf-f6da-4ad4-b800-4abf4fe1d50d
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=yes
IPADDR=x.x.x.x
NETMASK=255.255.255.0
GATEWAY=x.x.x.1

Reboot the server, and you should now see network interfaces using old CentOS 6 style names.

Additional changes for cPanel servers

Change public network interface in Basic cPanel & WHM Setup.

Go to Home »Server Configuration »Basic cPanel & WHM Setup and change public interface from old enoXXXXXXXX to new ethX name.

Change public interface to new name
Change public interface to new name

Restart ipaliases service with /scripts/restartsrv_ipaliases.

Reset root password on CentOS 6 machine

If you have forgotten root user password on your CentOS 6 machine, you can reset the password to new value, without knowing the old one, by booting the machine to single user mode, and resetting the password with passwd command.

Boot to single user mode

During the CentOS boot process, you will be presented with a countdown before CentOS boot process actually begins to load the OS.

Press any key to enter the GNU GRUB menu.

CentOS 6 boot countdown
CentOS boot countdown

On the GNU GRUB menu highlight the kernel you want to boot, and press ‘e’ key to edit the kernel commands before booting.

CentOS 6 kernel select
CentOS 6 kernel select

Once you have entered the kernel edit mode, find the line beginning with kernel, and highlight it, and press ‘e’ key to edit it.

Edit kernel line
Edit kernel line

On the end of the line add the word ‘single’, with white space before ‘single’, and press ENTER to accept the change.

Add single to then end of the kernel line
Add single to then end of the kernel line

Boot the machine with the edited kernel argument, and you will be logged in as root in single user mode.

Now just issue passwd command, and enter the new password two times, when asked.

Change password with passwd command
Change password with passwd command

Once you have changed the password, reboot the machine, and log to it with your new root password.

How to clear disk space on cPanel server

By default drives will come with 5% of all filesystems allocated as reserved disk space allocated for privileged users, and not shown as available space.

Since drives in use today tend to be large, reserved block percentage can be lowered to 1%, or specified to specific number of block.

To set the reserved space to 2500 blocks “tune2fs -r 2500” can be used.

tune2fs -r 2500 /dev/sda5

To set reserved space to 1% of disk size “tune2fs -m 1” can be used.

tune2fs -m 1 /dev/sda5

Files that can be cleared by main partitions/folders, to reduce disk usage on cPanel servers:

Reduce /home usage

Several files and folders can be truncated or removed on /home.

When EasyApache is run, it will leave file behind, that were used for Apache/PHP build, that can be removed if space is needed.

EasyApache files can be removed with following command.

rm -rfv /home/cpeasyapache

cPanel FileManager can leave temporary files, that were created during user uploads.
These can be removed with following command:

rm -fv /home/*/tmp/Cpanel_*

If you were moving any accounts to the server with WHM Transfer Tool, temporary account migration files can be left on drive.

These can be removed with following command:

rm -rvf /home*/cpanelpkgrestore.TMP*

Disk space can be recovered by deleting Softaculous and Fantastico backups from user folders, if they are used.

rm -fv /home*/*/.softaculous/backups/*
rm -rfv /home/*/fantastico_backups

If you were making cpmove file manually, they will by default be created inside /home.
You can clean any leftover cpmove files with following command:

rm -rvf /home/cpmove-*

Often large portions of disk space can be used up by large error_log files inside account home folders.

You can empty all error_log files to 0 bytes usage with following command:

find /home/ -name error_log -type f -print -exec truncate --size 0 "{}" \;

If users have large number of account backups in their home folders, those can use up a lot of space.

Accounts backups can be removed from user folders with following command:

for user in `/bin/ls -A /var/cpanel/users` ; do rm -fv /home/$user/backup-*$user.tar.gz ; done
Summary of all used commands, for clearing /home
#Emtpy all error logs
find /home/ -name error_log -type f -print -exec truncate --size 0 "{}" \;
#Remove EasyApache files
rm -rfv /home/cpeasyapache
#Remove Softaculous backups
rm -fv /home*/*/.softaculous/backups/*
#Remove account backups
for user in `/bin/ls -A /var/cpanel/users` ; do rm -fv /home/$user/backup-*$user.tar.gz ; done
#Remove Fantastico backups
rm -rfv /home/*/fantastico_backups
#Remove temporary cPanel files
rm -fv /home/*/tmp/Cpanel_*
#Remove any cpmove files
rm -rvf /home/cpmove-*
#Remove temporary account migration files
rm -rvf /home*/cpanelpkgrestore.TMP*
Reduce /var usage

Disk space in /var can be cleared by deleting archived logs, which will usually end with .gz, or contain year inside their name, like such as “maillog-20161113”.

Archived logs can be cleared with following commands:

rm -fv /var/log/*.gz
rm -fv /var/log/*201*

Disk space in /var can get also get used up by core dump files inside /var/spool/abrt/ directory, which get created in cases of kernel panic.

These file can be cleared up with following command:

rm rfv /var/spool/abrt/*

Check the size of exim stats database, which can sometimes take several gigabytes in size, depending on settings:

du -sh /var/lib/mysql/eximstats/

In case eximstats database is large, consider emptying the database, and changing retention settings.

How to clear or disable eximstats on cPanel

Reduce /usr usage

Disk space in /usr can be cleared by removing cPanel and Apache archived logs, or old installation files of Apache, and if installed, maldet.

Every time you rebuild Apache with EasyApache, old installation files will be moved to “apache.backup*” directory.

Remove old Apache files with following command:

rm -rfv /usr/local/apache.backup*

Similar thing happens with maldet, if it is installed, on updates, old installation will be moved to “maldet.bk*” folder.

Remove old maldet files with following command:

rm -rfv /usr/local/maldet.bk*

Clear disk space by removing archived cPanel logs:

rm -fv /usr/local/cpanel/logs/archive/*.gz

Remove archived Apache logs:

rm -fv /usr/local/apache/logs/*.gz
rm -fv /usr/local/apache/logs/archive/*.gz

Although not often, sometimes maldet logs can use up a lot of space.
Remove maldet logs:

rm -fv /usr/local/maldetect/logs/*
Summary of all used commands, for clearing /usr
#Remove old Apache files
rm -rfv /usr/local/apache.backup*
#Remove old maldet files
rm -rfv /usr/local/maldet.bk*
#Remove maldet logs
rm -fv /usr/local/maldetect/logs/*
#Remove archived cPanel logs
rm -fv /usr/local/cpanel/logs/archive/*.gz
#Remove archived Apache logs
rm -fv /usr/local/apache/logs/*.gz
rm -fv /usr/local/apache/logs/archive/*.gz
References:

How to Free Up Disk Space on a cPanel Server

11 Ways to Free Up Disk Space on a cPanel Server

Yum and curl returning “Illegal instruction (core dumped)” on Xen

When running yum or curl commands on a CentOS 6 XenServer Virtual Machine you might be getting an “Illegal instruction (core dumped)” error returned in your console output.

[email protected] [~]# yum update
Loaded plugins: fastestmirror, rhnplugin
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * cloudlinux-x86_64-server-6: xmlrpc.cln.cloudlinux.com
 * extras: mirror.teklinks.com
 * updates: mirror.teklinks.com
Illegal instruction (core dumped)

The issue is due to Python attempting to execute a CPU opcode advertised as available by the server’s host node virtualization system (XEN), but is not actually supported by the host node’s hardware.

Workaround

Issue can be resolved by running export NSS_DISABLE_HW_AES=1, and then running yum update, to update to newer packages, after which issue should not be happening anymore.

[email protected] [~]# export NSS_DISABLE_HW_AES=1
[email protected] [~]# yum -y update
References:

https://www.centos.org/forums/viewtopic.php?t=58002

https://forums.cpanel.net/threads/kvm-xen-hw_aes-detection-issues-yum-update-illegal-instruction.551681/

How to check for, and clean Ebury SSH Rootkit

What is Ebury

Ebury is a SSH Rootkit, and password sniffer which steals SSH login credentials from incoming and outgoing SSH connections, and also steals private SSH keys stored on the infected system.

Ebury can replace SSH binaries, and shared library files used by executables like sshd, wget, curl, …

How to detect Ebury on a system

From version 1.5 Ebury uses Unix domain sockets for interprocess communication.

Malicous process can be seen using netstat -plan | grep atd.

This command should not return any results on clean systems.

[email protected] [~]# netstat -plan | grep atd 
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Atd should not be listening on any network port or socket.

Ebury will also place additional shared library files, and patch installed libkeyutils file to link to those files.

Files usually found on Ebury infected machines can be one or more of the following:

/lib64/tls/libkeyutils.so.1.5
/lib64/tls/libkeyutils.so.1
/lib64/libns2.so
/lib64/libns5.so
/lib64/libkeyutils.so.1.3

If any of those files exist, check if the files were provided by any rpm using rpm -qf command.

[email protected] [~]# rpm -qf /lib64/tls/libkeyutils.so.1.5
file /lib64/tls/libkeyutils.so.1.5 is not owned by any package

On clean system command should return the name of the rpm package which installed that file.

[email protected] [/lib]# rpm -qf libkeyutils.so.1.3
keyutils-libs-1.4-5.el6.i686
Script to check for suspicious files, and processes

Here is a small script that can be used to check for possible Ebury infection.

#!/bin/bash

if [[ `netstat -pan | grep -w atd` ]]; then
    printf "This server appears to have atd process listening on Unix socket or network port\nCheck server for possible Ebury infection\n\n===\n`netstat -pan | grep -w atd`\n===\n\n"
fi

declare -a file_list=("/lib64/tls/libkeyutils.so.1.5" "/lib64/tls/libkeyutils.so.1" "/lib64/libns2.so" "/lib64/libns5.so" "/lib64/libkeyutils.so.1.3"); 

for file in "${file_list[@]}"; do 
    if [[ -f $file ]]; then
        if [[ `rpm -qf $file` == *'not owned'* ]]; then
            printf "===\nFile $file is not owned by any RPM package, and there is a possible rootkit infection\nCheck server for possible Ebury infection\n===\n"
        fi
    fi
done

Save a script like check4ebury.sh on your system, and run with bash check4ebury.sh

On an infected system, command will return something like this:

[[email protected] ~]# bash /root/check4ebury.sh
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[[email protected] ~]#

NOTE: Suspicious processes and fileS might not be visible over SSH connections


Some variants of Ebury will hide suspicious processes and files, if you are checking the system over SSH connection (link).

In cases like that, checks will need to be done over local terminal, remote management console, or through screen session, for all processes and files to be visible.

If you are unable to connect to the server without SSH, install screen with yum -y install screen, and run check4ebury.sh from screen session, to double check for any possible infection.

In some cases when checks are done over SSH, you might be getting different result if you check for processes and files over screen session.

[[email protected] ~]# /root/check4ebury.sh
[[email protected] ~]# screen -dmS ebury bash -c '/root/check4ebury.sh >> test'; sleep 30; cat test
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[[email protected] ~]#
How to clean Ebury infection

Most important thing to note is, that in case of root level infections like these ones, the only safe way is to do a complete server rebuild after you clean the infection, and make any necessary backups.

In order to clean Ebury infection, you need to kill the processes you found with netstat, remove suspicious library files, and reinstall keyutils-libs* rpm package. It would be also advisable to reinstall SSH packages.

Steps that can be taken to clean the system:

Check the actual keyutils-libs RPM packages you have installed on your system, and download them before removing any files from the system, as it is possible in some cases that some of the infected files are used by yum, curl, wget, and that you won’t be able to do install with yum after removing the files, or use curl, or wget to download RPMs for install.

  • Kill all SSH connections with killall sshd.
  • Kill the atd processes listening over Unix socket with kill -9 `lsof -Pt /usr/sbin/atd`.
  • Remove the suspicious files you found, that were not connected with any rpm package.
  • Reinstall keyutils-libs and SSH packages, preferably with rpm -ivh --replacefiles --replacepkgs on the predownloaded packages, but in most cases you can use yum:
    yum -y reinstall openssh* libssh* keyutils-libs*

After you have reinstalled necessary packages, change your root password, and all SSH keys on the server, and reboot the server to check if suspicious processes and files will return after it.

If possible, always do a full server rebuild, even if no signs of infection exist after reboot.

Avoid cleaning the infection over SSH connection

It would be advisable to kill all SSH connections that exist on the system you are about to clean, so you should be doing it while connected to the server some other way, but if you need to clean the server over SSH, a script like this can be used to accomplish that (you need to replace the files being referenced in the script, with the files you have found on your own system)

#!/bin/bash

killall sshd; 
kill -9 `lsof -Pt /usr/sbin/atd`; 
rm -f /lib64/tls/libkeyutils.so.1.5; 
rm -f /lib64/tls/libkeyutils.so.1; 
yum -y reinstall openssh* libssh* keyutils-libs*; 
service sshd start
References:

https://www.cert-bund.de/ebury-faq

http://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

https://forums.cpanel.net/threads/ebury-rootkit-backdoor-trojan.396081/

https://documentation.cpanel.net/display/CKB/Determine+Your+System’s+Status