Yum and curl returning “Illegal instruction (core dumped)” on Xen

When running yum or curl commands on a CentOS 6 XenServer Virtual Machine you might be getting an “Illegal instruction (core dumped)” error returned in your console output.

root@server [~]# yum update
Loaded plugins: fastestmirror, rhnplugin
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * cloudlinux-x86_64-server-6: xmlrpc.cln.cloudlinux.com
 * extras: mirror.teklinks.com
 * updates: mirror.teklinks.com
Illegal instruction (core dumped)

The issue is due to Python attempting to execute a CPU opcode advertised as available by the server’s host node virtualization system (XEN), but is not actually supported by the host node’s hardware.

Workaround

Issue can be resolved by running export NSS_DISABLE_HW_AES=1, and then running yum update, to update to newer packages, after which issue should not be happening anymore.

root@server [~]# export NSS_DISABLE_HW_AES=1
root@server [~]# yum -y update
References:

https://www.centos.org/forums/viewtopic.php?t=58002

https://forums.cpanel.net/threads/kvm-xen-hw_aes-detection-issues-yum-update-illegal-instruction.551681/

How to check for, and clean Ebury SSH Rootkit

What is Ebury

Ebury is a SSH Rootkit, and password sniffer which steals SSH login credentials from incoming and outgoing SSH connections, and also steals private SSH keys stored on the infected system.

Ebury can replace SSH binaries, and shared library files used by executables like sshd, wget, curl, …

How to detect Ebury on a system

From version 1.5 Ebury uses Unix domain sockets for interprocess communication.

Malicious process can be seen using netstat -plan | grep atd.

This command should not return any results on clean systems.

root@server [~]# netstat -plan | grep atd 
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Atd should not be listening on any network port or socket.

Ebury will also place additional shared library files, and patch installed libkeyutils file to link to those files.

Files usually found on Ebury infected machines can be one or more of the following:

/lib64/tls/libkeyutils.so.1.5
/lib64/tls/libkeyutils.so.1
/lib64/libns2.so
/lib64/libns5.so
/lib64/libkeyutils.so.1.3
/lib64/libpw3.so

If any of those files exist, check if the files were provided by any rpm using rpm -qf command.

root@server [~]# rpm -qf /lib64/tls/libkeyutils.so.1.5
file /lib64/tls/libkeyutils.so.1.5 is not owned by any package

On clean system command should return the name of the rpm package which installed that file.

root@server [/lib]# rpm -qf libkeyutils.so.1.3
keyutils-libs-1.4-5.el6.i686
Script to check for suspicious files, and processes

Here is a small script that can be used to check for possible Ebury infection.

#!/bin/bash

if [[ `netstat -pan | grep -w atd` ]]; then
    printf "This server appears to have atd process listening on Unix socket or network port\nCheck server for possible Ebury infection\n\n===\n`netstat -pan | grep -w atd`\n===\n\n"
fi

declare -a file_list=("/lib64/tls/libkeyutils.so.1.5" "/lib64/tls/libkeyutils.so.1" "/lib64/libns2.so" "/lib64/libns5.so" "/lib64/libkeyutils.so.1.3" "/lib64/libpw3.so"); 

for file in "${file_list[@]}"; do 
    if [[ -f $file ]]; then
        if [[ `rpm -qf $file` == *'not owned'* ]]; then
            printf "===\nFile $file is not owned by any RPM package, and there is a possible rootkit infection\nCheck server for possible Ebury infection\n===\n"
        fi
    fi
done

Save a script like check4ebury.sh on your system, and run with bash check4ebury.sh

On an infected system, command will return something like this:

[root@server ~]# bash /root/check4ebury.sh
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[root@server ~]#

NOTE: Suspicious processes and fileS might not be visible over SSH connections


Some variants of Ebury will hide suspicious processes and files, if you are checking the system over SSH connection (link).

In cases like that, checks will need to be done over local terminal, remote management console, or through screen session, for all processes and files to be visible.

If you are unable to connect to the server without SSH, install screen with yum -y install screen, and run check4ebury.sh from screen session, to double check for any possible infection.

In some cases when checks are done over SSH, you might be getting different result if you check for processes and files over screen session.

In this example script doesn’t return any signs of infection when run directly from SSH session, but shows running processes and files when run through a screen session.

[root@server ~]# /root/check4ebury.sh
[root@server ~]# screen -dmS ebury bash -c '/root/check4ebury.sh >> test'; sleep 30; cat test
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[root@server ~]#
How to clean Ebury infection

Most important thing to note is, that in case of root level infections like these ones, the only safe way is to do a complete server rebuild after you clean the infection, and make any necessary backups.

In order to clean Ebury infection, you need to kill the processes you found with netstat, remove suspicious library files, and reinstall keyutils-libs* rpm package. It would be also advisable to reinstall SSH packages.

Steps that can be taken to clean the system:

Check the actual keyutils-libs RPM packages you have installed on your system, and download them before removing any files from the system, as it is possible in some cases that some of the infected files are used by yum, curl, wget, and that you won’t be able to do install with yum after removing the files, or use curl, or wget to download RPMs for install.

  • Kill all SSH connections with killall sshd.
  • Kill the atd processes listening over Unix socket with kill -9 `lsof -Pt /usr/sbin/atd`.
  • Remove the suspicious files you found, that were not connected with any rpm package.
  • Reinstall keyutils-libs and SSH packages, preferably with rpm -ivh --replacefiles --replacepkgs on the predownloaded packages, but in most cases you can use yum:
    yum -y reinstall openssh* libssh* keyutils-libs*

After you have reinstalled necessary packages, change your root password, and all SSH keys on the server, and reboot the server to check if suspicious processes and files will return after it.

If possible, always do a full server rebuild, even if no signs of infection exist after reboot.

Avoid cleaning the infection over SSH connection

It would be advisable to kill all SSH connections that exist on the system you are about to clean, so you should be doing it while connected to the server some other way, but if you need to clean the server over SSH, a script like this can be used to accomplish that (you need to replace the files being referenced in the script, with the files you have found on your own system)

#!/bin/bash

killall sshd; 
kill -9 `lsof -Pt /usr/sbin/atd`; 
rm -f /lib64/tls/libkeyutils.so.1.5; 
rm -f /lib64/tls/libkeyutils.so.1; 
yum -y reinstall openssh* libssh* keyutils-libs*; 
service sshd start
References:

https://www.cert-bund.de/ebury-faq

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

https://forums.cpanel.net/threads/ebury-rootkit-backdoor-trojan.396081/

https://documentation.cpanel.net/display/CKB/Determine+Your+System’s+Status

CentOS 7 OpenVZ NetworkManager issues.

There is an issue with CentOS 7 containers on OpenVZ nodes, where NetworkManager service breaks the network, and only localhost 127.0.0.1 network interface is coming up.

When you try to check network interfaces, and IP address on the server, you will only see localhost interface.

root@server [/]# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 9059 bytes 1539654 (1.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9059 bytes 1539654 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

When issuing service network restart you will and output like this:

Starting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.
[FAILED]

When you run journalctl -xe you will see errors like this:

-- Unit network.service has begun starting up.
 Feb 16 05:16:40 server network[2262]: Bringing up loopback interface: Could not load file '/etc/sysconfig/network-scripts/ifcfg-Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: [ OK ]
 Feb 16 05:16:40 server network[2262]: Bringing up interface venet0: Error: Connection activation failed: No suitable device founFeb 16 05:16:40 server network[2262]: [FAILED]
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server systemd[1]: network.service: control process exited, code=exited status=1
 Feb 16 05:16:40 server systemd[1]: Failed to start LSB: Bring up/down networking.
 -- Subject: Unit network.service has failed
 -- Defined-By: systemd
 -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
 --
 -- Unit network.service has failed.
 --
 -- The result is failed.
 Feb 16 05:16:40 server systemd[1]: Unit network.service entered failed state.
 Feb 16 05:16:40 server systemd[1]: network.service failed.
 Feb 16 05:16:40 server polkitd[226]: Unregistered Authentication Agent for unix-process:2257:65393 (system bus name :1.37, object

Issue can be resolved by stopping and disabling the NetworkManager service.

root@server [/]# service NetworkManager stop
Redirecting to /bin/systemctl stop NetworkManager.service
root@server [/]# chkconfig NetworkManager off

You can also uninstall NetworkManager completely.

References:

https://lists.openvz.org/pipermail/users/2015-December/006691.html

Openvz / CentOS 7 – Network Manager can break networking

 

CSF chattr and urlget warnings on restart

On most CentOS 7 machines you might be getting this type of warnings when restarting or updating ConfigServer Firewall (CSF), if you are missing certain commands or packages on your server.

*WARNING* Binary location for [CHATTR] [/usr/bin/chattr] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable
*WARNING* Missing or incorrect binary locations will break csf and lfd functionality
*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny

You also might be getting notifications in mail from your Cron Daemon with subject /usr/sbin/csf -u and following message:

*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny

This warnings are showing because of missing packages not being installed on the server, and chattr command doesn’t exist.

[root@server ~]# type chattr
-bash: type: chattr: not found

To resolve CHATTR warning, you need to install e2fsprogs with

yum -y install e2fsprogs

To resolve URLGET warning you need to install perl-libwww-perl with

yum -y install perl-libwww-perl with

If you are getting following warning by mail notifications, or when running csf -u, you are missing perl-LWP-Protocol-https package.

[root@server ~]# csf -u
Oops: Unable to download: Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)

Install perl-LWP-Protocol-https with:

yum -y install perl-LWP-Protocol-https

You might also be getting a following warning, if you don’t have host or unzip commands installed on your server.

*WARNING* Binary location for [UNZIP] [/usr/bin/unzip] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable
*WARNING* Binary location for [HOST] [/usr/bin/host] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable

To resolve HOST warning, you needs to install bind-utils package with

yum -y install bind-utils

To resolve UNZIP warning you need to install unzip package with

yum -y install unzip

tl;dr required packages for CSF

If you are getting any kind of warnings when doing csf -r on CentOS 7 server make sure you have following packages installed.

iptables-services
unzip
bind-utils
perl-libwww-perl
e2fsprogs
perl-LWP-Protocol-https

Needed packages can be installed with following commands:

yum -y install iptables-services
yum -y install unzip
yum -y install bind-utils
yum -y install perl-libwww-perl
yum -y install e2fsprogs
yum -y install perl-LWP-Protocol-https

References:

http://forum.configserver.com/viewtopic.php?t=8864

http://www.liquidweb.com/kb/csf-config-server-firewall-installation/

https://community.centminmod.com/threads/ensure-perl-crypt-ssleay-perl-net-ssleay-are-installed-for-csf-firewall.2183/

systemd conflicts with initscripts during yum update

When running yum update on CentOS 7 OpenVZ container you might get a following error:

--> Processing Conflict: systemd-219-19.el7.x86_64 conflicts initscripts < 9.49.28-1 
--> Finished Dependency Resolution
Error: systemd conflicts with initscripts-9.49.17-1.el7.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Exact version I was running on this server is CentOS Linux release 7.1.1503 (Core), and updates were failing due to them requiring a newer systemd version, which required a newer version, v9.49.28 at least,  of initscripts installed.

Reason why systemd was conflicting with current version of initscripts, instead of just updating them to newer version, was that initscripts were excluded from update in yum.conf file.

[root@server ~]# grep exclude /etc/yum.conf
exclude=initscripts*

There are no default exclude packages in default CentOS/RHEL installs, and it wasn’t put there by me, so question is why was initscripts excluded from updates.

Issues with initscripts v9.49.24 in CentOS 7 OpenVZ containers

It turns out there were some issues with initscripts v9.49.24, which seemed to introduce problems with network connectivity on CentOS 7 OpenVZ VPS machines.

https://vpsboard.com/topic/1908-creating-cpanel-openvz-template-wwhm-setup-wizard-fix/?page=2
https://forum.proxmox.com/threads/centos-7-openvz-template-in-pve-3-4-issues.21808/

I commented out exclude=initscripts* from yum.conf, and checked the currently available version.

[root@server ~]# grep exclude /etc/yum.conf
#exclude=initscripts*
[root@server ~]# yum check-updates initscripts
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.greenmountainaccess.net
 * epel: mirror.nexcess.net
 * extras: mirrors.lga7.us.voxel.net
 * updates: mirror.dattobackup.com
 * webtatic: us-east.repo.webtatic.com

initscripts.x86_64                                               9.49.30-1.el7                                                base

Current version is v9.49.30, so I presumed issues with v9.49.24 were corrected in this version, and proceeded with the update.

Update has completed successfully, and all packages including initscrips were updated to latest version, as was CentOS.

[root@server ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

I haven’t noticed any network or any other issues on the VPS after the upgrade, so it seems any network issues from v9.49.24, don’t exist in v9.49.30.

[root@server ~]# rpm -qa | grep initscripts
initscripts-9.49.30-1.el7.x86_64