CloudLinux Archives ⋆ SysAdminStuff

Yum and curl returning “Illegal instruction (core dumped)” on Xen

When running yum or curl commands on a CentOS 6 XenServer Virtual Machine you might be getting an “Illegal instruction (core dumped)” error returned in your console output.

root@server [~]# yum update
Loaded plugins: fastestmirror, rhnplugin
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base: mirror.us.leaseweb.net
 * cloudlinux-x86_64-server-6: xmlrpc.cln.cloudlinux.com
 * extras: mirror.teklinks.com
 * updates: mirror.teklinks.com
Illegal instruction (core dumped)

The issue is due to Python attempting to execute a CPU opcode advertised as available by the server’s host node virtualization system (XEN), but is not actually supported by the host node’s hardware.

Workaround

Issue can be resolved by running export NSS_DISABLE_HW_AES=1, and then running yum update, to update to newer packages, after which issue should not be happening anymore.

root@server [~]# export NSS_DISABLE_HW_AES=1
root@server [~]# yum -y update

References:

https://www.centos.org/forums/viewtopic.php?t=58002

https://forums.cpanel.net/threads/kvm-xen-hw_aes-detection-issues-yum-update-illegal-instruction.551681/

How to check for, and clean Ebury SSH Rootkit

What is Ebury

Ebury is a SSH Rootkit, and password sniffer which steals SSH login credentials from incoming and outgoing SSH connections, and also steals private SSH keys stored on the infected system.

Ebury can replace SSH binaries, and shared library files used by executables like sshd, wget, curl, …

How to detect Ebury on a system

From version 1.5 Ebury uses Unix domain sockets for interprocess communication.

Malicious process can be seen using netstat -plan | grep atd.

This command should not return any results on clean systems.

root@server [~]# netstat -plan | grep atd 
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Atd should not be listening on any network port or socket.

Ebury will also place additional shared library files, and patch installed libkeyutils file to link to those files.

Files usually found on Ebury infected machines can be one or more of the following:

/lib64/tls/libkeyutils.so.1.5
/lib64/tls/libkeyutils.so.1
/lib64/libns2.so
/lib64/libns5.so
/lib64/libkeyutils.so.1.3
/lib64/libpw3.so

If any of those files exist, check if the files were provided by any rpm using rpm -qf command.

root@server [~]# rpm -qf /lib64/tls/libkeyutils.so.1.5
file /lib64/tls/libkeyutils.so.1.5 is not owned by any package

On clean system command should return the name of the rpm package which installed that file.

root@server [/lib]# rpm -qf libkeyutils.so.1.3
keyutils-libs-1.4-5.el6.i686

Script to check for suspicious files, and processes

Here is a small script that can be used to check for possible Ebury infection.

#!/bin/bash

if [[ `netstat -pan | grep -w atd` ]]; then
    printf "This server appears to have atd process listening on Unix socket or network port\nCheck server for possible Ebury infection\n\n===\n`netstat -pan | grep -w atd`\n===\n\n"
fi

declare -a file_list=("/lib64/tls/libkeyutils.so.1.5" "/lib64/tls/libkeyutils.so.1" "/lib64/libns2.so" "/lib64/libns5.so" "/lib64/libkeyutils.so.1.3" "/lib64/libpw3.so"); 

for file in "${file_list[@]}"; do 
    if [[ -f $file ]]; then
        if [[ `rpm -qf $file` == *'not owned'* ]]; then
            printf "===\nFile $file is not owned by any RPM package, and there is a possible rootkit infection\nCheck server for possible Ebury infection\n===\n"
        fi
    fi
done

Save a script like check4ebury.sh on your system, and run with bash check4ebury.sh

On an infected system, command will return something like this:

[root@server ~]# bash /root/check4ebury.sh
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[root@server ~]#

NOTE: Suspicious processes and fileS might not be visible over SSH connections

Some variants of Ebury will hide suspicious processes and files, if you are checking the system over SSH connection (link).

In cases like that, checks will need to be done over local terminal, remote management console, or through screen session, for all processes and files to be visible.

If you are unable to connect to the server without SSH, install screen with yum -y install screen, and run check4ebury.sh from screen session, to double check for any possible infection.

In some cases when checks are done over SSH, you might be getting different result if you check for processes and files over screen session.

In this example script doesn’t return any signs of infection when run directly from SSH session, but shows running processes and files when run through a screen session.

[root@server ~]# /root/check4ebury.sh
[root@server ~]# screen -dmS ebury bash -c '/root/check4ebury.sh >> test'; sleep 30; cat test
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[root@server ~]#

How to clean Ebury infection

Most important thing to note is, that in case of root level infections like these ones, the only safe way is to do a complete server rebuild after you clean the infection, and make any necessary backups.

In order to clean Ebury infection, you need to kill the processes you found with netstat, remove suspicious library files, and reinstall keyutils-libs* rpm package. It would be also advisable to reinstall SSH packages.

Steps that can be taken to clean the system:

Check the actual keyutils-libs RPM packages you have installed on your system, and download them before removing any files from the system, as it is possible in some cases that some of the infected files are used by yum, curl, wget, and that you won’t be able to do install with yum after removing the files, or use curl, or wget to download RPMs for install.

Kill all SSH connections with killall sshd.
Kill the atd processes listening over Unix socket with kill -9 `lsof -Pt /usr/sbin/atd`.
Remove the suspicious files you found, that were not connected with any rpm package.
Reinstall keyutils-libs and SSH packages, preferably with rpm -ivh --replacefiles --replacepkgs on the predownloaded packages, but in most cases you can use yum:
yum -y reinstall openssh* libssh* keyutils-libs*

After you have reinstalled necessary packages, change your root password, and all SSH keys on the server, and reboot the server to check if suspicious processes and files will return after it.

If possible, always do a full server rebuild, even if no signs of infection exist after reboot.

Avoid cleaning the infection over SSH connection

It would be advisable to kill all SSH connections that exist on the system you are about to clean, so you should be doing it while connected to the server some other way, but if you need to clean the server over SSH, a script like this can be used to accomplish that (you need to replace the files being referenced in the script, with the files you have found on your own system)

#!/bin/bash

killall sshd; 
kill -9 `lsof -Pt /usr/sbin/atd`; 
rm -f /lib64/tls/libkeyutils.so.1.5; 
rm -f /lib64/tls/libkeyutils.so.1; 
yum -y reinstall openssh* libssh* keyutils-libs*; 
service sshd start

References:

https://www.cert-bund.de/ebury-faq

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

https://forums.cpanel.net/threads/ebury-rootkit-backdoor-trojan.396081/

https://documentation.cpanel.net/display/CKB/Determine+Your+System’s+Status

cPanel a lot of accounts quota or bandwidth exceeded.

WHM/cPanel all packages get reapplied, and Upgrade/Downgrade notification is sent for all accounts.

lvemanager-1.0-9.9 has been released in production by CloudLinux yesterday, and it has caused issues on WHM/cPanel servers, by reapplying packages to all accounts.

https://cloudlinux.com/blog/clnews/lve-manager-1099-and-lveutils-1523-released-to-production.php

All accounts that had their quotas and limits modified, without modifying the related packages, have been reassigned to values that are specified in the package, they are assigned to. This will cause quota or bandwidth being exceeded on a lot of accounts on the server, if you haven’t been modifying them properly.

If you have received a following type of mail for all accounts on your server, that account Upgrade/Downgrade was made, then it is due to lvemanager update to lvemanager-1.0-9.9.

Upgrade/Downgrade: %account" from <root@%hostname> for email@domain.tld

If you check your yum update logs, you will see lvemanager-1.0-9.9 has been installed on the server overnight, which is the cause of the issues with the packages being reapplied.

# grep lve /var/log/yum.log | tail -2
Dec 17 05:09:12 Updated: lve-utils-1.5-2.3.el6.cloudlinux.i686
Dec 17 05:09:27 Updated: lvemanager-1.0-9.9.el6.cloudlinux.noarch
#

According to the comments on CloudLinux blog

“You can find their last allocated settings from your last backup –
less /cpb/incremental/accounts/domainname/cp/domainname”

https://cloudlinux.com/blog/clnews/lve-manager-1099-and-lveutils-1523-released-to-production.php#comments

Update by CloudLinux:

CloudLinux has in the meantime issued a statement regarding LVE Manager 1.0-9.9 issues, with some details on how to revert the values to one prior the upgrade.

http://cloudlinux.com/blog/clnews/lve-manager-1099-issues.php

Excerpt from their blog:

If you have cPanel backups enabled, you can get previous limits from backups. Here is an example how to do it:

cd /backup/cpbackup/daily
tar -zxvf username.tar.gz username/cp
tar -zxvf username.tar.gz username/quota

cat username/quota
cat username/cp/username | egrep "MAX|BWLIMIT"

Then set limits manually with WHM –> Modify account.

UPDATED: Dec 18, 07:16 AM UTC

Here are some commands that should help you returning quotas back, if you have cpanel backups enabled.

Get list of users over quota:

cd /backup/cpbackup/daily/
repquota -a | grep "+" | awk '{ print $1 }' > users_overquota

Extract only quota files for them:

for i in `cat users_overquota` ; do echo "tar -zxvf "$i".tar.gz "$i"/quota" ; done | sh

Echo username and quota limit, the output value means Mb:

for i in `cat users_overquota` ; do echo "echo -e '\n'; echo "$i"; cat "$i"/quota " ; done | sh

Now use cpanel’s script to set quotas, based on above output add ‘M’ key for it. Copy-paste username and %value% and run command for each:

/scripts/editquota username %value%M

Another update:

CloudLinux has made another updated about the issue, and have provided a script that could be used to restore those limits if cPanel backups were enabled.

http://cloudlinux.com/blog/clnews/autorestore-package-limits-script-after-lvemanager1099-update.php

Excerpt from their blog:

Due to number of servers affected with custom package limits reset to package defaults we prepared a script that could be used to restore those limits if cPanel backups were enabled. It restores all limits if:

package was not changed since backup time;
limits are the same as package limits;
all package limits in backup are the same as current package limits.

Required files: %backup_location%/files/_etc_quota.conf.gz and %backup_location%/dirs/_var_cpanel.tar.gz . Please, restore them from a day/week before LVE Manager update (before December 16). Better to place them in the same /root/ directory.

Download the following script and make it executable:

wget http://kb.cloudlinux.com/scripts/autorestore.py
chmod 755 autorestore.py

Run it the following way:

./autorestore.py /root/_var_cpanel.tar.gz /root/_etc_quota.conf.gz
The script will back up current user limits to /var/cpanel/users.%timestamp% before changing limits.

Tag: CloudLinux

Yum and curl returning “Illegal instruction (core dumped)” on Xen

Workaround

References:

Like this:

How to check for, and clean Ebury SSH Rootkit

What is Ebury

How to detect Ebury on a system

Script to check for suspicious files, and processes

NOTE: Suspicious processes and fileS might not be visible over SSH connections

How to clean Ebury infection

Avoid cleaning the infection over SSH connection

References:

Like this:

cPanel a lot of accounts quota or bandwidth exceeded.

WHM/cPanel all packages get reapplied, and Upgrade/Downgrade notification is sent for all accounts.

Update by CloudLinux:

Excerpt from their blog:

Another update:

Excerpt from their blog:

Like this:

Workaround

References:

Share this:

Like this:

What is Ebury

How to detect Ebury on a system

Script to check for suspicious files, and processes

NOTE: Suspicious processes and fileS might not be visible over SSH connections

How to clean Ebury infection

Avoid cleaning the infection over SSH connection

References:

Share this:

Like this:

WHM/cPanel all packages get reapplied, and Upgrade/Downgrade notification is sent for all accounts.

Update by CloudLinux:

Excerpt from their blog:

Another update:

Excerpt from their blog:

Share this:

Like this: