How to check for, and clean Ebury SSH Rootkit

What is Ebury

Ebury is a SSH Rootkit, and password sniffer which steals SSH login credentials from incoming and outgoing SSH connections, and also steals private SSH keys stored on the infected system.

Ebury can replace SSH binaries, and shared library files used by executables like sshd, wget, curl, …

How to detect Ebury on a system

From version 1.5 Ebury uses Unix domain sockets for interprocess communication.

Malicious process can be seen using netstat -plan | grep atd.

This command should not return any results on clean systems.

[email protected] [~]# netstat -plan | grep atd 
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Atd should not be listening on any network port or socket.

Ebury will also place additional shared library files, and patch installed libkeyutils file to link to those files.

Files usually found on Ebury infected machines can be one or more of the following:

/lib64/tls/libkeyutils.so.1.5
/lib64/tls/libkeyutils.so.1
/lib64/libns2.so
/lib64/libns5.so
/lib64/libkeyutils.so.1.3
/lib64/libpw3.so

If any of those files exist, check if the files were provided by any rpm using rpm -qf command.

[email protected] [~]# rpm -qf /lib64/tls/libkeyutils.so.1.5
file /lib64/tls/libkeyutils.so.1.5 is not owned by any package

On clean system command should return the name of the rpm package which installed that file.

[email protected] [/lib]# rpm -qf libkeyutils.so.1.3
keyutils-libs-1.4-5.el6.i686
Script to check for suspicious files, and processes

Here is a small script that can be used to check for possible Ebury infection.

#!/bin/bash

if [[ `netstat -pan | grep -w atd` ]]; then
    printf "This server appears to have atd process listening on Unix socket or network port\nCheck server for possible Ebury infection\n\n===\n`netstat -pan | grep -w atd`\n===\n\n"
fi

declare -a file_list=("/lib64/tls/libkeyutils.so.1.5" "/lib64/tls/libkeyutils.so.1" "/lib64/libns2.so" "/lib64/libns5.so" "/lib64/libkeyutils.so.1.3" "/lib64/libpw3.so"); 

for file in "${file_list[@]}"; do 
    if [[ -f $file ]]; then
        if [[ `rpm -qf $file` == *'not owned'* ]]; then
            printf "===\nFile $file is not owned by any RPM package, and there is a possible rootkit infection\nCheck server for possible Ebury infection\n===\n"
        fi
    fi
done

Save a script like check4ebury.sh on your system, and run with bash check4ebury.sh

On an infected system, command will return something like this:

[[email protected] ~]# bash /root/check4ebury.sh
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[[email protected] ~]#

NOTE: Suspicious processes and fileS might not be visible over SSH connections


Some variants of Ebury will hide suspicious processes and files, if you are checking the system over SSH connection (link).

In cases like that, checks will need to be done over local terminal, remote management console, or through screen session, for all processes and files to be visible.

If you are unable to connect to the server without SSH, install screen with yum -y install screen, and run check4ebury.sh from screen session, to double check for any possible infection.

In some cases when checks are done over SSH, you might be getting different result if you check for processes and files over screen session.

In this example script doesn’t return any signs of infection when run directly from SSH session, but shows running processes and files when run through a screen session.

[[email protected] ~]# /root/check4ebury.sh
[[email protected] ~]# screen -dmS ebury bash -c '/root/check4ebury.sh >> test'; sleep 30; cat test
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury
infection

===
unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k
===

===
File /lib64/tls/libkeyutils.so.1.5 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
===
File /lib64/tls/libkeyutils.so.1 is not owned by any RPM package, and there is a possible rootkit infection
Check
server for possible Ebury infection
===
[[email protected] ~]#
How to clean Ebury infection

Most important thing to note is, that in case of root level infections like these ones, the only safe way is to do a complete server rebuild after you clean the infection, and make any necessary backups.

In order to clean Ebury infection, you need to kill the processes you found with netstat, remove suspicious library files, and reinstall keyutils-libs* rpm package. It would be also advisable to reinstall SSH packages.

Steps that can be taken to clean the system:

Check the actual keyutils-libs RPM packages you have installed on your system, and download them before removing any files from the system, as it is possible in some cases that some of the infected files are used by yum, curl, wget, and that you won’t be able to do install with yum after removing the files, or use curl, or wget to download RPMs for install.

  • Kill all SSH connections with killall sshd.
  • Kill the atd processes listening over Unix socket with kill -9 `lsof -Pt /usr/sbin/atd`.
  • Remove the suspicious files you found, that were not connected with any rpm package.
  • Reinstall keyutils-libs and SSH packages, preferably with rpm -ivh --replacefiles --replacepkgs on the predownloaded packages, but in most cases you can use yum:
    yum -y reinstall openssh* libssh* keyutils-libs*

After you have reinstalled necessary packages, change your root password, and all SSH keys on the server, and reboot the server to check if suspicious processes and files will return after it.

If possible, always do a full server rebuild, even if no signs of infection exist after reboot.

Avoid cleaning the infection over SSH connection

It would be advisable to kill all SSH connections that exist on the system you are about to clean, so you should be doing it while connected to the server some other way, but if you need to clean the server over SSH, a script like this can be used to accomplish that (you need to replace the files being referenced in the script, with the files you have found on your own system)

#!/bin/bash

killall sshd; 
kill -9 `lsof -Pt /usr/sbin/atd`; 
rm -f /lib64/tls/libkeyutils.so.1.5; 
rm -f /lib64/tls/libkeyutils.so.1; 
yum -y reinstall openssh* libssh* keyutils-libs*; 
service sshd start
References:

https://www.cert-bund.de/ebury-faq

https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/

https://forums.cpanel.net/threads/ebury-rootkit-backdoor-trojan.396081/

https://documentation.cpanel.net/display/CKB/Determine+Your+System’s+Status

Fantastico returns error 404 when accessed from cPanel

When going to Fantastico De Luxe link in you cPanel under Paper Lantern theme, or some other theme you might get HTTP error 404 cPanel page shown, saying that requested page was not found.

 

Fantastico 404 page
Fantastico 404 page
Check your theme folder content for fantastico symlink

404 page will most likely be caused by the affected theme not having, a fantastico symlink, or symlink being misdirected.

Go to the folder of the theme, which in case of Paper Lantern theme will be /usr/local/cpanel/base/frontend/paper_lantern and check for file with name fantastico.

[email protected] [/usr/local/cpanel/base/frontend/paper_lantern]# ls -l | grep fantastico
[email protected] [/usr/local/cpanel/base/frontend/paper_lantern]#

Folder should have symlink in it, with name fantastico, pointing to /usr/local/cpanel/3rdparty/fantastico/, if the symlink is missing create one with ln -s command, also check if the file to which symlink is pointing is an existing file.

[email protected] [/usr/local/cpanel/base/frontend/paper_lantern]# ln -s /usr/local/cpanel/3rdparty/fantastico/ fantastico
[email protected] [/usr/local/cpanel/base/frontend/paper_lantern]# ls -l | grep fantastico
lrwxrwxrwx 1 root root 37 Mar 3 18:02 fantastico -> /usr/local/cpanel/3rdparty/fantastico/
[email protected] [/usr/local/cpanel/base/frontend/paper_lantern]#

After creating a needed symlink Fantastico De Luxe page should work as expected.

Reference:

http://legacy.netenberg.com/forum/index.php?topic=1942.0

Reverse DNS does not match SMTP Banner

UPDATE: WHM/cPanel removed support for in version 11.50, so changes below are not valid for versions 11.50+

https://documentation.cpanel.net/display/ALD/11.50+Release+Notes#id-11.50ReleaseNotes-/etc/mail_reverse_dnsremoved

 

If you make an SMTP test on http://mxtoolbox.com you might be getting a following error shown in the test results “Reverse DNS does not match SMTP Banner”.

This error is showing because your SMTP greeting message is not matching the PTR records for the IP of the SMTP server used in test.

Following files need to be used and configured properly, for SMTP banner to match reverse DNS records.

/etc/mailhelo
/etc/mailips
/etc/mail_reverse_dns
/etc/exim.conf

Configure Exim to use mailhelo and mailips file

Go to WHM to Home »Service Configuration »Exim Configuration Manager and in Basic Editor on Domains and IPs tab set following settings:

Send mail from account’s dedicated IP address: OFF
Reference /etc/mailhelo for outgoing SMTP HELO: ON
Reference /etc/mailips for outgoing SMTP connections: ON

Configure necessary values in configuration files

Edit or create  /etc/mail_reverse_dns file and set the following in it for needed IPs.

x.x.x.x: rdns of the IP x.x.x.x
y.y.y.y: rdns of IP y.y.y.y

Edit or create /etc/mailhelo file and set following in it for the domains that you want to setup SMTP banner for.

example.com: reverse dns of the IP used for example.com domain
*: default SMTP HELO for unconfigured domains

Edit or create /etc/mailips file and set following in it:

example.com: x.x.x.x #x.x.x.x is the IP used for outgoing mail for domain example.com
*: y.y.y.y #y.y.y.y is the default IP that will be used for unconfigured domains

Configure exim.conf to use correct SMTP Banner

Following values need to be configured in exim.conf for SMTP Banner to be set to rDNS values set in /etc/mail_reverse_dns.

smtp_active_hostname
message_id_header_domain
smtp_banner

Be default only smtp_banner is set on cPanel servers, and it has a different value then needed.

[email protected] [~]# egrep "smtp_active_hostname|message_id_header_domain|smtp_banner" /etc/exim.conf
smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \

smtp_banner will probably look like this on your cPanel server.

"${primary_hostname} ESMTP Exim ${version_number}  \#${compile_number} ${tod_full} \n   We do not authorize the use of this system to transport unsolicited, \n   and/or bulk e-mail."
Configure values in exim.conf over shell

Locate the line smtp_banner and change its value so it looks like following:

smtp_banner = "${smtp_active_hostname} ESMTP Exim ${version_number} \"

Add smtp_active_hostname line value to exim.conf to look line following:

smtp_active_hostname = ${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}

Add message_id_header_domain line to exim.conf to look like following:

message_id_header_domain = $smtp_active_hostname

In the end related values in exim.conf should look like this:

[email protected] [~]# egrep "smtp_active_hostname|message_id_header_domain|smtp_banner" /etc/exim.conf
smtp_banner = "${smtp_active_hostname} ESMTP Exim ${version_number} \"
smtp_active_hostname = ${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}
message_id_header_domain = $smtp_active_hostname

Restart exim with /scripts/restartsrv_exim and SMTP tests should now pass without the SMTP banner warning.

Configure values in exim.conf over WHM

In your WHM go to Home »Service Configuration »Exim Configuration Manager and go to Advanced Editor.

Search for the smtp_banner field and change default value to:

"${smtp_active_hostname} ESMTP Exim ${version_number} \"

 

Edit smtp_banner in WHM
Edit smtp_banner in WHM

Find the “Add additional configuration setting” button and add two new configuration settings smtp_active_hostname and message_id_header_domain.

additional configuration settings
Add additional configuration setting in WHM

For smtp_active_hostname set the following value:

${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}

For message_id_header_domain set the following value:

$smtp_active_hostname

References:

https://forums.cpanel.net/threads/easy-fix-your-smtp-banner-smtp-greeting-and-reverse-dns-for-dedicated-ips.391311/

https://forums.cpanel.net/threads/exim-banner-mail-headers-and-resellers-with-own-ip.100697/

How to Install ionCube Loader 5 on cPanel/WHM server

At this moment cPanel does not support ionCube Loader 5 on WHM/cPanel servers, which will cause issues for client running files made with ionCube v9.

cPanel currently installs ionCube PHP Loader v4.7.5, when you install PHP using cPanel EasyApache utility.

Your standard php -v output on cPanel server might look something like this.

[email protected] [~]# php -v
PHP 5.4.45 (cli) (built: Dec 14 2015 17:18:43)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd.

If you want to install, or upgrade current ionCube PHP Loader on your server to version 5, you can do so manually by downloading the latest version from http://www.ioncube.com/loaders.php, and editing your global or account custom php.ini file.

How to install ionCube Loader manually on cPanel or standard Linux servers with no control panel.

Download the latest loader to your server from http://www.ioncube.com/loaders.php.

If you are running 32 bit server used the following command:

wget http://downloads3.ioncube.com/loader_downloads/ioncube_loaders_lin_x86.tar.gz

If you are running 64 bit server use the following command:

wget http://downloads3.ioncube.com/loader_downloads/ioncube_loaders_lin_x86-64.tar.gz

Unpack the package:

On 32 bit server:

tar xvzf ioncube_loaders_lin_x86.tar.gz

On 64 bit server:

tar xvzf ioncube_loaders_lin_x86-64.tar.gz

It will create ioncube folder in the same directory where you downloaded the file.

If you need to install ionCube Loader for PHP 5.4  you will use ioncube_loader_lin_5.4* files, and if you need to install it for PHP 5.5 you will use ioncube_loader_lin_5.5* files, and similar for other PHP versions.

[email protected] [~]# ls ioncube
./                             ioncube_loader_lin_4.4_ts.so*  ioncube_loader_lin_5.3.so*     ioncube_loader_lin_5.6_ts.so*
../                            ioncube_loader_lin_5.0.so*     ioncube_loader_lin_5.3_ts.so*  LICENSE.txt
ioncube_loader_lin_4.1.so*     ioncube_loader_lin_5.0_ts.so*  ioncube_loader_lin_5.4.so*     loader-wizard.php
ioncube_loader_lin_4.2.so*     ioncube_loader_lin_5.1.so*     ioncube_loader_lin_5.4_ts.so*  README.txt
ioncube_loader_lin_4.3.so*     ioncube_loader_lin_5.1_ts.so*  ioncube_loader_lin_5.5.so*     USER-GUIDE.md
ioncube_loader_lin_4.3_ts.so*  ioncube_loader_lin_5.2.so*     ioncube_loader_lin_5.5_ts.so*  USER-GUIDE.txt
ioncube_loader_lin_4.4.so*     ioncube_loader_lin_5.2_ts.so*  ioncube_loader_lin_5.6.so*
[email protected] [~]#

To replace the ionCube Loader for all the users on the server, you can just replace the extension file defined in the global php.ini at /usr/local/lib/php.ini

[email protected] [~]# grep zend_extension /usr/local/lib/php.ini
zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.4.so"
[email protected] [~]#

You can rename the original file, for restore purposes, if something goes wrong, and place the new version file from ioncube folder that was created with package extraction.

mv /usr/local/IonCube/ioncube_loader_lin_5.4_ts.so /usr/local/IonCube/ioncube_loader_lin_5.4_ts.so_cporig
mv /usr/local/IonCube/ioncube_loader_lin_5.4.so /usr/local/IonCube/ioncube_loader_lin_5.4.so_cporig
cp ioncube/ioncube_loader_lin_5.4.so /usr/local/IonCube/
cp ioncube/ioncube_loader_lin_5.4_ts.so /usr/local/IonCube/

You can confirm the new version of ionCube Loader with php -v, and look for “ionCube PHP Loader” part of the output.

[email protected] [~]# php -v
PHP 5.4.45 (cli) (built: Dec 14 2015 17:18:43)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.19, Copyright (c) 2002-2015, by ionCube Ltd.
[email protected] [~]# php -v | grep "ionCube PHP Loader"
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.19, Copyright (c) 2002-2015, by ionCube Ltd.
[email protected] [~]#

Upgrade to ionCube Loader 5 on one cPanel account.

If you wanted to change the ion Cube Loader version for only one cPanel account, you can place the ioncube_loader_lin* files for the PHP version used by your client to some custom folder, and define zend_extension values inside a custom php.ini folder on a specific cPanel account.

tl;dr

ionCube Loader can be installed manually on any Linux server with following steps:

  1. Download latest version of ionCube Loaders from http://www.ioncube.com/loaders.php
  2. Unpack the downloaded package
  3. move the ioncube_loader_lin_* files for your PHP version to your extension folder.
  4. Point to the corresponding file in your php.ini file, example
    zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.4.so"

 

ConfigServer Internal Server Error 500, after cPanel update: fixed

This is a repost of a post from an old blog, made on June 29, 2013, that used to be on:

http://adminramble.com/configserver-internal-server-error-500-cpanel-update-fixed/

Original post:

After the WHM/cPanel update to 11.38 you might get the an error like this in your browser:

Internal Server Error
500
No response from subprocess (/usr/local/cpanel/whostmgr/docroot/cgi/addon_csf.cgi): subprocess exited with status 2

And something like this in /usr/local/cpanel/logs/error_log:

BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/JSON.pm line 15.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template.pm line 53.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template.pm line 53.
Compilation failed in require at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
Compilation failed in require at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
BEGIN failed–compilation aborted at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
For help, please send mail to this site’s webmaster, giving this error message and the time and date of the error.

when going to ConfigServer cPanel plugins in WHM like:

ConfigServer Explorer
ConfigServer Mail Manage
ConfigServer Mail Queues
ConfigServer ModSecurity Control
ConfigServer Security & Firewall

This happens if  the installed ConfigServer scripts on a cPanel/WHM server don’t get updated.

The solution for this error is simple.

To resolve this error simply SSH into your server as a root user and run the following command from command line:

curl -s configserver.com/free/csupdate | perl

This script will update: cmm, cmc, cmq, cse, csf, cxs, msinstall, msfe

You can see the ConfigServer blog post about the update here.