cdp_io processes stuck, causing high load or filling up disk space

R1Soft CDP Agent can sometimes have issues with its backup agent dying, and leaving its cdp_io processes running.

This can cause issues with high load on the server, disk space on the server being filled up by CDP agent, or backup policies on CDP server failing, as they won’t be able to start due to stuck cdp_io processes.

If you do ps auxf | grep cdp you might see a lot of processes like this.

root      587475  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:20 cdp_io/1/0

root      587481  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:14 cdp_io/1/1

root      587482  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/2

root      587483  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/3

root      587487  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/4

root      587488  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/5

root      587490  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/6

root      587557  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/7

root      587603  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/8

root      587604  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/9

root      587605  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/10

root      587607  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/11

root      587613  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/12

root      587691  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/13

root      587699  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:12 cdp_io/1/14

root      587710  0.0  0.3 2136156 128260 ?      Ss   Dec19   0:13 cdp_io/1/15

Trying to kill those processes with kill -9 will yield no results, and processes will be left running.

Doing a Google search on similar issues returns two possible options to kill those processes, and resolve any possible issues caused by them.

http://wiki.r1soft.com/display/CDP/Solved+Issue+of+CDP+Agent+Dying+During+Backup+Process

http://www.webhostingtalk.com/showthread.php?t=1331701

Three possible options to kill the stuck processes are:

  1. Reboot of the device where cdp_io processes are stuck.
  2. Doing a complete remove, and reinstall of CDP agent.
    NOTE: yum reinstall will not work, you need to remove the packages completely, and install them again.
  3. Doing an update of CDP agent to new version, if newer version is available.

On RedHat-compatible distributions, update to a new version can be made by issuing yum update serverbackup*, or in some cases yum update r1soft* command in shell of the device with CDP agent.

Remove and reinstall CDP agent on RedHat-compatible distributions.

To completely remove, and install CDP agent again, first check what CDP agent packages you have installed on your device.

Check currently installed version.

Currently installed CDP agent packages can be checked with rpm -qa | egrep "serverbackup|r1soft", which will show currently installed serverbackup packages, or r1soft packages, depending on which naming version of the packages you have installed on your machine.

[~]# rpm -qa | egrep "serverbackup|r1soft"
serverbackup-enterprise-agent-5.10.1-8.x86_64
serverbackup-async-agent-2-6-5.12.0-21.x86_64
serverbackup-agent-5.12.0-21.x86_64
serverbackup-setup-5.12.0-21.x86_64

Remove and reinstall CDP agent packages.

Remove CDP agent packages with yum remove serverbackup* or yum remove r1soft*, depending on which packages are installed on your machine.

After you remove the packages, all cdp_io processes should be removed, and any load or disk space usage that was caused by the processes will go down.

Reinstall CDP agent packages again with yum install command on packages you had installed before removal.

For example, for the output of rpm -qa | egrep “serverbackup|r1soft” in the example above, you will run the following command.

[~]# yum install serverbackup-agent serverbackup-async-agent-2-6 serverbackup-enterprise-agent serverbackup-setup

..

How to Install ionCube Loader 5 on cPanel/WHM server

At this moment cPanel does not support ionCube Loader 5 on WHM/cPanel servers, which will cause issues for client running files made with ionCube v9.

cPanel currently installs ionCube PHP Loader v4.7.5, when you install PHP using cPanel EasyApache utility.

Your standard php -v output on cPanel server might look something like this.

[email protected] [~]# php -v
PHP 5.4.45 (cli) (built: Dec 14 2015 17:18:43)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd.

If you want to install, or upgrade current ionCube PHP Loader on your server to version 5, you can do so manually by downloading the latest version from http://www.ioncube.com/loaders.php, and editing your global or account custom php.ini file.

How to install ionCube Loader manually on cPanel or standard Linux servers with no control panel.

Download the latest loader to your server from http://www.ioncube.com/loaders.php.

If you are running 32 bit server used the following command:

wget http://downloads3.ioncube.com/loader_downloads/ioncube_loaders_lin_x86.tar.gz

If you are running 64 bit server use the following command:

wget http://downloads3.ioncube.com/loader_downloads/ioncube_loaders_lin_x86-64.tar.gz

Unpack the package:

On 32 bit server:

tar xvzf ioncube_loaders_lin_x86.tar.gz

On 64 bit server:

tar xvzf ioncube_loaders_lin_x86-64.tar.gz

It will create ioncube folder in the same directory where you downloaded the file.

If you need to install ionCube Loader for PHP 5.4  you will use ioncube_loader_lin_5.4* files, and if you need to install it for PHP 5.5 you will use ioncube_loader_lin_5.5* files, and similar for other PHP versions.

[email protected] [~]# ls ioncube
./                             ioncube_loader_lin_4.4_ts.so*  ioncube_loader_lin_5.3.so*     ioncube_loader_lin_5.6_ts.so*
../                            ioncube_loader_lin_5.0.so*     ioncube_loader_lin_5.3_ts.so*  LICENSE.txt
ioncube_loader_lin_4.1.so*     ioncube_loader_lin_5.0_ts.so*  ioncube_loader_lin_5.4.so*     loader-wizard.php
ioncube_loader_lin_4.2.so*     ioncube_loader_lin_5.1.so*     ioncube_loader_lin_5.4_ts.so*  README.txt
ioncube_loader_lin_4.3.so*     ioncube_loader_lin_5.1_ts.so*  ioncube_loader_lin_5.5.so*     USER-GUIDE.md
ioncube_loader_lin_4.3_ts.so*  ioncube_loader_lin_5.2.so*     ioncube_loader_lin_5.5_ts.so*  USER-GUIDE.txt
ioncube_loader_lin_4.4.so*     ioncube_loader_lin_5.2_ts.so*  ioncube_loader_lin_5.6.so*
[email protected] [~]#

To replace the ionCube Loader for all the users on the server, you can just replace the extension file defined in the global php.ini at /usr/local/lib/php.ini

[email protected] [~]# grep zend_extension /usr/local/lib/php.ini
zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.4.so"
[email protected] [~]#

You can rename the original file, for restore purposes, if something goes wrong, and place the new version file from ioncube folder that was created with package extraction.

mv /usr/local/IonCube/ioncube_loader_lin_5.4_ts.so /usr/local/IonCube/ioncube_loader_lin_5.4_ts.so_cporig
mv /usr/local/IonCube/ioncube_loader_lin_5.4.so /usr/local/IonCube/ioncube_loader_lin_5.4.so_cporig
cp ioncube/ioncube_loader_lin_5.4.so /usr/local/IonCube/
cp ioncube/ioncube_loader_lin_5.4_ts.so /usr/local/IonCube/

You can confirm the new version of ionCube Loader with php -v, and look for “ionCube PHP Loader” part of the output.

[email protected] [~]# php -v
PHP 5.4.45 (cli) (built: Dec 14 2015 17:18:43)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2014 Zend Technologies
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.19, Copyright (c) 2002-2015, by ionCube Ltd.
[email protected] [~]# php -v | grep "ionCube PHP Loader"
    with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.19, Copyright (c) 2002-2015, by ionCube Ltd.
[email protected] [~]#

Upgrade to ionCube Loader 5 on one cPanel account.

If you wanted to change the ion Cube Loader version for only one cPanel account, you can place the ioncube_loader_lin* files for the PHP version used by your client to some custom folder, and define zend_extension values inside a custom php.ini folder on a specific cPanel account.

tl;dr

ionCube Loader can be installed manually on any Linux server with following steps:

  1. Download latest version of ionCube Loaders from http://www.ioncube.com/loaders.php
  2. Unpack the downloaded package
  3. move the ioncube_loader_lin_* files for your PHP version to your extension folder.
  4. Point to the corresponding file in your php.ini file, example
    zend_extension="/usr/local/IonCube/ioncube_loader_lin_5.4.so"

 

MySQL failing to start with message “Can’t find file: ‘./mysql/plugin.frm’ (errno: 23)”

This is a repost of a post from an old blog, made on December 28, 2013, that used to be on:

http://adminramble.com/mysql-failing-start-message-cant-find-file-mysqlplugin-frm-errno-23/

Original post:

Recently I had a problem, where MySQL service was failing to start.
When tailing the MySQL log the following would be recorded while service was being started.

131224 06:04:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/libexec/mysqld: Can't find file: './mysql/plugin.frm' (errno: 23)
131224 6:04:53 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
/usr/libexec/mysqld: Can't create/write to file '/tmp/ibqcFQMW' (Errcode: 23)
131224 6:04:53 InnoDB: Error: unable to create temporary file; errno: 23
131224 6:04:53 [ERROR] Plugin 'InnoDB' init function returned error.
131224 6:04:53 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131224 6:04:53 [ERROR] Can't create IP socket: Too many open files in system
131224 6:04:53 [ERROR] Aborting

131224 6:04:53 [Note] /usr/libexec/mysqld: Shutdown complete

131224 06:04:53 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Now, the message ‘Can’t find file: ‘./mysql/plugin.frm’ (errno: 23)‘ at the begging of the startup process might make you think that the problems is a missing file, but you can see that at the end of a startup process  this message is logged[ERROR] Can’t create IP socket: Too many open files in system.

This suggest that there is a problem with the number of files that is open on the system.
You can confirm this by using a perror utility, which prints system error messages.

If we check the error number 23, which is reported in the error message, we see that the cause of the failure is not a missing file, but that the file can’t be open because to many file handles are in use.

# perror 23
OS error code 23: Too many open files in system

You can check the current maximum number of file descriptors by checking the fs.file-max value in /etc/sysctl.conf, or use the sysctl command to check the current value.

# sysctl -a | grep file-max
 fs.file-max = 65536

To increase the maximum number of file handlesm you can edit /etc/sysctl.conf, change the value of fs.file-max to 200000 or some other value higher then the one you currently have, and then run sysctl -p to apply the new value to the system.

Now, after the file handle number has been increased, you should be able to start the MySQL service normally.

ConfigServer Internal Server Error 500, after cPanel update: fixed

This is a repost of a post from an old blog, made on June 29, 2013, that used to be on:

http://adminramble.com/configserver-internal-server-error-500-cpanel-update-fixed/

Original post:

After the WHM/cPanel update to 11.38 you might get the an error like this in your browser:

Internal Server Error
500
No response from subprocess (/usr/local/cpanel/whostmgr/docroot/cgi/addon_csf.cgi): subprocess exited with status 2

And something like this in /usr/local/cpanel/logs/error_log:

BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/JSON.pm line 15.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template/Plugin/JSON.pm line 12.
Compilation failed in require at /usr/local/cpanel/Cpanel/Template.pm line 53.
BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Template.pm line 53.
Compilation failed in require at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
BEGIN failed–compilation aborted at /usr/local/cpanel/Whostmgr/HTMLInterface.pm line 12.
Compilation failed in require at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
BEGIN failed–compilation aborted at /usr/local/cpanel/whostmgr/docroot/cgi/addon_cmc.cgi line 25.
For help, please send mail to this site’s webmaster, giving this error message and the time and date of the error.

when going to ConfigServer cPanel plugins in WHM like:

ConfigServer Explorer
ConfigServer Mail Manage
ConfigServer Mail Queues
ConfigServer ModSecurity Control
ConfigServer Security & Firewall

This happens if  the installed ConfigServer scripts on a cPanel/WHM server don’t get updated.

The solution for this error is simple.

To resolve this error simply SSH into your server as a root user and run the following command from command line:

curl -s configserver.com/free/csupdate | perl

This script will update: cmm, cmc, cmq, cse, csf, cxs, msinstall, msfe

You can see the ConfigServer blog post about the update here.

How to reset lost root password on SUSE Linux Enterprise Server

I had to use one of my virtual machines that I didn’t use for a while today, and of course  I couldn’t remember the password I used for it, so I had to change it.

Here is how to reset a forgotten root password on Novell SUSE

This is a guide for Novell SUSE Linux Enterprise Server 10 SP3.

On the boot menu select the first option “SUSE Linux enterprise server” and press ‘e’ for edit.

SLES boot menu
SLES boot menu

On the second menu select the kernel option and press ‘e’ for edit.

Select kernel line
Select kernel line

type init=/bin/bash (leave empty space at the begging), and press ‘Enter’ and then ‘b’ to boot with that option.

Add init=/bin/bash one the end of the line
Add init=/bin/bash one the end of the line

The system will boot with the root user logged on, type passwd to change the password and input your new password, or just press ‘Enter’ for no password (blank password)

Use passwd to change root password
Use passwd to change root password
Reboot the server and log on with your new password.