Category: Linux

CentOS 7 OpenVZ NetworkManager issues.

There is an issue with CentOS 7 containers on OpenVZ nodes, where NetworkManager service breaks the network, and only localhost 127.0.0.1 network interface is coming up.

When you try to check network interfaces, and IP address on the server, you will only see localhost interface.

root@server [/]# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 9059 bytes 1539654 (1.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 9059 bytes 1539654 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

When issuing service network restart you will and output like this:

Starting network (via systemctl): Job for network.service failed because the control process exited with error code. See "systemctl status network.service" and "journalctl -xe" for details.
[FAILED]

When you run journalctl -xe you will see errors like this:

-- Unit network.service has begun starting up.
 Feb 16 05:16:40 server network[2262]: Bringing up loopback interface: Could not load file '/etc/sysconfig/network-scripts/ifcfg-Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
 Feb 16 05:16:40 server network[2262]: [ OK ]
 Feb 16 05:16:40 server network[2262]: Bringing up interface venet0: Error: Connection activation failed: No suitable device founFeb 16 05:16:40 server network[2262]: [FAILED]
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server network[2262]: RTNETLINK answers: File exists
 Feb 16 05:16:40 server systemd[1]: network.service: control process exited, code=exited status=1
 Feb 16 05:16:40 server systemd[1]: Failed to start LSB: Bring up/down networking.
 -- Subject: Unit network.service has failed
 -- Defined-By: systemd
 -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
 --
 -- Unit network.service has failed.
 --
 -- The result is failed.
 Feb 16 05:16:40 server systemd[1]: Unit network.service entered failed state.
 Feb 16 05:16:40 server systemd[1]: network.service failed.
 Feb 16 05:16:40 server polkitd[226]: Unregistered Authentication Agent for unix-process:2257:65393 (system bus name :1.37, object

Issue can be resolved by stopping and disabling the NetworkManager service.

root@server [/]# service NetworkManager stop
Redirecting to /bin/systemctl stop NetworkManager.service
root@server [/]# chkconfig NetworkManager off

You can also uninstall NetworkManager completely.

References:

https://lists.openvz.org/pipermail/users/2015-December/006691.html

Openvz / CentOS 7 – Network Manager can break networking

 

CSF chattr and urlget warnings on restart

On most CentOS 7 machines you might be getting this type of warnings when restarting or updating ConfigServer Firewall (CSF), if you are missing certain commands or packages on your server.

*WARNING* Binary location for [CHATTR] [/usr/bin/chattr] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable
*WARNING* Missing or incorrect binary locations will break csf and lfd functionality
*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny

You also might be getting notifications in mail from your Cron Daemon with subject /usr/sbin/csf -u and following message:

*WARNING* URLGET set to use LWP but perl module is not installed, reverting to HTTP::Tiny

This warnings are showing because of missing packages not being installed on the server, and chattr command doesn’t exist.

[root@server ~]# type chattr
-bash: type: chattr: not found

To resolve CHATTR warning, you need to install e2fsprogs with

yum -y install e2fsprogs

To resolve URLGET warning you need to install perl-libwww-perl with

yum -y install perl-libwww-perl with

If you are getting following warning by mail notifications, or when running csf -u, you are missing perl-LWP-Protocol-https package.

[root@server ~]# csf -u
Oops: Unable to download: Protocol scheme 'https' is not supported (LWP::Protocol::https not installed)

Install perl-LWP-Protocol-https with:

yum -y install perl-LWP-Protocol-https

You might also be getting a following warning, if you don’t have host or unzip commands installed on your server.

*WARNING* Binary location for [UNZIP] [/usr/bin/unzip] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable
*WARNING* Binary location for [HOST] [/usr/bin/host] in /etc/csf/csf.conf is either incorrect, is not installed or is not executable

To resolve HOST warning, you needs to install bind-utils package with

yum -y install bind-utils

To resolve UNZIP warning you need to install unzip package with

yum -y install unzip

tl;dr required packages for CSF

If you are getting any kind of warnings when doing csf -r on CentOS 7 server make sure you have following packages installed.

iptables-services
unzip
bind-utils
perl-libwww-perl
e2fsprogs
perl-LWP-Protocol-https

Needed packages can be installed with following commands:

yum -y install iptables-services
yum -y install unzip
yum -y install bind-utils
yum -y install perl-libwww-perl
yum -y install e2fsprogs
yum -y install perl-LWP-Protocol-https

References:

http://forum.configserver.com/viewtopic.php?t=8864

http://www.liquidweb.com/kb/csf-config-server-firewall-installation/

https://community.centminmod.com/threads/ensure-perl-crypt-ssleay-perl-net-ssleay-are-installed-for-csf-firewall.2183/

systemd conflicts with initscripts during yum update

When running yum update on CentOS 7 OpenVZ container you might get a following error:

--> Processing Conflict: systemd-219-19.el7.x86_64 conflicts initscripts < 9.49.28-1 
--> Finished Dependency Resolution
Error: systemd conflicts with initscripts-9.49.17-1.el7.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Exact version I was running on this server is CentOS Linux release 7.1.1503 (Core), and updates were failing due to them requiring a newer systemd version, which required a newer version, v9.49.28 at least,  of initscripts installed.

Reason why systemd was conflicting with current version of initscripts, instead of just updating them to newer version, was that initscripts were excluded from update in yum.conf file.

[root@server ~]# grep exclude /etc/yum.conf
exclude=initscripts*

There are no default exclude packages in default CentOS/RHEL installs, and it wasn’t put there by me, so question is why was initscripts excluded from updates.

Issues with initscripts v9.49.24 in CentOS 7 OpenVZ containers

It turns out there were some issues with initscripts v9.49.24, which seemed to introduce problems with network connectivity on CentOS 7 OpenVZ VPS machines.

https://vpsboard.com/topic/1908-creating-cpanel-openvz-template-wwhm-setup-wizard-fix/?page=2
https://forum.proxmox.com/threads/centos-7-openvz-template-in-pve-3-4-issues.21808/

I commented out exclude=initscripts* from yum.conf, and checked the currently available version.

[root@server ~]# grep exclude /etc/yum.conf
#exclude=initscripts*
[root@server ~]# yum check-updates initscripts
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.greenmountainaccess.net
 * epel: mirror.nexcess.net
 * extras: mirrors.lga7.us.voxel.net
 * updates: mirror.dattobackup.com
 * webtatic: us-east.repo.webtatic.com

initscripts.x86_64                                               9.49.30-1.el7                                                base

Current version is v9.49.30, so I presumed issues with v9.49.24 were corrected in this version, and proceeded with the update.

Update has completed successfully, and all packages including initscrips were updated to latest version, as was CentOS.

[root@server ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

I haven’t noticed any network or any other issues on the VPS after the upgrade, so it seems any network issues from v9.49.24, don’t exist in v9.49.30.

[root@server ~]# rpm -qa | grep initscripts
initscripts-9.49.30-1.el7.x86_64

Reverse DNS does not match SMTP Banner

UPDATE: WHM/cPanel removed support for in version 11.50, so changes below are not valid for versions 11.50+

https://documentation.cpanel.net/display/ALD/11.50+Release+Notes#id-11.50ReleaseNotes-/etc/mail_reverse_dnsremoved

 

If you make an SMTP test on http://mxtoolbox.com you might be getting a following error shown in the test results “Reverse DNS does not match SMTP Banner”.

This error is showing because your SMTP greeting message is not matching the PTR records for the IP of the SMTP server used in test.

Following files need to be used and configured properly, for SMTP banner to match reverse DNS records.

/etc/mailhelo
/etc/mailips
/etc/mail_reverse_dns
/etc/exim.conf

Configure Exim to use mailhelo and mailips file

Go to WHM to Home »Service Configuration »Exim Configuration Manager and in Basic Editor on Domains and IPs tab set following settings:

Send mail from account’s dedicated IP address: OFF
Reference /etc/mailhelo for outgoing SMTP HELO: ON
Reference /etc/mailips for outgoing SMTP connections: ON

Configure necessary values in configuration files

Edit or create  /etc/mail_reverse_dns file and set the following in it for needed IPs.

x.x.x.x: rdns of the IP x.x.x.x
y.y.y.y: rdns of IP y.y.y.y

Edit or create /etc/mailhelo file and set following in it for the domains that you want to setup SMTP banner for.

example.com: reverse dns of the IP used for example.com domain
*: default SMTP HELO for unconfigured domains

Edit or create /etc/mailips file and set following in it:

example.com: x.x.x.x #x.x.x.x is the IP used for outgoing mail for domain example.com
*: y.y.y.y #y.y.y.y is the default IP that will be used for unconfigured domains

Configure exim.conf to use correct SMTP Banner

Following values need to be configured in exim.conf for SMTP Banner to be set to rDNS values set in /etc/mail_reverse_dns.

smtp_active_hostname
message_id_header_domain
smtp_banner

Be default only smtp_banner is set on cPanel servers, and it has a different value then needed.

root@server1 [~]# egrep "smtp_active_hostname|message_id_header_domain|smtp_banner" /etc/exim.conf
smtp_banner = "${primary_hostname} ESMTP Exim ${version_number} \

smtp_banner will probably look like this on your cPanel server.

"${primary_hostname} ESMTP Exim ${version_number}  \#${compile_number} ${tod_full} \n   We do not authorize the use of this system to transport unsolicited, \n   and/or bulk e-mail."
Configure values in exim.conf over shell

Locate the line smtp_banner and change its value so it looks like following:

smtp_banner = "${smtp_active_hostname} ESMTP Exim ${version_number} \"

Add smtp_active_hostname line value to exim.conf to look line following:

smtp_active_hostname = ${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}

Add message_id_header_domain line to exim.conf to look like following:

message_id_header_domain = $smtp_active_hostname

In the end related values in exim.conf should look like this:

root@server1 [~]# egrep "smtp_active_hostname|message_id_header_domain|smtp_banner" /etc/exim.conf
smtp_banner = "${smtp_active_hostname} ESMTP Exim ${version_number} \"
smtp_active_hostname = ${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}
message_id_header_domain = $smtp_active_hostname

Restart exim with /scripts/restartsrv_exim and SMTP tests should now pass without the SMTP banner warning.

Configure values in exim.conf over WHM

In your WHM go to Home »Service Configuration »Exim Configuration Manager and go to Advanced Editor.

Search for the smtp_banner field and change default value to:

"${smtp_active_hostname} ESMTP Exim ${version_number} \"

 

Edit smtp_banner in WHM
Edit smtp_banner in WHM

Find the “Add additional configuration setting” button and add two new configuration settings smtp_active_hostname and message_id_header_domain.

additional configuration settings
Add additional configuration setting in WHM

For smtp_active_hostname set the following value:

${lookup{$interface_address}lsearch{/etc/mail_reverse_dns}{$value}{$primary_hostname}}

For message_id_header_domain set the following value:

$smtp_active_hostname

References:

https://forums.cpanel.net/threads/easy-fix-your-smtp-banner-smtp-greeting-and-reverse-dns-for-dedicated-ips.391311/

https://forums.cpanel.net/threads/exim-banner-mail-headers-and-resellers-with-own-ip.100697/

How to provide SSH password inside a script or oneliner

If you ever need to provide a password for SSH login inside a bash script or a shell command, to avoid being asked a password when SSH keys are not used, it can be done with usage of expect command, or sshpass utility.

Using expect

Expect is a program that “talks” to other interactive programs according to a script.

http://www.tcl.tk/man/expect5.31/expect.1.html

Lets say you want to SSH to a server and run a remote command, ls for instance, with a command like this:

ssh user@server "ls -lh file"

If you are not using SSH keys, you will be prompted a password, and will need to enter it manually.

If you want to avoid entering a password, and have it inputted to login prompt automatically you can use expect command.

Provide SSH password inside a script.

Expect reads cmdfile for a list of commands to execute. Expect may also be invoked implicitly on systems which support the #! notation by marking the script executable, and making the first line in your script:
#!/usr/local/bin/expect -f

Example of a script which runs a remote command over SSH, with password being provided inside a script.

#!/usr/bin/expect -f
spawn ssh [email protected] "ls /file"
expect "assword:"
send "mypassword\r"
interact

One-liner

Running a SSH command with provided password inside a one-liner, can be done using expect -c and then putting the commands inside single quotes.

The -c flag prefaces a command to be executed before any in the script. The command should be quoted to prevent being broken up by the shell. This option may be used multiple times. Multiple commands may be executed with a single -c by separating them with semicolons.

Example of a one-line command:

expect -c 'spawn ssh user@server "ls -lh file"; expect "assword:"; send "mypassword\r"; interact'

Using sshpass

SSH password prompt can also be bypassed by connecting with sshpass command, which is available in EPEL repo in CentOS

To install sshpass, first make sure you have EPEL repo on your server.

You can install EPEL with

yum -y install epel-release

Install sshpass with

yum -y install sshpass

You can then connect to remote server with sshpass using a command similar to this:

sshpass -p 'password' ssh user@server

If you have never connected to the server before, you will probably not get connected to remote server due to host key checking.
To bypass host key checking use -o StrictHostKeyChecking=no option:

sshpass -p 'password' ssh -o StrictHostKeyChecking=no  user@server
References:

http://unix.stackexchange.com/questions/252777/use-she-bang-in-oneliner

http://stackoverflow.com/questions/1924464/bash-controlling-ssh

http://stackoverflow.com/questions/16928004/how-to-enter-ssh-password-using-bash

http://linux.die.net/man/1/expect

http://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/

Find jobs for SysAdmins with Jooble