VirtualBox returns “Kernel driver not installed” on Ubuntu

When trying to run VirtualBox on Ubuntu 16.04, or some other version of Linux, you might get a following type an error message, when trying to start a virtual machine, you just created on a fresh installation of VirtualBox.

VirtualBox might return error saying that it failed to open a session for  the virtual machine.

Details of error message will say that virtual machine has terminated unexpectedly during startup with exit code 1 (0x1).

VirtualBox failed session message
VirtualBox failed session message

You will also get a “Kernel driver not installed (rc=-1908” message.

VirtualBox Kernel driver error
VirtualBox Kernel driver error

Error will say “The VirtualBox Linux kernel driver (vboxdrv) is either not loaded or there is a permission problem with /dev/vboxdrv.”

Error message will ask you to try to reinstall the kernel module by executing /sbin/vboxconfig as root, to reinstall the module, which could fail to complete successfully when you try to run the command in terminal.

One of the possible causes of this type of behavior is that you have Secure Boot enabled in your BIOS on the host machine, which is preventing the install of third-party drivers, and causing issues with VirtualBox installation, which fails to install all kernel modules.

To resolve this error, disable Secure Boot in your BIOS, and then try to run /sbin/vboxconfig command as root, or reinstall VirtualBox, with Secure Boot disabled, to install all the necessary modules.

VMware rename CentOS 7 NIC names

CentOS 7 virtual machines on VMware will by default use predictable network device naming for network interfaces on the machine, causing their names to be in enoXXXXXXXX format.

This will cause issues when adding 10 or more additional IPs in WHM, as network interface name will be longer than the 15 characters.

Maximum length supported for network interface name on cPanel servers is 15 characters.

When starting ipaliases service, only first 9 additional IPs will be added, and for rest of the IPs error “RTNETLINK answers: Numerical result out of range” will be shown, and IPs will not be shown in ip addr, or ifconfig output.

[[email protected] ~]# /scripts/restartsrv_ipaliases
Waiting for "ipaliases" to stop ...finished.
Waiting for "ipaliases" to start ...finished.
Service Status
Startup Log
 Oct 03 20:29:20 ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 ipaliases[233833]: Bringing up eno33559296:cp14 RTNETLINK answers: Numerical
result out of range
 Oct 03 20:29:20 ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 ipaliases[233833]: Routing RTNETLINK answers: Invalid argument
 Oct 03 20:29:20 ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 ipaliases[233833]: Bringing up eno33559296:cp15 RTNETLINK answers: Numerical
result out of range
 Oct 03 20:29:20 ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 ipaliases[233833]: Routing RTNETLINK answers: Invalid argument
 Oct 03 20:29:20 ipaliases[233833]: [FAILED]
 Oct 03 20:29:20 systemd[1]: Started cPanel IP aliases service.
Log Messages
 Oct 3 20:29:20 server ipaliases: [FAILED]
 Oct 3 20:29:20 server ipaliases: Routing x.x.x.x RTNETLINK answers: Invalid argument
 Oct 3 20:29:20 server ipaliases: [FAILED]
 Oct 3 20:29:20 server ipaliases: Bringing up eno33559296:cp15 RTNETLINK answers: Numerical result out of range

To resolve the issues, network devices can be renamed back to old ethX type of naming.

To rename network devices to old names following steps are needed.

  1. Edit /etc/sysconfig/grub
  2. Update GRUB configuration with new kernel parameters
  3. Rename network files
  4. Edit renamed network files
  5. Reboot the server

To rename devices do the following

Edit /etc/sysconfig/grub

Find a line containing “GRUB_CMDLINE_LINUX”, and append “net.ifnames=0 biosdevname=0“ on the line.

File should look something like this.

[[email protected] ~]# cat /etc/sysconfig/grub
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet net.ifnames=0 biosdevname=0"

Update GRUB configuration with new kernel parameters, with following command:

grub2-mkconfig -o /boot/grub2/grub.cfg

Rename enoXXXXXXXX network files of all interfaces to ethX network file.

For example:

mv /etc/sysconfig/network-scripts/ifcfg-eno16777984 /etc/sysconfig/network-scripts/ifcfg-eth0
mv /etc/sysconfig/network-scripts/ifcfg-eno33557248 /etc/sysconfig/network-scripts/ifcfg-eth1

This will rename file ifcfg-eno16777984, to ifcfg-eth0, renaming interface eno16777984 to eth0, and will rename file ifcfg-eno33557248, to ifcfg-eth1, renaming interface eno33557248, to eth1.

Edit new ethX network files.

Replace value of both NAME and DEVICE field with new ethX names.

File should look something like this.

[[email protected] ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0

Reboot the server, and you should now see network interfaces using old CentOS 6 style names.

Additional changes for cPanel servers

Change public network interface in Basic cPanel & WHM Setup.

Go to Home »Server Configuration »Basic cPanel & WHM Setup and change public interface from old enoXXXXXXXX to new ethX name.

Change public interface to new name
Change public interface to new name

Restart ipaliases service with /scripts/restartsrv_ipaliases.

Yum and curl returning “Illegal instruction (core dumped)” on Xen

When running yum or curl commands on a CentOS 6 XenServer Virtual Machine you might be getting an “Illegal instruction (core dumped)” error returned in your console output.

[email protected] [~]# yum update
Loaded plugins: fastestmirror, rhnplugin
Setting up Update Process
Loading mirror speeds from cached hostfile
 * base:
 * cloudlinux-x86_64-server-6:
 * extras:
 * updates:
Illegal instruction (core dumped)

The issue is due to Python attempting to execute a CPU opcode advertised as available by the server’s host node virtualization system (XEN), but is not actually supported by the host node’s hardware.


Issue can be resolved by running export NSS_DISABLE_HW_AES=1, and then running yum update, to update to newer packages, after which issue should not be happening anymore.

[email protected] [~]# export NSS_DISABLE_HW_AES=1
[email protected] [~]# yum -y update

How to check for, and clean Ebury SSH Rootkit

What is Ebury

Ebury is a SSH Rootkit, and password sniffer which steals SSH login credentials from incoming and outgoing SSH connections, and also steals private SSH keys stored on the infected system.

Ebury can replace SSH binaries, and shared library files used by executables like sshd, wget, curl, …

How to detect Ebury on a system

From version 1.5 Ebury uses Unix domain sockets for interprocess communication.

Malicous process can be seen using netstat -plan | grep atd.

This command should not return any results on clean systems.

[email protected] [~]# netstat -plan | grep atd 
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsL

Atd should not be listening on any network port or socket.

Ebury will also place additional shared library files, and patch installed libkeyutils file to link to those files.

Files usually found on Ebury infected machines can be one or more of the following:


If any of those files exist, check if the files were provided by any rpm using rpm -qf command.

[email protected] [~]# rpm -qf /lib64/tls/
file /lib64/tls/ is not owned by any package

On clean system command should return the name of the rpm package which installed that file.

[email protected] [/lib]# rpm -qf
Script to check for suspicious files, and processes

Here is a small script that can be used to check for possible Ebury infection.


if [[ `netstat -pan | grep -w atd` ]]; then
    printf "This server appears to have atd process listening on Unix socket or network port\nCheck server for possible Ebury infection\n\n===\n`netstat -pan | grep -w atd`\n===\n\n"

declare -a file_list=("/lib64/tls/" "/lib64/tls/" "/lib64/" "/lib64/" "/lib64/"); 

for file in "${file_list[@]}"; do 
    if [[ -f $file ]]; then
        if [[ `rpm -qf $file` == *'not owned'* ]]; then
            printf "===\nFile $file is not owned by any RPM package, and there is a possible rootkit infection\nCheck server for possible Ebury infection\n===\n"

Save a script like on your system, and run with bash

On an infected system, command will return something like this:

[[email protected] ~]# bash /root/
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury

unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k

File /lib64/tls/ is not owned by any RPM package, and there is a possible rootkit infection
server for possible Ebury infection
File /lib64/tls/ is not owned by any RPM package, and there is a possible rootkit infection
server for possible Ebury infection
[[email protected] ~]#

NOTE: Suspicious processes and fileS might not be visible over SSH connections

Some variants of Ebury will hide suspicious processes and files, if you are checking the system over SSH connection (link).

In cases like that, checks will need to be done over local terminal, remote management console, or through screen session, for all processes and files to be visible.

If you are unable to connect to the server without SSH, install screen with yum -y install screen, and run from screen session, to double check for any possible infection.

In some cases when checks are done over SSH, you might be getting different result if you check for processes and files over screen session.

[[email protected] ~]# /root/
[[email protected] ~]# screen -dmS ebury bash -c '/root/ >> test'; sleep 30; cat test
This server appears to have atd process listening on Unix socket or network port
Check server for possible Ebury

unix 2 [ ACC ] STREAM LISTENING 1278995234 127563/atd @/tmp/dbus-BmCahxCc3k

File /lib64/tls/ is not owned by any RPM package, and there is a possible rootkit infection
server for possible Ebury infection
File /lib64/tls/ is not owned by any RPM package, and there is a possible rootkit infection
server for possible Ebury infection
[[email protected] ~]#
How to clean Ebury infection

Most important thing to note is, that in case of root level infections like these ones, the only safe way is to do a complete server rebuild after you clean the infection, and make any necessary backups.

In order to clean Ebury infection, you need to kill the processes you found with netstat, remove suspicious library files, and reinstall keyutils-libs* rpm package. It would be also advisable to reinstall SSH packages.

Steps that can be taken to clean the system:

Check the actual keyutils-libs RPM packages you have installed on your system, and download them before removing any files from the system, as it is possible in some cases that some of the infected files are used by yum, curl, wget, and that you won’t be able to do install with yum after removing the files, or use curl, or wget to download RPMs for install.

  • Kill all SSH connections with killall sshd.
  • Kill the atd processes listening over Unix socket with kill -9 `lsof -Pt /usr/sbin/atd`.
  • Remove the suspicious files you found, that were not connected with any rpm package.
  • Reinstall keyutils-libs and SSH packages, preferably with rpm -ivh --replacefiles --replacepkgs on the predownloaded packages, but in most cases you can use yum:
    yum -y reinstall openssh* libssh* keyutils-libs*

After you have reinstalled necessary packages, change your root password, and all SSH keys on the server, and reboot the server to check if suspicious processes and files will return after it.

If possible, always do a full server rebuild, even if no signs of infection exist after reboot.

Avoid cleaning the infection over SSH connection

It would be advisable to kill all SSH connections that exist on the system you are about to clean, so you should be doing it while connected to the server some other way, but if you need to clean the server over SSH, a script like this can be used to accomplish that (you need to replace the files being referenced in the script, with the files you have found on your own system)


killall sshd; 
kill -9 `lsof -Pt /usr/sbin/atd`; 
rm -f /lib64/tls/; 
rm -f /lib64/tls/; 
yum -y reinstall openssh* libssh* keyutils-libs*; 
service sshd start

Linux SysAdmin DevOps Interview questions

My answers to some of the questions from a collection of Linux Sysadmin/DevOps interview questions found here:

General Questions:

  • What did you learn yesterday/this week?
  • Talk about your preferred development/administration environment. (OS, Editor, Browsers, Tools etc.)
  • Tell me about the last major Linux project you finished.
  • Tell me about the biggest mistake you’ve made in [some recent time period] and how you would do it differently today. What did you learn from this experience?
  • Why we must choose you?
  • What function does DNS play on a network?
     DNS resolves hostnames to IPs, and IPs back to hostnames, and allows using easily memorable domains and hostnames, instead of more hard to remember IPs.
  • What is HTTP?
    Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems.[1] HTTP is the foundation of data communication for the World Wide Web.
    Hypertext is structured text that uses logical links (hyperlinks) between nodes containing text. HTTP is the protocol to exchange or transfer hypertext.
  • What is an HTTP proxy and how does it work?
    An HTTP Proxy serves two intermediary roles as an HTTP Client and an HTTP Server for security, management, and caching functionality. The HTTP Proxy routes HTTP Client requests from a Web browser to the Internet, while supporting the caching of Internet data.
    Proxy server advantages include:
    - Maintaining identity anonymity as a security precaution.
    - Accelerating caching rates.
    - Facilitating access to prohibited sites.
    - Enforcing access policies on certain websites.
    - Allowing a site to make external server requests.
    - Avoiding security controls.
    - Bypassing Internet filtering for access to prohibited content.
  • Describe briefly how HTTPS works.
    - Browser checks the certificate to make sure that the site you are connecting to is the real site and not someone intercepting.
    - Determine encryption types that the browser and web site server can both use to understand each other.
    - Browser and Server send each other unique codes to use when scrambling (or encrypting) the information that will be sent.
    - The browser and server start talking using the encryption, the web browser shows the encrypting icon, and web pages are processed secured.
  • What is SMTP? Give the basic scenario of how a mail message is delivered via SMTP.
    Simple Mail Transfer Protocol (SMTP) is an Internet standard for electronic mail (email) transmission.
    Simple Mail Transfer Protocol (SMTP) client contacts the destination host's Simple Mail Transfer Protocol (SMTP) server on well-known port 25, to deliver the mail. 
    Client waits for the server to send a 220 READY FOR MAIL message.
    Upon receipt of the 220 message, the client sends a HELO command.
    Server then responds with a "250 Requested mail action okay" message.
    After this, the mail transaction will begin with a MAIL command that gives the sender identification as well as a FROM: field that contains the address to which errors should be reported.
    After a successful MAIL command, the sender issues a series of RCPT commands that identify recipients of the mail message. The receiver will the acknowledge each RCPT command by sending 250 OK or by sending the error message 550 No such user here.
    After all RCPT commands have been acknowledged, the sender issues a DATA command to inform the receiver that the sender is ready to transfer a complete mail message. 
    The receiver responds with message 354 Start mail command with an ending sequence that the sender should use to terminate the message data. 
    The termination sequence consists of 5 characters: carriage return, line feed, period, carriage return, and line feed (<CRLF>.<CRLF>).
    The client now sends the data line by line, ending with the 5-character sequence <CRLF>.<CRLF> line, upon which the receiver will acknowledge with a 250 OK, or an appropriate error message if anything went wrong.
    After the sending is completed, the client can follow any of these actions.
    Terminate Session: If the current Simple Mail Transfer Protocol (SMTP) client has no more messages to send, the connection can be closed with a QUIT command, which will be answered with a 221 Service closing transmission channel reply.
    Exchange Roles: If the current Simple Mail Transfer Protoco (SMTP) client has no more messages to send, but is ready to receive any messages from the current Simple Mail Transfer Protoco (SMTP) server, it can issue the TURN command. Now the SMTP client and the SMTP server will switch their role of sender/receiver, and the sender (previous receiver) can now send messages by issuing a MAIL command.
    Send Another Mail: If the Simple Mail Transfer Protoco (SMTP) client (sender) has another message to send, it can issue a new MAIL command.
  • What is RAID? What is RAID0, RAID1, RAID5, RAID10?
    RAID (redundant array of independent disks; originally redundant array of inexpensive disks) provides a way of storing the same data in different places on multiple hard disks.
    RAID 0: This configuration has striping across multiple disks but no redundancy of data. It offers the best performance but no fault-tolerance.
    RAID 1: Also known as disk mirroring, this configuration consists of at least two drives that duplicate the storage of data. There is no striping. Read performance is improved since either disk can be read at the same time. Write performance is the same as for single disk storage.
    RAID 5: This level is based on block-level striping with parity. The parity information is striped across each drive, allowing the array to function even if one drive were to fail. The array’s architecture allows read and write operations to span multiple drives. This results in performance that is usually better than that of a single drive, but not as high as that of a RAID 0 array. RAID 5 requires at least three disks, but it is often recommended to use at least five disks for performance reasons.
      RAID 5 arrays are generally considered to be a poor choice for use on write-intensive systems because of the performance impact associated with writing parity information. When a disk does fail, it can take a long time to rebuild a RAID 5 array. Performance is usually degraded during the rebuild time and the array is vulnerable to an additional disk failure until the rebuild is complete.
    RAID 10 (RAID 1+0): Combining RAID 1 and RAID 0, this level is often referred to as RAID 10, which offers higher performance than RAID 1 but at a much higher cost. In RAID 1+0, the data is mirrored and the mirrors are striped.
  • What is a level 0 backup? What is an incremental backup?
    A full backup would be level 0, and the different levels of incremental backups levels 1, 2, 3, etc. At each incremental backup level you back up everything that has changed since the previous backup at the same or a previous level.
  • Describe the general file system hierarchy of a Linux system.
    In the FHS, all files and directories appear under the root directory /, even if they are stored on different physical or virtual devices. 
    /boot/ directory contains static files required to boot the system, such as the Linux kernel. These files are essential for the system to boot properly.
    /dev/ directory contains file system entries which represent devices that are attached to the system. These files are essential for the system to function properly.
    /etc/ directory is reserved for configuration files that are local to the machine. No binaries are to be put in /etc/. Any binaries that were once located in /etc/ should be placed into /sbin/ or /bin/.
    /lib/ directory should contain only those libraries needed to execute the binaries in /bin/ and /sbin/. These shared library images are particularly important for booting the system and executing commands within the root file system.
    /mnt/ directory is for temporarily mounted file systems, such as CD-ROMs
    /opt/ directory provides storage for large, static application software packages.
    A package placing files in the /opt/ directory creates a directory bearing the same name as the package. This directory, in turn, holds files that otherwise would be scattered throughout the file system, giving the system administrator an easy way to determine the role of each file within a particular package.
    For example, if sample is the name of a particular software package located within the /opt/ directory, then all of its files are placed in directories inside the /opt/sample/ directory, such as /opt/sample/bin/ for binaries and /opt/sample/man/ for manual pages.
    Large packages that encompass many different sub-packages, each of which accomplish a particular task, are also located in the /opt/ directory, giving that large package a way to organize itself. In this way, our sample package may have different tools that each go in their own sub-directories, such as /opt/sample/tool1/ and /opt/sample/tool2/, each of which can have their own bin/, man/, and other similar directories.
    /proc/ directory contains special files that either extract information from or send information to the kernel.
    /sbin/ directory stores executables used by the root user. The executables in /sbin/ are only used at boot time and perform system recovery operations.
    /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable among a group of hosts, but not found in /usr.
    /var/ is for variable data files. This includes spool directories and files, administrative and logging data, and transient and temporary files.
    System log files such as messages/ and lastlog/ go in the /var/log/ directory. The /var/lib/rpm/ directory contains RPM system databases. Lock files go in the /var/lock/ directory, usually in directories for the program using the file. The /var/spool/ directory has subdirectories for programs in which data files are stored.

Simple Linux Questions:

  • What is the name and the UID of the administrator user?
    $ id root
    uid=0(root) gid=0(root) groups=0(root)
  • How to list all files, including hidden ones, in a directory?
    ls -a
    ls --all
    ls - list directory contents
    -a, --all
     do not ignore entries starting with .
  • What is the Unix/Linux command to remove a directory and its contents?
    rm -r ./dir/
    rm - remove files or directories
    -r, -R, --recursive
     remove directories and their contents recursively
  • Which command will show you free/used memory? Does free memory exist on Linux?
    free - Display amount of free and used memory in the system
  • How to search for the string “my konfi is the best” in files of a directory recursively?
    $ find ./* -type f -exec grep -H 'my konfi is the best' {} \;
    $ grep -r 'my konfi is the best' ./*
  • How to connect to a remote server or what is SSH?
    ssh [email protected]_server
  • How to get all environment variables and how can you use them?
    $ printenv
    $ printenv PATH HOME
  • I get “command not found” when I run ifconfig -a. What can be wrong?
    net-tools package is not installed or ifconfig is not inside $PATH
  • What happens if I type TAB-TAB?
    It depends where you are type this.
    If we are talking about shells like bash\zsh, so you type TAB-TAB it will enable built in "completion" function.
    Most shells allow command completion, typically bound to the TAB key, which allow you to complete the names of commands stored upon your PATH, file names, or directory names. This is typically used like so:
    $ ls /bo[TAB]
    When you press the TAB key the argument /bo is automatically replaced with the value /boot.
  • What command will show the available disk space on the Unix/Linux system?
    df -h
    df - report file system disk space usage
  • What commands do you know that can be used to check DNS records?
    $ host
    $ nslookup
    $ dig
    dig - DNS lookup utility
    host - DNS lookup utility
    nslookup - query Internet name servers interactively
  • What Unix/Linux commands will alter a files ownership, files permissions?
    chown - change file owner and group
    chmod - change file mode bits
  • What does chmod +x FILENAMEdo?
    This command will set executation bit to FILENAME for everybody owner\group\other.
  • What does the permission 0750 on a file mean?
    0750 means that owner can read\write\execute this file, also members of group can read and execute, other users can do nothing with it.
  • What does the permission 0750 on a directory mean?
    0750 means that owner can read\write\execute(see file list of directory) this directory, also members of group can read and list, other users can do nothing with it.
  • How to add a new system user without login permissions?
    useradd -r newuser --shell=/sbin/nologin
    -r, --system
     Create a system account.
     System users will be created with no aging information in /etc/shadow, and their numeric identifiers are chosen in
     the SYS_UID_MIN-SYS_UID_MAX range, defined in /etc/login.defs, instead of UID_MIN-UID_MAX (and their GID
     counterparts for the creation of groups).
     Note that useradd will not create a home directory for such an user, regardless of the default setting in
     /etc/login.defs (CREATE_HOME). You have to specify the -m options if you want a home directory for a system account
     to be created.
     -s, --shell SHELL
     The name of the user's login shell. The default is to leave this field blank, which causes the system to select the
     default login shell specified by the SHELL variable in /etc/default/useradd, or an empty string by default.
  • How to add/remove a group from a user?
    deluser user group
  • What is a bash alias?
    A Bash alias is essentially nothing more than a keyboard shortcut, an abbreviation, a means of avoiding typing a long command sequence. If, for example, we include alias lm="ls -l | more" in the ~/.bashrc file, then each lm [1] typed at the command-line will automatically be replaced by a ls -l | more. This can save a great deal of typing at the command-line and avoid having to remember complex combinations of commands and options. Setting alias rm="rm -i" (interactive mode delete) may save a good deal of grief, since it can prevent inadvertently deleting important files.
  • How do you set the mail address of the root/a user?
    Mail address can be put in /root/.forwards, or set in /etc/aliases file.
  • What does CTRL-c do?
    Ctrl-C sends SIGINT, a signal that causes the process to terminate.
  • What is in /etc/services?
    /etc/services maps port numbers to named services.
  • How to redirect STDOUT and STDERR in bash? (> /dev/null 2>&1)
    some_command >file.log 2>&1 
    some_command >file.log 2>file.err
  • What is the difference between UNIX and Linux.
    UNIX is copyrighted name for an OS.
    Linux is a Unix clone written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. It aims towards POSIX compliance.
    Linux is just a kernel and Linux distribution makes it complete usable operating systems by adding various applications.
  • What is the difference between Telnet and SSH?
    key difference between Telnet and SSH is that SSH uses encryption
  • Explain the three load averages and what do they indicate. What command can be used to view the load averages?
    From left to right, these numbers show you the average load over the last one minute, the last five minutes, and the last fifteen minutes.
    On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.
  • Can you name a lower-case letter that is not a valid option for GNU ls?
    ls: invalid option -- 'y'
    Try 'ls --help' for more information.
    ls: invalid option -- 'z'
    Try 'ls --help' for more information.

Medium Linux Questions:

  • What do the following commands do and how would you use them?
    • tee
       tee - read from standard input and write to standard output and files
      command | tee file.log
      will show output of command on screen, and will also write the output to file.log
    • awk
       AWK is a programming language designed for text processing and typically used as a data extraction and reporting tool.
      awk '{print$1}' file
      will print out only the first column from file.
    • tr
       tr - translate or delete characters
      tr -d '\r'
      removes carriage return characters.
    • cut
       cut - remove sections from each line of files
      cut -d':' -f1 /etc/passwd
       displays only first field of each lines from /etc/passwd file using the field delimiter : (colon).
    • tac
       tac - concatenate and print files in reverse
      reads file in reverse, oposite to cat.
      tac file
    • curl
       curl - transfer a URL
    • wget
        Wget - The non-interactive network downloader.
      wget -O file.tar
    • watch
       watch - execute a program periodically, showing output fullscreen
      watch 'command'
    • head
       head - output the first part of files
      head -20 file
    • tail
       tail - output the last part of files
      tail -20 file
  • What does an & after a command do?
    & makes the command run in the background.
     If a command is terminated by the control operator &, the shell executes the command in the background in a
     subshell. The shell does not wait for the command to finish, and the return status is 0.
  • What does & disown after a command do?
    runs the process within the Terminal's current bash instance, in the background, but the process is detached from the bash's jobs' list (i.e. the process is not listed as a bash foreground / background job and stdin, stdout and stderr are still bound to the terminal); immune to hangups;
    & puts the job in the background, that is, makes it block on attempting to read input, and makes the shell not wait for its completion.
    disown removes the process from the shell's job control, but it still leaves it connected to the terminal. One of the results is that the shell won't send it a SIGHUP. Obviously, it can only be applied to background jobs, because you cannot enter it when a foreground job is running.
  • What is a packet filter and how does it work?
    Packet filters act by inspecting the "packets" which are transferred between computers on the Internet. If a packet does not match the packet filter's set of filtering rules, the packet filter will drop.
  • What is Virtual Memory?
    In computing, virtual memory is a memory management technique that is implemented using both hardware and software. It maps memory addresses used by a program, called virtual addresses, into physical addresses in computer memory. Main storage as seen by a process or task appears as a contiguous address space or collection of contiguous segments. The operating system manages virtual address spaces and the assignment of real memory to virtual memory.
  • What is swap and what is it used for?
    Swap is a special type of memory.
    Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.
    Swapping is a useful technique that enables a computer to execute programs and manipulate data files larger than main memory. The operating system copies as much data as possible into main memory, and leaves the rest on the disk. When the operating system needs data from the disk, it exchanges a portion of data (called a page or segment) in main memory with a portion of data on the disk.
  • What is an A record, an NS record, a PTR record, a CNAME record, an MX record?
    A record: map hostnames to an IP address of the host, but it is also used for DNSBLs, storing subnet masks in RFC 1101, etc.
    NS record: Name server record
    PTR record: Pointer record, common use is for implementing reverse DNS lookups, but other uses include such things as DNS-SD.
    CNAME record: Canonical name record. Alias of one name to another: the DNS lookup will continue by retrying the lookup with the new name.
    MX record: Mail exchange record. Maps a domain name to a list of message transfer agents for that domain
  • Are there any other RRs and what are they used for?
  • What is a Split-Horizon DNS?
    split-horizon DNS, split-view DNS, split-brain DNS, or split DNS is the facility of a Domain Name System (DNS) implementation to provide different sets of DNS information, selected by, usually, the source address of the DNS request.
    This facility can provide a mechanism for security and privacy management by logical or physical separation of DNS information for network-internal access (within an administrative domain, e.g., company) and access from an unsecure, public network (e.g. the Internet).
  • What is the sticky bit?
    Sticky Bit is mainly used on folders in order to avoid deletion of a folder and it’s content by other users though they having write permissions on the folder contents. If Sticky bit is enabled on a folder, the folder contents are deleted by only owner who created them and the root user. No one else can delete other users data in this folder(Where sticky bit is set). This is a security measure to avoid deletion of critical folders and their content(sub-folders and files), though other users have full permissions.
  • What does the immutable bit do to a file?
    A file with an immutable attribute can not be:
    No soft or hard link created by anyone including root user.
  • What is the difference between hardlinks and symlinks? What happens when you remove the source to a symlink/hardlink?
    symbolic link is a link to another name, it links to another file or folder.
    hard link points to an inode, not a file.
    Once a hard link has been made the link is to the inode. deleting renaming or moving the original file will not affect the hard link as it links to the underlying inode. Any changes to the data on the inode is reflected in all files that refer to that inode.
    Hard links are only valid within the same File System. Symbolic links can span file systems as they are simply the name of another file.
  • What is an inode and what fields are stored in an inode?
    node is a data structure in a Unix-style file system which describes a filesystem object such as a file or a directory. Each inode stores the attributes and disk block location(s) of the object's data.Filesystem object attributes may include metadata (times of last change,access, modification), as well as owner and permission data.
    Directories are lists of names assigned to inodes. A directory contains an entry for itself, its parent, and each of its children.
  • How to force/trigger a file system check on next reboot?
    touch /forcefsck
    set drive to be checked on every reboot, and return to default value after reboot
    tune2fs -c 1 /dev/sda1
  • What is SNMP and what is it used for?
    Simple Network Management Protocol (SNMP) is a popular protocol for network management. It is used for collecting information from, and configuring, network devices, such as servers, printers, hubs, switches, and routers on an Internet Protocol (IP) network.
  • What is a runlevel and how to get the current runlevel?
    A runlevel is a preset operating state on a Unix-like operating system.
    who -r
    0 - System halt; no activity, the system can be safely powered down. 
    1 - Single user; rarely used. 
    2 - Multiple users, no NFS (network filesystem); also used rarely. 
    3 - Multiple users, command line (i.e., all-text mode) interface; the standard runlevel for most Linux-based server hardware. 
    4 - User-definable 
    5 - Multiple users, GUI (graphical user interface); the standard runlevel for most Linux-based desktop systems. 
    6 - Reboot; used when restarting the system.
    By default Linux boots either to runlevel 3 or to runlevel 5.
  • What is SSH port forwarding?
    SSH port forwarding, or TCP/IP connection tunneling, is a process whereby a TCP/IP connection that would otherwise be insecure is tunneled through a secure SSH link, thus protecting the tunneled connection from network attacks. Port forwarding can be used to establish a form of a virtual private network (VPN).
  • What is the difference between local and remote port forwarding?
    Local Port Forwarding (Outgoing Tunnel):
    Principle: Local host forwards/displays content of remote host. Local host acts as proxy. Tunneling opens a listening socket on localhost and transfers content to remote server
    Command: ssh -L local_port:remote_host:remote_port [email protected]
    Tunnel: local host -(SSH tunnel)→ remote host -(SSH tunnel)→ local host
    Example: check remote host behind load-balancer or firewall on localhost
    Remote Port Forwarding (Incoming Tunnel):
    Principle: remote host forwards content of localhost. Remote host acts as proxy. Tunneling opens a listening socket on the remote server host and transfers the content to the local host
    Command: ssh -R remote_port:local_host:local_port [email protected]
    Tunnel: remote host -(SSH tunnel)→ local host -(SSH tunnel)→ remote host
    Example: make localhost visible in the internet or giving access to a service on your home machine to people at work
  • What are the steps to add a user to a system without using useradd/adduser?
    To create a new account manually, follow these steps:
    Edit /etc/passwd with vipw and add a new line for the new account. Be careful with the syntax. Do not edit directly with an editor. vipw locks the file, so that other commands won't try to update it at the same time. You should make the password field be `*', so that it is impossible to log in.
    Similarly, edit /etc/group with vigr, if you need to create a new group as well.
    Create the home directory of the user with mkdir.
    Copy the files from /etc/skel to the new home directory.
    Fix ownerships and permissions with chown and chmod. The -R option is most useful. The correct permissions vary a little from one site to another, but usually the following commands do the right thing:
    cd /home/newusername
    chown -R .
    chmod -R go=u,go-w .
    chmod go= .
    Set the password with passwd.
    After you set the password in the last step, the account will work. You shouldn't set it until everything else has been done, otherwise the user may inadvertently log in while you're still copying the files.
  • What is MAJOR and MINOR numbers of special files?
    Char devices are accessed through names in the filesystem. Those names are called special files or device files or simply nodes of the filesystem tree; they are conventionally located in the /dev directory. Special files for char drivers are identified by a "c" in the first column of the output of ls -l. Block devices appear in /dev as well, but they are identified by a "b." The focus of this chapter is on char devices, but much of the following information applies to block devices as well.
    If you issue the ls -l command, you'll see two numbers (separated by a comma) in the device file entries before the date of the last modification, where the file length normally appears. These numbers are the major and minor device number for the particular device. The following listing shows a few devices as they appear on a typical system. Their major numbers are 1, 4, 7, and 10, while the minors are 1, 3, 5, 64, 65, and 129.
     crw-rw-rw- 1 root root 1, 3 Apr 11 2002 null
     crw------- 1 root root 10, 1 Apr 11 2002 psaux
     crw------- 1 root root 4, 1 Oct 28 03:04 tty1
     crw-rw-rw- 1 root tty 4, 64 Apr 11 2002 ttys0
     crw-rw---- 1 root uucp 4, 65 Apr 11 2002 ttyS1
     crw--w---- 1 vcsa tty 7, 1 Apr 11 2002 vcs1
     crw--w---- 1 vcsa tty 7, 129 Apr 11 2002 vcsa1
     crw-rw-rw- 1 root root 1, 5 Apr 11 2002 zero
    Traditionally, the major number identifies the driver associated with the device. For example, /dev/null and /dev/zero are both managed by driver 1, whereas virtual consoles and serial terminals are managed by driver 4; similarly, both vcs1 and vcsa1 devices are managed by driver 7. Modern Linux kernels allow multiple drivers to share major numbers, but most devices that you will see are still organized on the one-major-one-driver principle.
    The minor number is used by the kernel to determine exactly which device is being referred to. Depending on how your driver is written (as we will see below), you can either get a direct pointer to your device from the kernel, or you can use the minor number yourself as an index into a local array of devices. Either way, the kernel itself knows almost nothing about minor numbers beyond the fact that they refer to devices implemented by your driver.
  • Describe the mknod command and when you’d use it.
    mknod was originally used to create the character and block devices that populate /dev/. Nowadays software like udev automatically creates and removes device nodes on the virtual filesystem when the corresponding hardware is detected by the kernel, but originally /dev was just a directory in / that was populated during install.
  • Describe a scenario when you get a “filesystem is full” error, but ‘df’ shows there is free space.
    Filesystem run out of inodes.
  • Describe a scenario when deleting a file, but ‘df’ not showing the space being freed.
    df checks freed inodes, and files can be held open and take space after they're deleted.
  • Describe how ‘ps’ works.
    On Linux, the ps command works by reading files in the proc filesystem. The directory /proc/PID contains various files that provide information about process PID. The content of these files is generated on the fly by the kernel when a process reads them.
  • What happens to a child process that dies and has no parent process to wait for it and what’s bad about this?
    When a child exits, some process must wait on it to get its exit code. That exit code is stored in the process table until this happens. The act of reading that exit code is called "reaping" the child. Between the time a child exits and is reaped, it is called a zombie.
    Zombies only occupy space in the process table. They take no memory or CPU. However, the process table is a finite resource, and excessive zombies can fill it, meaning that no other processes can launch.
    If a process exits with children still running, those children are orphans. Orphaned children are immediately "adopted" by init. An orphan is just a process. It will use whatever resources it uses.
  • Explain briefly each one of the process states.
    This is the initial state when a process is first started/created.
    The process is waiting to be assigned to a processor. Ready processes are waiting to have the processor allocated to them by the operating system so that they can run. Process may come into this state after Start state or while running it by but interrupted by the scheduler to assign CPU to some other process.
    Once the process has been assigned to a processor by the OS scheduler, the process state is set to running and the processor executes its instructions.
    Process moves into the waiting state if it needs to wait for a resource, such as waiting for user input, or waiting for a file to become available.
    Terminated or Exit
    Once the process finishes its execution, or it is terminated by the operating system, it is moved to the terminated state where it waits to be removed from main memory.
  • How to know which process listens on a specific port?
    netstat -tulpn | grep :port#
    fuser port#/tcp|udp
    lsof -i :port#
  • What is a zombie process and what could be the cause of it?
    When a process dies on Linux, it isn’t all removed from memory immediately — its process descriptor stays in memory (the process descriptor only takes a tiny amount of memory). The process’s status becomes EXIT_ZOMBIE and the process’s parent is notified that its child process has died with the SIGCHLD signal. The parent process is then supposed to execute the wait() system call to read the dead process’s exit status and other information. This allows the parent process to get information from the dead process. After wait() is called, the zombie process is completely removed from memory.
    This normally happens very quickly, so you won’t see zombie processes accumulating on your system. However, if a parent process isn’t programmed properly and never calls wait(), its zombie children will stick around in memory until they’re cleaned up.
    Zombie processes don’t use up any system resources. (Actually, each one uses a very tiny amount of system memory to store its process descriptor.) However, each zombie process retains its process ID (PID). Linux systems have a finite number of process IDs – 32767 by default on 32-bit systems. If zombies are accumulating at a very quick rate – for example, if improperly programmed server software is creating zombie processes under load — the entire pool of available PIDs will eventually become assigned to zombie processes, preventing other processes from launching.
  • You run a bash script and you want to see its output on your terminal and save it to a file at the same time. How could you do it?
    Using tee command.
    ./ | tee -a file.log
  • Explain what echo “1” > /proc/sys/net/ipv4/ip_forward does.
    Enables IP forwarding, but will not preserve the change after reboot.
  • Describe briefly the steps you need to take in order to create and install a valid certificate for the site
    Make a CSR (certificate signing request) on your web server.
    Purchase a certificate
    Install your SSL certificate and intermediate certificates on your server.
  • Can you have several HTTPS virtual hosts sharing the same IP?
    Yes, if your server supports SNI (Server Name Indication).
    With Apache v2.2.12 and OpenSSL v0.9.8j and later you can use a transport layer security (TLS) called SNI. SNI can secure multiple Apache sites using a single SSL Certificate and use multiple SSL Certificates to secure various websites on a single domain (e.g., or across multiple domains (,—all from a single IP address. The benefits of using SNI are obvious—you can secure more websites without purchasing more IP addresses or additional hardware.
  • What is a wildcard certificate?
    certificate which can be used with multiple subdomains of a domain.
  • Which Linux file types do you know?
    Regular files
    Directory files
    Special files:
      Block file(b)
      Character device file(c)
      Named pipe file or just a pipe file(p)
      Symbolic link file(l)
      Socket file(s)
  • What is the difference between a process and a thread? And parent and child processes after a fork system call?
    The processes and threads are independent sequences of execution, the typical difference is that threads run in a shared memory space, while processes run in separate memory spaces.
    A process has a self contained execution environment that means it has a complete, private set of basic run time resources purticularly each process has its own memory space. Threads exist within a process and every process has at least one thread.
    When a process calls fork, it is deemed the parent process and the newly created process is its child. After the fork, both processes not only run the same program, but they resume execution as though both had called the system call. They can then inspect the call's return value to determine their status, child or parent, and act accordingly.
  • What is the difference between exec and fork?
    The fork call basically makes a duplicate of the current process, identical in almost every way (not everything is copied over, for example, resource limits in some implementations but the idea is to create as close a copy as possible).
    The new process (child) gets a different process ID (PID) and has the the PID of the old process (parent) as its parent PID (PPID). Because the two processes are now running exactly the same code, they can tell which is which by the return code of fork - the child gets 0, the parent gets the PID of the child. This is all, of course, assuming the fork call works - if not, no child is created and the parent gets an error code.
    The exec call is a way to basically replace the entire current process with a new program. It loads the program into the current process space and runs it from the entry point.
  • What is “nohup” used for?
    nohup keep processes running after you exit from a shell.
  • What is the difference between these two commands?
    export myvar=hello
  • How many NTP servers would you configure in your local ntp.conf?
  • What does the column ‘reach’ mean in ntpq -p output?
  • You need to upgrade kernel at 100-1000 servers, how you would do this?
  • How can you get Host, Channel, ID, LUN of SCSI disk?
  • How can you limit process memory usage?
  • What is bash quick substitution/caret replace(^x^y)?
  • Do you know of any alternative shells? If so, have you used any?
  • What is a tarpipe (or, how would you go about copying everything, including hardlinks and special files, from one server to another)?

Hard Linux Questions:

  • What is a tunnel and how you can bypass a http proxy?
  • What is the difference between IDS and IPS?
  • What shortcuts do you use on a regular basis?
  • What is the Linux Standard Base?
  • What is an atomic operation?
  • Your freshly configured http server is not running after a restart, what can you do?
  • What kind of keys are in ~/.ssh/authorized_keys and what it is this file used for?
  • I’ve added my public ssh key into authorized_keys but I’m still getting a password prompt, what can be wrong?
  • Did you ever create RPM’s, DEB’s or solaris pkg’s?
  • What does :(){ :|:& };: do on your system?
  • How do you catch a Linux signal on a script?
  • Can you catch a SIGKILL?
  • What’s happening when the Linux kernel is starting the OOM killer and how does it choose which process to kill first?
  • Describe the linux boot process with as much detail as possible, starting from when the system is powered on and ending when you get a prompt.
  • What’s a chroot jail?
  • When trying to umount a directory it says it’s busy, how to find out which PID holds the directory?
  • What’s LD_PRELOAD and when it’s used?
  • You ran a binary and nothing happened. How would you debug this?
  • What are cgroups? Can you specify a scenario where you could use them?

Expert Linux Questions:

  • A running process gets EAGAIN: Resource temporarily unavailable on reading a socket. How can you close this bad socket/file descriptor without killing the process?

Networking Questions:

  • What is localhost and why would ping localhost fail?
  • What is the similarity between “ping” & “traceroute” ? How is traceroute able to find the hops.
  • What is the command used to show all open ports and/or socket connections on a machine?
  • Is 300.168.0.123 a valid IPv4 address?
  • Which IP ranges/subnets are “private” or “non-routable” (RFC 1918)?
  • What is a VLAN?
  • What is ARP and what is it used for?
  • What is the difference between TCP and UDP?
  • What is the purpose of a default gateway?
  • What is command used to show the routing table on a Linux box?
  • A TCP connection on a network can be uniquely defined by 4 things. What are those things?
  • When a client running a web browser connects to a web server, what is the source port and what is the destination port of the connection?
  • How do you add an IPv6 address to a specific interface?
  • You have added an IPv4 and IPv6 address to interface eth0. A ping to the v4 address is working but a ping to the v6 address gives yout the response sendmsg: operation not permitted. What could be wrong?
  • What is SNAT and when should it be used?
  • Explain how could you ssh login into a Linux system that DROPs all new incoming packets using a SSH tunnel.
  • How do you stop a DDoS attack?
  • How can you see content of an ip packet?
  • What is IPoAC (RFC 1149)?

MySQL questions:

  • How do you create a user?
  • How do you provide privileges to a user?
  • What is the difference between a “left” and a “right” join?
  • Explain briefly the differences between InnoDB and MyISAM.
  • Describe briefly the steps you need to follow in order to create a simple master/slave cluster.
  • Why should you run “mysql_secure_installation” after installing MySQL?
  • How do you check which jobs are running?

DevOps Questions:

  • Can you describe your workflow when you create a script?
  • What is GIT?
  • What is a dynamically/statically linked file?
  • What does “./configure && make && make install” do?
  • What is puppet/chef/ansible used for?
  • What is Nagios/Zenoss/NewRelic used for?
  • What is the difference between Containers and VMs?
  • How do you create a new postgres user?
  • What is a virtual IP address? What is a cluster?
  • How do you print all strings of printable characters present in a file?
  • How do you find shared library dependencies?
  • What is Automake and Autoconf?
  • ./configure shows an error that libfoobar is missing on your system, how could you fix this, what could be wrong?
  • What are the advantages/disadvantages of script vs compiled program?
  • What’s the relationship between continuous delivery and DevOps?
  • What are the important aspects of a system of continuous integration and deployment?

Fun Questions:

  • A careless sysadmin executes the following command: chmod 444 /bin/chmod – what do you do to fix this?
  • I’ve lost my root password, what can I do?
  • I’ve rebooted a remote server but after 10 minutes I’m still not able to ssh into it, what can be wrong?
  • If you were stuck on a desert island with only 5 command-line utilities, which would you choose?
  • You come across a random computer and it appears to be a command console for the universe. What is the first thing you type?
  • Tell me about a creative way that you’ve used SSH?
  • You have deleted by error a running script, what could you do to restore it?
  • What will happen on 19 January 2038?

Demo Time:

  • Unpack test.tar.gz without man pages or google.
  • Remove all “*.pyc” files from testdir recursively?
  • Search for “my konfu is the best” in all *.py files.
  • Replace the occurrence of “my konfu is the best” with “I’m a linux jedi master” in all *.txt files.
  • Test if port 443 on a machine with IP address X.X.X.X is reachable.
  • Get http://myinternal.webserver.local/test.html via telnet.
  • How to send an email without a mail client, just on the command line?
  • Write a get_prim method in python/perl/bash/pseudo.
  • Find all files which have been accessed within the last 30 days.
  • Explain the following command (date ; ps -ef | awk '{print $1}' | sort | uniq | wc -l ) >> Activity.log
  • Write a script to list all the differences between two directories.
  • In a log file with contents as <TIME> : [MESSAGE] : [ERROR_NO] - Human readable text display summary/count of specific error numbers that occurred every hour or a specific hour.