After short research, I decided to code one — for fun and education. It’s designed to operate in a multi-user way, so if you want to give it a try, go on!

My online implementation is named “xs RSS reader“, short for extra-simple RSS reader:

http://www.famzah.net/xs-rss-reader/

Here is a sample demo screenshot:

Recently the machine started to exhaust its CPU resources. First we overclocked it a bit which gave us 10% more CPU idle time. Then we decided to try to compile Nagios with the popen-noshell library. This gave us another 10% CPU idle and now the machine is working great again.

I’ll focus on the popen-noshell integration and results, since CPU overclocking is a well-known topic. Here is the chart which shows the CPU usage before and after we re-compiled Nagios with the popen-noshell library:

As we can see, the system-CPU usage dropped from 38% to 31%, which is an 18% improvement. The user-CPU usage dropped from 44% to 41%, which is a 7% improvement. Overall, we gained a 12% speed-up for our workload by just re-compiling Nagios with the popen-noshell library. I’m stressing out that the speed-up depends a lot on your workload. If this machine was busy only with Nagios and the active checks were more CPU efficient (i.e. not written in Perl but in C), then the speed-up could have been much higher, since popen_noshell() is about 10 times faster than the standard popen().

A list with the other machine metrics which were also affected by the workload change:

• Used memory: 39% => 24% (38% less)
• Load average: 39 => 46 (18% higher)
• Forks rates: 8*61 => 8*61 (created processes/second – no change)

Here are the steps that you need to perform, in order to re-compile the Nagios Debian package by integrating it with the popen-noshell library:

apt-get install devscripts

apt-get build-dep nagios3-core
# No need to run as "root" from here on
apt-get source nagios3-core

svn checkout http://popen-noshell.googlecode.com/svn/trunk/ popen-noshell

cd nagios3-3.2.1/

# BEGIN: patch Nagios to use popen_noshell_compat()

cp ../popen-noshell/popen_noshell.* base/
vi base/Makefile.in
	OBJS=$(BROKER_O) popen_noshell.o 

vi base/utils.c
	#include "popen_noshell.h"
	
        /* run the command */
        struct popen_noshell_pass_to_pclose pclose_arg;
        fp=(FILE *)popen_noshell_compat(cmd,"r",&pclose_arg);

            /* close the command and get termination status */
            status=pclose_noshell(&pclose_arg);

vi base/checks.c
	2x the same as above

# END: patch Nagios to use popen_noshell_compat()

EDITOR=vim dch -i
	# 3.2.1-2+squeeze1 -> 3.2.1-2+squeeze1-noshell1
	# you must have a trailing number in the added version name
	# after exit, this renames the original directory name

cd ..
mv nagios3_3.2.1.orig.tar.gz nagios3_3.2.1-2+squeeze1.orig.tar.gz

# the source directory was renamed by "dch"
cd nagios3-3.2.1-2+squeeze1/
DEB_BUILD_OPTIONS=nocheck debuild -us -uc

cd ..
sudo dpkg -i nagios3-core_3.2.1-2+squeeze1-noshell1_i386.deb \
	nagios3-common_3.2.1-2+squeeze1-noshell1_all.deb \
	nagios3-cgi_3.2.1-2+squeeze1-noshell1_i386.deb \
	nagios3-doc_3.2.1-2+squeeze1-noshell1_all.deb \
	nagios3_3.2.1-2+squeeze1-noshell1_i386.deb

Reading a bit more brought me to the astonishing fact that “xargs” can run commands in parallel. The “xargs” utility is something I use every day and this parallelism feature made it even more useful.

Let’s try it by running the following:

famzah@vbox:~$ echo 10 20 30 40 50 60 | xargs -n 1 -P 4 sleep

The use of “-n 1″ is vital if you want to pass only one command-line argument from the list to each parallel process.

Here is the result:

# right after we launched "xargs"
famzah@vbox:~$ ps f -o pid,command
  PID COMMAND
 5068 /bin/bash
 7007  \_ xargs -n 1 -P 4 sleep
 7008      \_ sleep 10
 7009      \_ sleep 20
 7010      \_ sleep 30
 7011      \_ sleep 40

# 10 seconds later (the first "sleep" has just exited)
famzah@vbox:~$ ps f -o pid,command
  PID COMMAND
 5068 /bin/bash
 7007  \_ xargs -n 1 -P 4 sleep
 7009      \_ sleep 20
 7010      \_ sleep 30
 7011      \_ sleep 40
 7017      \_ sleep 50

# 20 seconds later (the second and third "sleep" commands have exited)
# we now have only 3 simultaneous processes (no more arguments to process)
famzah@vbox:~$ ps f -o pid,command
  PID COMMAND
 5068 /bin/bash
 7007  \_ xargs -n 1 -P 4 sleep
 7011      \_ sleep 40
 7017      \_ sleep 50
 7023      \_ sleep 60

It’s worth mentioning that if “xargs” fails to execute the binary, it prematurely terminates the failed parallel processing queue, which leaves some of the stdin arguments not processed:

famzah@vbox:~$ echo 10 20 30 40 50 60 | xargs -n 1 -P 4 badexec-name
xargs: badexec-namexargs: badexec-name: No such file or directory: No such file or directory

xargs: badexec-namexargs: badexec-name: No such file or directory
: No such file or directory

The output is scrambled because all parallel processes write to the screen with no locking synchronization. This seems to be a known issue. The point is that we could expect that “xargs” would try to execute “badexec-name” for every command-line argument (total of six attempts in our example). It turns out that “xargs” bails out the same way even if we don’t use the “-P” option:

# standard usage of "xargs"
famzah@vbox:~$ echo 10 20 30 40 50 60 | xargs -n 1 badexec-name
xargs: badexec-name: No such file or directory

Not a very cool behavior. I’ve reported this as a bug to the GNU community. If you review the responses to the bug report, you will find out that this actually is an intended feature.

If the provided command to “xargs” is a valid one but it fails during the execution, there are no surprises and “xargs” continues with the next command-line argument by executing a new command:

famzah@vbox:~$ echo 10 20 30 40 50 60 | xargs -n 1 -P 4 rm
rm: rm: cannot remove `10'cannot remove `40': No such file or directory
: No such file or directory
rm: cannot remove `20': No such file or directory
rm: cannot remove `30': No such file or directory
rm: cannot remove `60': No such file or directory
rm: cannot remove `50': No such file or directory

The output here is scrambled too because all parallel processes write to the screen with no locking synchronization. We see however that all command-line arguments from “10″ to “60″ were processed by executing a command for each of them.

RESULT="$(echo "$LINE"| awk '{print $1}')" # executes in a sub-shell 

Processing thousands of lines this way however fork()’s thousands of processes, which affects performance and makes your script CPU hungry.

Here is the effective solution which I found with my colleagues at work:

COLS=( $LINE ); # parses columns without executing a sub-shell
RESULT="${COLS[0]}"; # returns first column (0-based indexes)

Here is an example:

LINE="col0 col1  col2     col3  col4      " # white-space including tab chars
COLS=( $LINE ); # parses columns without executing a sub-shell

echo "${COLS[0]}"; # prints "col0"
echo "${COLS[1]}"; # prints "col1"
echo "${COLS[2]}"; # prints "col2"
echo "${COLS[3]}"; # prints "col3"
echo "${COLS[4]}"; # prints "col4"

If you want to split not by white-space but by any other character, you can temporarily change the IFS variable which determines how Bash recognizes fields and word boundaries.

• Net::Ping with ICMP protocol, everything else is defaults: “$p = new Net::Ping(‘icmp’)“, no replies from remote host, note that the length is just 8 bytes:
  
12:29:02.898083 IP source-addr > source-addr: ICMP echo request, id 2194, seq 41, length 8
12:29:03.711595 IP source-addr > dest-addr: ICMP echo request, id 2194, seq 42, length 8

• Linux “ping” command-line utility, remote host replies accordingly, the length is 64 bytes total:
  
12:30:18.278865 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 1, length 64
12:30:18.289922 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 1, length 64
12:30:18.790610 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 2, length 64
12:30:18.811029 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 2, length 64

• Net::Ping with ICMP protocol with user-defined length, “$p = new Net::Ping(‘icmp’, 1, 56)“, remote host replies accordingly, the length is 64 bytes total:
  
12:30:48.377496 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 6, length 64
12:30:48.433690 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 6, length 64
12:30:48.934310 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 7, length 64
12:30:48.946152 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 7, length 64


Bottom line is that if you are going to use Net::Ping with ICMP, specify 56 for the “bytes” parameter when creating an instance of the Net::Ping object. This way you will be sending standard ICMP requests with total lenght of 64 bytes.

The authenticity of host '176.34.91.245 (176.34.91.245)' can't be established.
...
Are you sure you want to continue connecting (yes/no)? yes

Besides from being annoying, it is also a security risk to blindly accept this warning and continue connecting. And be honest — almost none of us check the fingerprint in advance every time.

A common scenario for this use case is when you have an EC2 server in Amazon AWS which you temporarily stop and then start, in order to cut costs. I have a backup server which I use in this way.

In order to securely avoid this SSH warning and still be sure that you connect to your trusted server, you have to save the fingerprint in a separate file and update the IP address in it every time before you connect. Here are the connect commands, which you can also encapsulate in a Bash wrapper script:

IP=176.34.91.245 # use an IP address here, not a hostname
FPFILE=~/.ssh/aws-backup-server.fingerprint

test -e "$FPFILE" && perl -pi -e "s/^\S+ /$IP /" "$FPFILE"
ssh -o StrictHostKeyChecking=ask -o UserKnownHostsFile="$FPFILE" root@$IP

Note that the FPFILE is not required to exist on the first SSH connect. The first time you connect to the server, the FPFILE will be created when you accept the SSH warning. Further connects will not show an SSH warning or ask you to accept the fingerprint again.

My initial impression was that iSCSI over Internet is not usable as a backup device even though my Internet connection is relatively fast — a simple ext4 file-system format took about 24 minutes. I though that the connection latency is killing the performance. Well, I was wrong. Even after making latency two times lower by working on a server which was geographically closer, the ext4 format still took 24 minutes.

Eventually I did some tests and analysis, and finally started to use the iSCSI over Internet volume for backup purposes — and it works flawlessly so far.

Ext4 format benchmark

It turns out that it’s not the latency but my upload bandwidth which was slowing things down:

• 1 Mbit/s upload Internet connection and Ping latency of 75 ms:
  • Time: 24 minutes.
  • Average transfer rates snapshot:
    • Total rates: 967.7 kbits/sec (212.6 packets/sec).
    • Incoming rates: 83.0 kbits/sec (92.8 packets/sec).
    • Outgoing rates: 884.6 kbits/sec (119.8 packets/sec).
  • About 200 MBytes outgoing transfer; only 12 MBytes incoming transfer (no SSH tunnel compression).
  • About 200.000 packets sent and about 130.000 received.
• 3 Mbit/s upload Internet connection and Ping latency of 75 ms:
  • Time: 8 minutes.
  • Average transfer rates snapshot:
    • Total rates: 2580.0 kbits/sec (417.8 packets/sec).
    • Incoming rates: 128.5 kbits/sec (149.6 packets/sec).
    • Outgoing rates: 2451.5 kbits/sec (268.2 packets/sec).
  • About 160 MBytes outgoing transfer; only 9 MBytes incoming transfer (with SSH tunnel compression).
  • About 140.000 packets sent and about 80.000 received.

I know I’m missing two tests with and without SSH tunnel compression but it seems compression doesn’t make such a difference. It’s upload speed which affects the total completion time.

File copy benchmark

All tests were done without SSH compression and we make the same conclusion — it is bandwidth which affects the total completion time:

• 1 Mbit/s upload Internet connection and Ping latency of 75 ms:
  • SSH direct file copy to server: 100 seconds (11 MBytes file).
  • File copy to an iSCSI mounted file-system: 105 seconds.
• 3 Mbit/s upload Internet connection and Ping latency of 75 ms:
  • SSH direct file copy to server: 39 seconds (11 MBytes file).
  • File copy to an iSCSI mounted file-system: 39 seconds.

The SSH direct file copy (SCP) transfer command was “scp testf root@172.18.0.1:/tmp/”, and the file copy command was “cp testf /mnt/ ; sync”.

Server and client load during transfer, other benchmarks

During the transfer both the client and server machines were almost idle in regards to CPU. The iSCSI block storage device on the server was not saturated even at 1%.

Note that the iSCSI target was exported via an SSH tunnel, as described here. Ping tests shown no difference between a direct server ping and a ping via the SSH tunnel.

The file copy tests were done on a regular iSCSI mounted volume, and on an iSCSI volume which was encrypted using TrueCrypt. The same speeds were achieved.

Encountered problems

During the backup runs, I got several of the following kernel messages in “dmesg”. This seems like a normal warning for the iSCSI use-case scenario:

[13200.272157] INFO: task jbd2/dm-0-8:1931 blocked for more than 120 seconds.
[13200.272164] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[13200.272168] jbd2/dm-0-8 D f2abdc80 0 1931 2 0×00000000

• Amazon staff would have access to my data.
• A breach in Amazon’s systems would expose my data.
• A breach in my remote server OS would expose my data.

One of the solutions which I considered was to encrypt my local file-system with eCryptfs but it has some issues with relatively long file names.

Finally I came out with the following working backup solution which I currently use to backup both my Windows and Linux partitions. I share the Windows root directory with the VirtualBox Linux machine and run the backup scripts from there. Here is a short explanation of the properties and features of the backup setup:

• Locally encrypted — all files which I store on the iSCSI volume are encrypted on my personal desktop, before being sent to the remote server. This ensures that the files cannot be read by anyone else.
• Secure — besides the local volume encryption, the whole communication is done over an SSH tunnel which secures the Internet point-to-point client-to-server communication.
• Remote — having a remote backup ensures that even if someone breached in my house and steals my laptop and my offline backup, I can still recover my data from the remote server. Furthermore, it is more convenient to frequently backup on a remote machine, because we have Internet access everywhere now. Note that remote backups are not a substitution for offline backups.
• Over Internet — very convenient. Of course, this backup scheme can be used in any TCP/IP network — private LAN, WAN, VPN networks, etc.

The following two articles provide detailed instructions on how to setup the backup solution:

• Secure iSCSI setup via an SSH tunnel on Linux
• Locally encrypt an iSCSI volume with TrueCrypt on Linux

Daily usage example

Here are the commands which I execute, in order to make a backup of my laptop. Those can be further scripted and automated if a daily or more frequent backup is required:

IP=23.21.98.10 # the public DNS IP address of the EC2 instance / server

## Execute the following, in order to mount the remote encrypted iSCSI volume:

sudo -E \
  ssh -F /dev/null \
  -o PermitLocalCommand=yes \
  -o LocalCommand="ifconfig tun0 172.18.0.2 pointopoint 172.18.0.1 netmask 255.255.255.0" \
  -o ServerAliveInterval=60 \
  -w 0:0 root@"$IP" \
  'sudo ifconfig tun0 172.18.0.1 pointopoint 172.18.0.2 netmask 255.255.255.0; hostname; echo tun0 ready'

sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --login
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/sdb
sudo mount /dev/mapper/truecrypt1 /mnt

## You can now work on /mnt -- make a backup, copy files, etc.

ls -la /mnt

## Execute the following, in order to unmount the encrypted iSCSI volume:

sync
sudo umount /mnt
sudo truecrypt -d /dev/sdb
sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --logout
# stop the SSH tunnel

Disaster recovery plan

Any backup is useless if you cannot restore your data. If your main computer is totally out, you would need the following, in order to access your backed up data:

• The Amazon AWS login credentials to start the EC2 backup instance.
• The root password for the EC2 backup instance, so that you can log in there and then access your backup data by mounting the encrypted iSCSI volume locally on the remote server.

In order to be able to log in to the remote server via SSH, you need to set up the following:

vi /etc/ssh/sshd_config # PasswordAuthentication yes
/etc/init.d/ssh restart
passwd root # set a very long password which you CAN remember

Make sure that you test if you can log in using an SSH client which does not have your SSH key and thus requires you to enter the root password manually.

I do not consider password authentication for the root account to be a security threat here. The backup server is online only during the time a backup is being made, after which I shut it down in order to save money from Amazon AWS. Furthermore, the backup has a new IP address on each new EC2 machine start, so an attacker cannot continue a brute-force attack easily, even if they started it.

Locally encrypting a remote iSCSI volume with TrueCrypt has the following advantages:

• You don’t need to trust the administrators of the remote machine — they cannot see your files because you are using their storage in a locally encrypted format. Thus your private data is completely safe, as long as your encryption password/key is strong enough.
• You have the option to temporarily mount the exported iSCSI volume on the remote server, if you are the owner of the remote server and know the encryption password/key. This is handy if you want to make a local copy of a file from the backup volume without storing the encryption password on the remote server.
• TrueCrypt is cross-platform (Windows / Mac OS X / Linux), fast, free, and open-source.

Download and install TrueCrypt

You need to install TrueCrypt wherever you are going to use it — on the client machine and optionally on the server.

# Download the distribution file from the official page:
#   http://www.truecrypt.org/downloads
# Linux -> Console-only (choose 32-bit or 64-bit depending on your local Linux installation)

tar -zxf truecrypt-7.1a-linux-console-x86.tar.gz # 32-bit in this example
sudo ./truecrypt-7.1a-setup-console-x86

truecrypt --version
#>> TrueCrypt 7.1a

Encrypt an iSCSI volume with TrueCrypt

The instructions below assume that the iSCSI volume is attached under “/dev/sdb“. The output of the commands is quoted with “#>>”.

# Encrypt the iSCSI volume
sudo truecrypt -t --create /dev/sdb --volume-type=normal --encryption=AES --hash=RIPEMD-160 --filesystem=ext4 --quick -k ""

# Mount the *volume* (there is no file-system, yet)
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/sdb

# Check that a new "dm-0" device with the same size appeared
cat /proc/partitions
#>> major minor  #blocks  name
#>> ...
#>> 8        16  83886080 sdb
#>> 252       0  83885824 dm-0

# Double-check that this is a TrueCrypt volume
ls -la /dev/mapper/truecrypt1
# /dev/mapper/truecrypt1 -> ../dm-0

# Create a file-system.
# This takes about 30 min for a 80 GB volume @ 1 MBit Internet connection.
sudo mkfs.ext4 /dev/mapper/truecrypt1

# You can now mount and use /dev/mapper/truecrypt1 in any mount-point, as 
# this is a regular block device with an ext4 file-system.
# Remember to unmount it when you are done.
mount /dev/mapper/truecrypt1 /mnt
ls -la /mnt
umount /mnt

# Unmount the encrypted *volume*.
# Make sure that you have ALREADY unmounted the file-system!
sync
sudo truecrypt -d /dev/xvdf

Mount an encrypted iSCSI volume locally on the remote server

The output of the commands is quoted with “#>>”.

# The local block device is "/dev/xvdf"
cat /proc/partitions 
#>> major minor  #blocks  name
#>> ...
#>>   202    80  83886080 xvdf

#
# MAKE SURE that no iSCSI clients are using the volume now
#

# Mount an encrypted volume (/dev/xvdf).
# The unencrypted volume will be presented under a different device name (/dev/mapper/truecrypt1).
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/xvdf

# Mount the file-system
sudo mount /dev/mapper/truecrypt1 /mnt
# Access the encrypted files
ls -la /mnt
# Unmount the file-system
sudo umount /mnt

# Unmount the encrypted volume (/dev/mapper/truecrypt1 -> /dev/xvdf).
# Make sure that you have ALREADY unmounted the file-system!
sudo truecrypt -d /dev/xvdf