RESULT="$(echo "$LINE"| awk '{print $1}')" # executes in a subshell
Processing thousands of lines this way however fork()’s thousands of processes, which affects performance and makes your script CPU hungry.
Here is a more efficient way to do it:
LINE="col0 col1 col2 col3 col4 "
COLS=()
for val in $LINE ; do
COLS+=("$val")
done
echo "${COLS[0]}"; # prints "col0"
echo "${COLS[1]}"; # prints "col1"
echo "${COLS[2]}"; # prints "col2"
echo "${COLS[3]}"; # prints "col3"
echo "${COLS[4]}"; # prints "col4"
If you want to split not by white-space but by any other character, you can temporarily change the IFS variable which determines how Bash recognizes fields and word boundaries.
P.S. For the record, here is the old solution:
#
# OLD CODE
# Update: Aug/2016: I've encountered a bug in Bash where this splitting doesn't work as expected! Please see the comments below.
#
# Here is the effective solution which I found with my colleagues at work:
COLS=( $LINE ); # parses columns without executing a subshell
RESULT="${COLS[0]}"; # returns first column (0-based indexes)
# Here is an example:
LINE="col0 col1 col2 col3 col4 " # white-space including tab chars
COLS=( $LINE ); # parses columns without executing a subshell
echo "${COLS[0]}"; # prints "col0"
echo "${COLS[1]}"; # prints "col1"
echo "${COLS[2]}"; # prints "col2"
echo "${COLS[3]}"; # prints "col3"
echo "${COLS[4]}"; # prints "col4"
Here is my success story on how to re-compile a Debian/Ubuntu kernel, in order to enable or tune kernel features which are not available as kernel modules:
# Install required software for the kernel compilation
apt-get install fakeroot build-essential devscripts
apt-get build-dep linux-image-$(uname -r) # make sure you have the appropriate "deb-src" in "sources.list"
apt-get install libncurses5-dev # required for "make menuconfig"
apt-get install ccache # to re-compile the kernel faster (http://wiki.debian.org/OverridingDSDT)
# Prepare some environent variables for our architecture, for later use
ARCH=$(uname -r|cut -d- -f3)
CPUCNT=$(( $(cat /proc/cpuinfo |egrep ^processor |wc -l) * 2))
# Get the kernel sources
rm -rf /root/krebuild && mkdir /root/krebuild
cd /root/krebuild
apt-get source linux-image-$(uname -r)
cd linux-$(uname -r|cut -d- -f1|cut -d. -f1-2)* # cd linux-3.2.20
# http://kernel-handbook.alioth.debian.org/ch-common-tasks.html # 4.2.5 Building packages for one flavour
# The target in this command has the general form of target_arch_featureset_flavour. Replace the featureset with none if you do not want any of the extra featuresets.
# Prepare a Debian kernel to compile
fakeroot make -f debian/rules.gen setup_${ARCH}_none_${ARCH} >/dev/null
cd debian/build/build_${ARCH}_none_${ARCH}
make menuconfig # make any kernel config changes now
cd ../../..
# No debug info => faster kernel build
perl -pi -e 's/debug-info:\s+true/debug-info: false/' debian/config/$ARCH/defines
echo binary-arch_${ARCH}_none_${ARCH}
vi debian/rules.gen # find the Make target and change DEBUG and DEBUG_INFO to False/n respectively
# Bugfix: http://lists.debian.org/debian-user/2008/02/msg01455.html
vi debian/bin/buildcheck.py +51 # add "return 0" right after "def __call__(self, out):"
# Compile the kernel
time DEBIAN_KERNEL_USE_CCACHE=true DEBIAN_KERNEL_JOBS=$CPUCNT \
fakeroot make -j$CPUCNT -f debian/rules.gen binary-arch_${ARCH}_none_${ARCH} > compile-progress.log
# If needed, the linux-headers-version-common binary package (http://kernel-handbook.alioth.debian.org/ch-common-tasks.html -> 4.2.5)
#fakeroot make -j$CPUCNT -f debian/rules.gen binary-arch_${ARCH}_none_real
# Install the newly compiled kernel
cd ..
dpkg -i linux-image-*.deb
#dpkg -i linux-headers-*.deb # only if you need them and/or have them installed already
If you tried to ping a host with Perl Net::Ping using the ICMP protocol and that failed, even though the “ping” command-line utility can ping the host, you’re not alone 🙂 I had the same problem and it turned out to be due to the fact that Net::Ping by default sends no DATA in the ICMP request and thus its requests are rather short and non-standard. Here are some tcpdump examples:
Net::Ping with ICMP protocol, everything else is defaults: “$p = new Net::Ping(‘icmp’)“, no replies from remote host, note that the length is just 8 bytes:
12:29:02.898083 IP source-addr > source-addr: ICMP echo request, id 2194, seq 41, length 8
12:29:03.711595 IP source-addr > dest-addr: ICMP echo request, id 2194, seq 42, length 8
Linux “ping” command-line utility, remote host replies accordingly, the length is 64 bytes total:
12:30:18.278865 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 1, length 64
12:30:18.289922 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 1, length 64
12:30:18.790610 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 2, length 64
12:30:18.811029 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 2, length 64
Net::Ping with ICMP protocol with user-defined length, “$p = new Net::Ping(‘icmp’, 1, 56)“, remote host replies accordingly, the length is 64 bytes total:
12:30:48.377496 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 6, length 64
12:30:48.433690 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 6, length 64
12:30:48.934310 IP source-addr > dest-addr: ICMP echo request, id 2488, seq 7, length 64
12:30:48.946152 IP dest-addr > source-addr: ICMP echo reply, id 2488, seq 7, length 64
Bottom line is that if you are going to use Net::Ping with ICMP, specify 56 for the “bytes” parameter when creating an instance of the Net::Ping object. This way you will be sending standard ICMP requests with total length of 64 bytes.
If you have servers that change their IP address, you’ve probably already been used to the following SSH warning:
The authenticity of host '176.34.91.245 (176.34.91.245)' can't be established.
...
Are you sure you want to continue connecting (yes/no)? yes
Besides from being annoying, it is also a security risk to blindly accept this warning and continue connecting. And be honest — almost none of us check the fingerprint in advance every time.
A common scenario for this use case is when you have an EC2 server in Amazon AWS which you temporarily stop and then start, in order to cut costs. I have a backup server which I use in this way.
In order to securely avoid this SSH warning and still be sure that you connect to your trusted server, you have to save the fingerprint in a separate file and update the IP address in it every time before you connect. Here are the connect commands, which you can also encapsulate in a Bash wrapper script:
IP=176.34.91.245 # use an IP address here, not a hostname
FPFILE=~/.ssh/aws-backup-server.fingerprint
test -e "$FPFILE" && perl -pi -e "s/^\S+ /$IP /" "$FPFILE"
ssh -o StrictHostKeyChecking=ask -o UserKnownHostsFile="$FPFILE" root@$IP
Note that the FPFILE is not required to exist on the first SSH connect. The first time you connect to the server, the FPFILE will be created when you accept the SSH warning. Further connects will not show an SSH warning or ask you to accept the fingerprint again.
My initial impression was that iSCSI over Internet is not usable as a backup device even though my Internet connection is relatively fast — a simple ext4 file-system format took about 24 minutes. I though that the connection latency is killing the performance. Well, I was wrong. Even after making latency two times lower by working on a server which was geographically closer, the ext4 format still took 24 minutes.
Eventually I did some tests and analysis, and finally started to use the iSCSI over Internet volume for backup purposes — and it works flawlessly so far.
Ext4 format benchmark
It turns out that it’s not the latency but my upload bandwidth which was slowing things down:
1 Mbit/s upload Internet connection and Ping latency of 75 ms:
About 160 MBytes outgoing transfer; only 9 MBytes incoming transfer (with SSH tunnel compression).
About 140.000 packets sent and about 80.000 received.
I know I’m missing two tests with and without SSH tunnel compression but it seems compression doesn’t make such a difference. It’s upload speed which affects the total completion time.
File copy benchmark
All tests were done without SSH compression and we make the same conclusion — it is bandwidth which affects the total completion time:
1 Mbit/s upload Internet connection and Ping latency of 75 ms:
SSH direct file copy to server: 100 seconds (11 MBytes file).
File copy to an iSCSI mounted file-system: 105 seconds.
3 Mbit/s upload Internet connection and Ping latency of 75 ms:
SSH direct file copy to server: 39 seconds (11 MBytes file).
File copy to an iSCSI mounted file-system: 39 seconds.
The SSH direct file copy (SCP) transfer command was “scp testf root@172.18.0.1:/tmp/”, and the file copy command was “cp testf /mnt/ ; sync”.
Server and client load during transfer, other benchmarks
During the transfer both the client and server machines were almost idle in regards to CPU. The iSCSI block storage device on the server was not saturated even at 1%.
Note that the iSCSI target was exported via an SSH tunnel, as described here. Ping tests shown no difference between a direct server ping and a ping via the SSH tunnel.
The file copy tests were done on a regular iSCSI mounted volume, and on an iSCSI volume which was encrypted using TrueCrypt. The same speeds were achieved.
Encountered problems
During the backup runs, I got several of the following kernel messages in “dmesg”. This seems like a normal warning for the iSCSI use-case scenario:
[13200.272157] INFO: task jbd2/dm-0-8:1931 blocked for more than 120 seconds.
[13200.272164] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[13200.272168] jbd2/dm-0-8 D f2abdc80 0 1931 2 0x00000000
Recently I decided to start using Amazon AWS as my backup storage but my paranoid soul wasn’t satisfied until I figured it out how to secure my private data. It’s not that I don’t trust Amazon but a lot of bad things could happen if I decided that I just copy my data to a remote server on Amazon:
Amazon staff would have access to my data.
A breach in Amazon’s systems would expose my data.
A breach in my remote server OS would expose my data.
Finally I came out with the following working backup solution which I currently use to backup both my Windows and Linux partitions. I share the Windows root directory with the VirtualBox Linux machine and run the backup scripts from there. Here is a short explanation of the properties and features of the backup setup:
Locally encrypted — all files which I store on the iSCSI volume are encrypted on my personal desktop, before being sent to the remote server. This ensures that the files cannot be read by anyone else.
Secure — besides the local volume encryption, the whole communication is done over an SSH tunnel which secures the Internet point-to-point client-to-server communication.
Remote — having a remote backup ensures that even if someone breached in my house and steals my laptop and my offline backup, I can still recover my data from the remote server. Furthermore, it is more convenient to frequently backup on a remote machine, because we have Internet access everywhere now. Note that remote backups are not a substitution for offline backups.
Over Internet — very convenient. Of course, this backup scheme can be used in any TCP/IP network — private LAN, WAN, VPN networks, etc.
The following two articles provide detailed instructions on how to setup the backup solution:
Here are the commands which I execute, in order to make a backup of my laptop. Those can be further scripted and automated if a daily or more frequent backup is required:
IP=23.21.98.10 # the public DNS IP address of the EC2 instance / server
## Execute the following, in order to mount the remote encrypted iSCSI volume:
sudo -E \
ssh -F /dev/null \
-o PermitLocalCommand=yes \
-o LocalCommand="ifconfig tun0 172.18.0.2 pointopoint 172.18.0.1 netmask 255.255.255.0" \
-o ServerAliveInterval=60 \
-w 0:0 root@"$IP" \
'sudo ifconfig tun0 172.18.0.1 pointopoint 172.18.0.2 netmask 255.255.255.0; hostname; echo tun0 ready'
sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --login
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/sdb
sudo mount /dev/mapper/truecrypt1 /mnt
## You can now work on /mnt -- make a backup, copy files, etc.
ls -la /mnt
## Execute the following, in order to unmount the encrypted iSCSI volume:
sync
sudo umount /mnt
sudo truecrypt -d /dev/sdb
sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --logout
# stop the SSH tunnel
Disaster recovery plan
Any backup is useless if you cannot restore your data. If your main computer is totally out, you would need the following, in order to access your backed up data:
The Amazon AWS login credentials to start the EC2 backup instance.
In order to be able to log in to the remote server via SSH, you need to set up the following:
vi /etc/ssh/sshd_config # PasswordAuthentication yes
/etc/init.d/ssh restart
passwd root # set a very long password which you CAN remember
Make sure that you test if you can log in using an SSH client which does not have your SSH key and thus requires you to enter the root password manually.
I do not consider password authentication for the root account to be a security threat here. The backup server is online only during the time a backup is being made, after which I shut it down in order to save money from Amazon AWS. Furthermore, the backup has a new IP address on each new EC2 machine start, so an attacker cannot continue a brute-force attack easily, even if they started it.
While this article focuses on iSCSI volumes, it also applies for regular directly attached block devices. If you are in doubt on how to export and attach an iSCSI volume over Internet, you can review the “Secure iSCSI setup via an SSH tunnel on Linux” article.
Locally encrypting a remote iSCSI volume with TrueCrypt has the following advantages:
You don’t need to trust the administrators of the remote machine — they cannot see your files because you are using their storage in a locally encrypted format. Thus your private data is completely safe, as long as your encryption password/key is strong enough.
You have the option to temporarily mount the exported iSCSI volume on the remote server, if you are the owner of the remote server and know the encryption password/key. This is handy if you want to make a local copy of a file from the backup volume without storing the encryption password on the remote server.
TrueCrypt is cross-platform (Windows / Mac OS X / Linux), fast, free, and open-source.
Download and install TrueCrypt
You need to install TrueCrypt wherever you are going to use it — on the client machine and optionally on the server.
# Download the distribution file from the official page:
# http://www.truecrypt.org/downloads
# Linux -> Console-only (choose 32-bit or 64-bit depending on your local Linux installation)
tar -zxf truecrypt-7.1a-linux-console-x86.tar.gz # 32-bit in this example
sudo ./truecrypt-7.1a-setup-console-x86
truecrypt --version
#>> TrueCrypt 7.1a
Encrypt an iSCSI volume with TrueCrypt
The instructions below assume that the iSCSI volume is attached under “/dev/sdb“. The output of the commands is quoted with “#>>”.
# Encrypt the iSCSI volume
sudo truecrypt -t --create /dev/sdb --volume-type=normal --encryption=AES --hash=RIPEMD-160 --filesystem=ext4 --quick -k ""
# Mount the *volume* (there is no file-system, yet)
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/sdb
# Check that a new "dm-0" device with the same size appeared
cat /proc/partitions
#>> major minor #blocks name
#>> ...
#>> 8 16 83886080 sdb
#>> 252 0 83885824 dm-0
# Double-check that this is a TrueCrypt volume
ls -la /dev/mapper/truecrypt1
# /dev/mapper/truecrypt1 -> ../dm-0
# Create a file-system.
# This takes about 30 min for a 80 GB volume @ 1 MBit Internet connection.
sudo mkfs.ext4 /dev/mapper/truecrypt1
# You can now mount and use /dev/mapper/truecrypt1 in any mount-point, as
# this is a regular block device with an ext4 file-system.
# Remember to unmount it when you are done.
mount /dev/mapper/truecrypt1 /mnt
ls -la /mnt
umount /mnt
# Unmount the encrypted *volume*.
# Make sure that you have ALREADY unmounted the file-system!
sync
sudo truecrypt -d /dev/xvdf
Mount an encrypted iSCSI volume locally on the remote server
The output of the commands is quoted with “#>>”.
# The local block device is "/dev/xvdf"
cat /proc/partitions
#>> major minor #blocks name
#>> ...
#>> 202 80 83886080 xvdf
#
# MAKE SURE that no iSCSI clients are using the volume now
#
# Mount an encrypted volume (/dev/xvdf).
# The unencrypted volume will be presented under a different device name (/dev/mapper/truecrypt1).
sudo truecrypt --filesystem=none -k "" --protect-hidden=no /dev/xvdf
# Mount the file-system
sudo mount /dev/mapper/truecrypt1 /mnt
# Access the encrypted files
ls -la /mnt
# Unmount the file-system
sudo umount /mnt
# Unmount the encrypted volume (/dev/mapper/truecrypt1 -> /dev/xvdf).
# Make sure that you have ALREADY unmounted the file-system!
sudo truecrypt -d /dev/xvdf
This article will demonstrate how to export a raw block storage device over Internet in a secure manner. Re-phrased this means that you can export a hard disk from a remote machine and use it on your local computer as it was a directly attached disk, thanks to iSCSI. Authentication and secure transport channel is provided by an SSH tunnel (more info). The setup has been tested on Ubuntu 11.10 Oneiric.
Server provisioning
Amazon AWS made it really simple to deploy a server setup in a minute:
Launch a Micro EC2 instance and then install Ubuntu server by clicking on the links in the Ubuntu EC2StartersGuide, section “Official Ubuntu Cloud Guest Amazon Machine Images (AMIs)”.
Create an EBS volume in the same availability zone. Attach it to the EC2 instance as “/dev/sdf” (seen as “/dev/xvdf” in latest Ubuntu versions).
(optionally) Allocate an Elastic IP address and associate it with the EC2 instance.
Note that you can lower your AWS bill by buying a Reserved instance slot. Those slots are non-refundable and non-transferrable, so shop wisely. You can also stop the EC2 instance when you’re not using it and you won’t be billed for it but only for the allocated EBS volume storage.
You can use any other dedicated or virtual server which you own and can access by IP. An Amazon AWS EC2 instance is given here only as an example.
iSCSI server-side setup
Execute the following on your server (iSCSI target):
IP=23.21.98.10 # the public DNS IP address of the EC2 instance / server
# Log in to the server
ssh ubuntu@$IP
# Update your SSH key in ".ssh/authorized_keys", if needed.
sudo bash
cp /home/ubuntu/.ssh/authorized_keys /root/.ssh/ # so that we can log in directly as root
apt-get update
apt-get upgrade
apt-get install linux-headers-virtual # virtual because we're running an EC2 instance
apt-get install iscsitarget iscsitarget-dkms
perl -pi -e 's/^ISCSITARGET_ENABLE=.*$/ISCSITARGET_ENABLE=true/' /etc/default/iscsitarget
# We won't use any iSCSI authentication because the server is totally firewalled
# and we access it only using an SSH tunnel.
# NOTE: If you don't use Amazon EC2, make sure that you firewall this machine completely,
# leaving only SSH access (TCP port 22).
# update your block device location in "Path", if needed
cat >> /etc/iet/ietd.conf <<EOF
Target iqn.2012-03.net.famzah:storage.backup
Lun 0 Path=/dev/xvdf,Type=fileio
EOF
/etc/init.d/iscsitarget restart
echo 'PermitTunnel yes' >> /etc/ssh/sshd_config
/etc/init.d/ssh restart
iSCSI client-side setup
Execute the following on your client / desktop machine (iSCSI initiator):
# Install the iSCSI client
sudo apt-get install open-iscsi
How to attach an iSCSI volume on the client
The following commands show how to attach and detach a remote iSCSI volume on the client machine. The output of the commands is quoted with “#>>”.
IP=23.21.98.10 # the public DNS IP address of the EC2 instance / server
# Establish the secure SSH tunnel to the remote server
sudo -E \
ssh -F /dev/null \
-o PermitLocalCommand=yes \
-o LocalCommand="ifconfig tun0 172.18.0.2 pointopoint 172.18.0.1 netmask 255.255.255.0" \
-o ServerAliveInterval=60 \
-w 0:0 root@"$IP" \
'sudo ifconfig tun0 172.18.0.1 pointopoint 172.18.0.2 netmask 255.255.255.0; hostname; echo tun0 ready'
# Make sure that we can reach the remote server via the SSH tunnel
ping 172.18.0.1
# Execute this one-time; it discovers the available iSCSI volumes
sudo iscsiadm -m discovery -t st -p 172.18.0.1
#>> 172.18.0.1:3260,1 iqn.2012-03.net.famzah:storage.backup
# Attach the remote iSCSI volume on the local machine
sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --login
#>> Logging in to [iface: default, target: iqn.2012-03.net.famzah:storage.backup, portal: 172.18.0.1,3260]
#>> Login to [iface: default, target: iqn.2012-03.net.famzah:storage.backup, portal: 172.18.0.1,3260]: successful
# Check the kernel log
dmesg
#>> [ 1237.538172] scsi3 : iSCSI Initiator over TCP/IP
#>> [ 1238.657846] scsi 3:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4
#>> [ 1238.662985] sd 3:0:0:0: Attached scsi generic sg2 type 0
#>> [ 1239.578079] sd 3:0:0:0: [sdb] 167772160 512-byte logical blocks: (85.8 GB/80.0 GiB)
#>> [ 1239.751271] sd 3:0:0:0: [sdb] Write Protect is off
#>> [ 1239.751279] sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
#>> [ 1240.099649] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
#>> [ 1241.962729] sdb: unknown partition table
#>> [ 1243.568470] sd 3:0:0:0: [sdb] Attached SCSI disk
# Double-check that the iSCSI volume is with the expected size (80 GB in our case)
cat /proc/partitions
#>> major minor #blocks name
#>> ...
#>> 8 16 83886080 sdb
# The remote iSCSI volume is now available under /dev/sdb on our local machine.
# You can use it as any other locally attached hard disk (block device).
# Detach the iSCSI volume from the local machine
sync
sudo iscsiadm -m node --targetname "iqn.2012-03.net.famzah:storage.backup" --portal "172.18.0.1:3260" --logout
#>> Logging out of session [sid: 1, target: iqn.2012-03.net.famzah:storage.backup, portal: 172.18.0.1,3260]
#>> Logout of [sid: 1, target: iqn.2012-03.net.famzah:storage.backup, portal: 172.18.0.1,3260]: successful
# Check the kernel log
dmesg
#>> [ 1438.942277] connection1:0: detected conn error (1020)
# Double-check that the iSCSI volume is no longer available on the local machine
cat /proc/partitions
#>> no "sdb"
Once you have the iSCSI block device volume attached on your local computer, you can use it as you need, just like it was a normal hard disk. Only it will be slower because each I/O operation takes place over Internet. For example, you can locally encrypt the iSCSI volume with TrueCrypt, in order to prevent the administrators of the remote machine to be able to see your files.
There are situations when a friend is in need of Linux help, and the only way for you to help them is to log in to their machine and fix the problem yourself, instead of trying to explain over the phone all the steps to your friend.
Such a problem has two sub-problems:
The remote machine must accept incoming connections and provide you with shell access. The obvious way to achieve this is an SSH daemon. Many Desktop Linux distributions don’t install an SSH server by default though, for security reasons. Setting up an SSH server in this moment is slow, and could even not be possible, if your friend messed up with the packaging system, for example. So we need to find an easy way to bind a network shell on the remote machine.
We must be able to connect to the remote machine. Usually desktop machines are protected behind a firewall or NAT, and we cannot connect to them directly. If this is not the case for you, you can skip this step and just connect to the remote machine IP address. A common approach to overcome this problem is that the remote machine connects to a machine of yours, which has an accessible real IP address and has a running SSH server. Most Desktop Linux distributions have an SSH client installed by default. So all you need to do is quickly and temporarily set up an account with password authentication for your friend on your machine. Then let them log in there which will create a reverse tunnel back to their machine.
Bind a shell
Another useful tool which is usually available on Linux is the Netcat, the Swiss-army knife for TCP/IP. In order to bind a shell using the Netcat version available on Ubuntu/Debian, you need to execute the following:
I got this awesome idea from a user comment. I only extended it a bit by adding “2>&1” which redirects the STDERR error messages to the remote network client too.
Once the above has been executed on the remote machine, anyone can connect on TCP port 6000, assuming that there is no firewall. Note that you have to connect via Netcat again. A connection via Telnet adds an additional “\r” at every line end, which confuses Bash. If you need to perform actions as “root” on the remote machine, the shell needs to be executed as “root”:
If you are worried that your friend will mistype something, save the commands to a text file on a web server, and let them download it using “wget” or “curl”. Example:
The ssh client has the ability to forward a local port (review Reversing an ssh connection for a detailed example). Once you’ve set up an account for your friend, you ask them to connect to your machine:
ssh -R 6000:127.0.0.1:6000 $IP_OF_YOUR_MACHINE
Once your friend has connected to your machine, you can connect to theirs using the reverse SSH tunnel by executing the following:
nc 127.0.0.1 6000
The connection to 127.0.0.1 on TCP port 6000 is actually forwarded by SSH to the remote machine of your friend on their TCP port 6000.
Note that once you disconnect from the “nc” session, the Netcat server on the remote machine exists and needs to be restarted if you need to connect again.
Monitoring and controlling relative humidity is important for humans health. Too low or too high humidity feels uncomfortable, but most importantly high moisture is a factor for growing mold in your home, which could be health threatening (according to EPA and CDC). I will not go into details on how to control humidity. Instead I’ll describe what motivated me to design and create my own temperature and humidity sensor which reports its readings every minute to a central Linux server.
The main requirements for my design were the following:
Affordable price, as I wanted to install four sensors.
Great accuracy both for temperature and humidity readings.
Over-the-air communication, as I wanted to be able to install a sensor even in my bathroom, where I can’t run data or power wires. Support for wired communication too, so that we can reduce the overall price by not installing the wireless module.
Data logging to a computer, because both temperature and humidity change with time, for example when you sleep in the room, and you can’t look at a mechanical temperature or humidity meter every minute, in order to write down the results.
Battery operated, in order to avoid any wiring.
Open-source hardware and software toolchain, so I chose Atmel AVR microcontrollers. I got sick of Microchip and their commercial C compilers.
To have fun with electronics but at the same time create the device as fast as possible, as free time turned out to be a pretty limited resource recently.
I managed to accomplish most of the requirements I set with two exceptions: the device operates only a month on batteries, and cumulatively I spent almost a week to design, solder, develop the firmware, and test the device. Now all the sensors operate from a wall-plug power adapter, and my hunger for environmental control in my house is satisfied.
I’ll now try to describe the whole process and the reasons behind my engineer decisions. Note that I’m an amateur hobbyist.
Idea and requirements
I wrote down all my thoughts in a text editor. Then re-designed all the sticky notes into requirements, and did so a few more times, in order to finally decide what I want to design and not get distracted by new random ideas in my head.
Power supply
I wanted the device to be able to operate both via USB, and thus be powered by 5V, as well as to be powered by an accumulator or a battery with an input voltage up to 12V, so that it could be used in a car too. I put a polarity protection diode D1 in series with the power line, so that an accidental polarity mismatch doesn’t burn out the power regulator. Such a protection diode must have very low voltage drop and thus low power loss, and the Schottky diode 1N5819 seemed like a good match.
Operating from a battery also means that the voltage regulator must be extremely efficient and with a low bias current consumption, which means that it should draw almost nothing while there is nothing connected to it at its output as a load. Most battery operated devices “sleep” during most of their life cycle, so their consumption is close to zero. I used the ultra low-dropout fixed voltage regulator LP2986-33, marked as U1 in the schematics. The whole circuit operates at 3.3V because of the XBee wireless modules, and also because operating at a lower voltage usually gives lower power consumption.
Since we can have two different power sources, there must be a way to choose which one is active. You can switch between the power sources using the PWR_SELECT jumpers.
Wired communication via USB
I wanted to have the option to use the sensors by directly connecting them to a computer. This way we could save the money for an XBee wireless module. I used the classical USB-to-Serial solution FT232R, which is also quite inexpensive and requires almost no external components. You can see it in the schematics as U2. Note that the I/O lines of FT232R must be configured to operate at 3.3V too. This is done by connecting pin 17, which is the internal 3.3V regulator of FT232R, to pin 4. The internal 3.3V regulator is not used for anything else, and in theory I could have powered the I/O lines, pin 4, directly from the main voltage regulator U1.
Wireless communication
The XBee modules is something I wanted to play with for a long time. They seem very easy to work with and are packed with all kind of features. Though in my case I’m not using almost any of them, not even the AES encryption which could secure the data channel. I’m using the Series 1 XBee low-power embedded RF modules (XB24), which have a power of 1 mW and 30 m indoor range. There are many comments in Internet that the indoor range of the XBee modules is poor and I can confirm that. The range really depends on what the signal must travel through. Sometimes you lose the link even through one wall, sometimes it can go through a few walls. The XBee & XBee-PRO OEM RF Module Antenna Considerations is a great article by the XBee manufacturers. After all, probably by using such a low-power module, we shouldn’t expect so great results. It works well in my apartment though — all rooms report to the central XBee module successfully. On the server’s side, the receiver, I first had an XBee with chip antenna, which I replaced with an XBee-PRO with whip antenna. This made no difference.
Wiring the XBee module is very easy. It requires no external components. If you read the PDF datasheet, you’ll see how many great features an XBee has. I’m using only three of them:
Sleep mode — the microcontroller puts the XBee to sleep by controlling the SLEEP_RQ pin 9.
Networking addressing — each XBee is configured with a unique address, so that the receiver on the server side knows which reading belongs to which sensor probe.
API operation — the receiver XBee module operates in an API mode, which is a frame-based protocol that provides greater flexibility and more control. For example, besides the received data payload, an API frame gives information about the sender’s address and the signal quality.
Temperature and humidity sensor
I wanted to interface the sensor directly using a digital protocol, so that we can minimize the ADC stuff and errors. The SHT11 turned out to be the sensor I was looking for:
Relative humidity accuracy: +/- 3% in the range 20% to 80% RH. The sensor comes fully calibrated.
Temperature accuracy: +/- 1.5 degrees Celsius in the range -15 to +65 degrees Celsius.
Digital two-wire interface.
Very low energy consumption: 80uW (at 12bit, 3V, 1 measurement/s).
The SHT11 is a bit pricey but works very easily and accurately out of the box, so I decided to go with it. There is a very good alternative at Sparkfun — the RHT03 humidity and temperature sensor (also known as “RHT-22”). There were some contradictive comments by Sparkfun users — some say it works very well, some doubt its accuracy. I haven’t tried it but have left space JP7 on the current board, so that at some later time I could solder one RHT03 and use it with the existing schematics.
One note about the SHT11 two-wire interface. Definitely use a pull-up resistor on the DATA wire, as advised in the PDF! I tried to do some magic by the microcontoller and failed. With the exception of the pull-up resistor, everything else worked with no other problems with the SHT11 sensor. The manufacturer Sensirion provides Sample code for the SHTxx sensors which turned out to be very useful. I was able to re-code it for the AVR GNU C compiler in a couple of minutes.
The CRC calculation got me a bit confused. There are multiple different ways to calculate a CRC checksum, and they all provide different results. Each CRC calculation depends on the selected CRC polynomial, which is something like a bit-mask that defines the algorithm for the CRC calculation. After lots of struggle, I finally found an excellent Online CRC Calculation web wizard, which also includes a hardware implementation example, and sample C and VERILOG implementations, which you can copy-paste in your program. Thank you Kay Gorontzi!
Microcontroller
Initially I worked with ATmega8. Then I switched to ATmega168 because of the much lower power consumption. I could have used any other Atmel AVR microcontroller which has USART, internal oscillator, and sleep mode. Though ATmega8 or ATmega168 are always available in my local electronics shop, so I chose one of them. Besides the lower power consumption, ATmega168 has one other major advantage for my application — the watchdog timer can wake the chip from sleep mode and directly execute an interrupt, thus not re-starting the program from the very beginning.
Firmware
I’m working on Windows 7 64-bit and used a USBasp programmer to download the code into the microcontroller. The whole development toolchain is packaged into the WinAVR suite. It includes the AVR GCC compiler and the avrdude programmer. I also downloaded a sample Makefile which makes compilation and firmware download easy.
The main loop of the program does two tasks — measures and displays the readings over the serial port (which goes to the USB or over-the-air via XBee), and sleeps for about 60 seconds. As already mentioned, I use the new feature of ATmega168 which allows for the Watchdog timer to generate an interrupt, which wakes the chip from sleep mode. This is very handy as it allows you to continue the program at the point where you put it to sleep. The sleep mode was something new for me; there are some URLs in the source code which show what online articles helped me to master it. Note that the XBee RF transmitter is also put into sleep mode, in order to save battery.
Data collector
All the sensor readings are collected to a Linux server over-the-air. I use an XBee Explorer USB by Sparkfun to connect the XBee receiver with the Linux server. The XBee is seen as a serial device on the Linux box. The frame protocol of the XBee API is easy to understand and I implemented a Perl script to parse it. Here is a sample reading which is received from one of my wireless sensors (0x0001 is the address of the probe standing outside of my apartment):
[Mon Dec 26 17:32:11 2011] RX_packet: source=0x0001, rssi=-55dBm (opt=0x00): 4.24;69.04
As you can see, now it’s winter here — 4.24 degrees Celsius temperature; 69.04% relative humidity (RH).
Board design
Both the schematics and PCB board were designed using Eagle PCB by CadSoft. This is a great piece of software. Most PCB factories accept Eagle board files directly. You’ll find my Eagle files in the Resources section at the end of this article.
Lessons learned
There are a few things which I discovered only once I already built and tested the schematics:
Battery-operated devices are hard to design — in theory my sensors were supposed to last for about 3 months with a 9V battery. In practice only one of them lasted for a month, the others – for a week.
Electronics components, boards and/or assembly could differ a lot — see above. Also one of the SHT11 sensors is sometimes giving CRC errors.
XBee indoor range is not excellent.
Research and development takes a lot of time, usually 2x or 3x the time you planned. Furthermore, building something with love takes even more time, but in the end it pays off with great results and satisfaction.
You can create an electronics device with a lower price than what is currently offered on the marked. But this has its price too — your time, and you get no guarantee whatsoever.
I had different plans for this blog article but it got so lengthy that I wrote it in four different days (and it’s Christmas now). The main idea was to sketch the device and all its components, and to show that they can work together as a finished product. If there is any interest by other people, I’m happy to answer to any questions.
/contrib/famzah 