Enthusiasm never stops

Leave a comment

Secure NAS on Bifferboard running Debian

This NAS solution uses OpenSSH for secure transport over a TCP connection, and NFS to mount the volume on your local computer. The hardware of the NAS server is the low-cost Bifferboard.

I’m using an external hard disk via USB which is partitioned in two parts – /dev/sda1 (1GB) and the rest in /dev/sda2. Once you have installed Debian on Bifferboard, here are the commands which further transform your Bifferboard into a secure NAS:

apt-get update
apt-get -y install nfs-kernel-server

vi /etc/default/nfs-common 
  # update: STATDOPTS='--port 2231'
vi /etc/default/nfs-kernel-server 
  # update: RPCMOUNTDOPTS='-p 2233'

mkdir -m 700 /root/.ssh
  # add your public key for "root" in /root/.ssh/authorized_keys

echo '/mnt/storage,no_root_squash,no_subtree_check,insecure,async)' >> /etc/exports
mkdir /mnt/storage
chattr +i /mnt/storage # so that we don't accidentally write there without a mounted volume

cat > /etc/rc.local <<EOF

# allow only SSH access via the network
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT
/sbin/iptables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT # TCP initiated by server
/sbin/iptables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT # DNS traffic

# mount the storage volume here, so that any errors with it don't interfere with the system startup
/bin/mount /dev/sda2 /mnt/storage
/etc/init.d/nfs-kernel-server restart

# allow only public key authentication
fgrep -i -v PasswordAuthentication /etc/ssh/sshd_config > /tmp/sshd_config && \
  mv -f /tmp/sshd_config /etc/ssh/sshd_config && \
  echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config


There are two things you should consider with this setup:

  1. You must trust the “root” user who mounts the directory! They have full shell access to your NAS.
  2. A not-so-strong SSH encryption cipher is used, in order to improve the performance of the SSH transfer.

On the machine which is being backed up, I use the following script which mounts the NAS volume, starts the rsnapshot backup process and finally unmounts the NAS volume:

set -u


echo "Mounting NFS volume on $HOST:$REMOTEPORT (SSH-key='$SSHKEY')."
for port in 2049 2233 ; do
	N=$(($N + 1))
	LPORT=$((61000 + $N))
	ssh -f -i "$SSHKEY" -c arcfour128 -L"$LPORT":"$port" -p "$REMOTEPORT" "$SSHUSER@$HOST" sleep 600d
	echo "Forwarding: $HOST: Local port: $LPORT -> Remote port: $port"
sudo mount -t nfs -o noatime,nfsvers=2,proto=tcp,intr,rw,bg,port=61001,mountport=61002 "$REMOTEDIR" "$LOCALDIR"

echo "Doing backup."
time sudo /usr/bin/rsnapshot weekly

echo "Unmounting NFS volume and closing SSH tunnels."
sudo umount "$LOCALDIR"
for pid in $(ps axuww|grep ssh|grep 6100|grep arcfour|grep -v grep|awk '{print $2}') ; do
	kill "$pid" # possibly dangerous...

Update, 29/Sep/2010 – performance tunes:

  • Added “async” in “/etc/exports”.
  • Removed the “rsize=8192,wsize=8192” mount options – they are auto-negotiated by default.
  • Added the “noatime” mount option.
  • Put the SSH username in a variable.



1 Comment

Bifferboard performance benchmarks

The benchmarks were done while Bifferboard was running Linux kernel and Debian Lenny.

Boot time
Total boot time: 1 minute 11 seconds (standard Debian Lenny base installation)

The boot process goes like this:

  • Initial boot. Mounted root device (5 seconds wasted on waiting for the USB mass storage to be initialized). Executing INIT. [+21 seconds elapsed]
  • Waiting for udev to be initialized (most of the time spent here), configuring some misc settings, no dhcp network, entering Runlevel 2. [+41 seconds elapsed]
  • Started “rsyslogd”, “sshd”, “crond”. Got prompt on the serial console. [+9 seconds elapsed]

Therefore, if a very limited and custom Linux installation is used, the total boot time could be reduced almost twice.

CPU speed
Calculated BogoMips: 56.32
According to a quite complete BogoMips list table, this is an equivalent of Pentium@133MHz or 486DX4@100MHz.

According to another CPU benchmarks comparison table for Linux, Bifferboard falls into the category of Pentium@100Mhz.

Memory speed
A “dd” write to “/dev/shm” performs with a speed of 6.3 MB/s.
The MBW memory bandwidth benchmark shows the following results:

bifferboard:/tmp# wget
bifferboard:/tmp# dpkg -i mbw_1.1.1-1_i386.deb
bifferboard:/tmp# mbw 4 -n 20|egrep ^AVG
AVG Method: MEMCPY Elapsed: 0.15602 MiB: 4.00000 Copy:
25.637 MiB/s
AVG Method: DUMB Elapsed: 0.06611 MiB: 4.00000 Copy:
60.502 MiB/s
AVG Method: MCBLOCK Elapsed: 0.06619 MiB: 4.00000 Copy:
60.431 MiB/s

For comparison, my Pentium Dual-Core @ 2.50GHz with DDR2 @ 800 MHz (1.2 ns) shows a “dd” copy speed to “/dev/shm” of 271 MB/s, while the MBW test shows a maximum average speed of 7670.954 MiB/s. Bifferboard is an embedded device after all… 🙂

Available memory for applications
My base Debian installation with “udevd”, “dhclient3”, “rsyslogd”, “sshd”, “getty” and one tty session running shows 24908 kbytes free memory. You surely cannot put CNN.com on this little machine, but compared to the PIC16F877A which has 368 bytes (yes, bytes) total RAM memory, Bifferboard is a monster.

Disk system
All tests are done on an Ext3 file-system and a very fast USB Flash 8GB A-Data Xupreme 200X.
A “dd” copy to a file completes with a write speed of 6.1 MB/s.
The Bonnie++ benchmark test shows the following results:

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
bifferboard.lo 300M   822  96  5524  61  4305  42   855  99 16576  67 143.2  12
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                 16  1274  94 14830 100  1965  85  1235  90 22519 100 2015  87


Therefore, the sequential write speed is about 5.5 MB/s, while the sequential read speed is about 16.5 MB/s.

It’s worth mentioning that while the write tests were running, there was a very high CPU System load (not CPU I/O waiting) which indicates that the write throughput of Bifferboard may be a bit better if the file-system is not a journaling one. However, the tests for “Memory speed” above show that writing to “/dev/shm” (a memory-based file-system) completes with a rate of 6.3 MB/s. Therefore, this is probably the limit with this configuration.

Both Netperf and Wget show a throughput of 6.5 MB/s.
The packets-per-second tests complete at a rate of 8000 packets/second.
Modern systems can handle several hundred thousand packets-per-second without an issue. However, the measured network performance of Bifferboard is more than enough for trivial network communication with the device. During the network benchmark tests, there was very high CPU System usage, but that was expected.

Encryption and SSH transfers
The maximum encryption rate for an eCryptfs mount with AES cipher and 16 bytes key length is 536 KB/s. The standard SSH Protocol 2 transfer rate using the OpenSSH server is about the same – 587.1 KB/s. If you try to transfer a file over SSH and store it on an eCryptfs mounted volume, the transfer rate is 272.2 KB/s, which is logical, as the processing power is split between the SSH transfer and the eCryptfs encryption.
You can try to tweak your OpenSSH ciphers, in order to get much better performance. The OpenSSH ciphers performance benchmark page will give you a starting point.

Bifferboard performs pretty well for its price. It’s my personal choice over the 8-bit 16F877A and the other 16-bit Microchip / ARM microcontrollers, when a project does not require very fast I/O.


Running Debian on Bifferboard

There are three major steps in installing Debian on your Bifferboard:

  1. Kernel boot command line.
  2. Kernel installation on the Bifferboard.
  3. Rootfs installation on a USB device or an SD/MMC card.

Kernel boot command line

Since Biffboot v3.3, dated 19.July.2010, the kernel boot command line no longer specifies an external block device for the root file system. As a result of this, you need to update the boot configuration before you can boot from a USB device or an SD/MMC card. You have two options to configure the boot command line:

You need to set the kernel boot command line (“Kernel cmndline”) to:

console=uart,io,0x3f8 root=/dev/sda1 rootwait

Kernel installation on the Bifferboard

Download a pre-built kernel binary image:

The kernel is compiled with (almost) all possible modules, so your Bifferboard should be able to easily use any device supported on Debian. Once you have downloaded the kernel image, you can then upload it to the Bifferboard, as advised at the Biffboot Wiki page. You have two options to upload the kernel – via the serial port or over the ethernet. Both work well.

Example: Assuming that you have the Bifferboard SVN repository checked out in “~/biffer/svn“, you have downloaded the “vmlinuz-” kernel image in “/tmp“, your Bifferboard has a MAC address of “00:B3:F6:00:37:A9“, and you have connected it on the Ethernet port “eth0” of your computer, here are the commands that you would need to use:

cd ~/biffer/svn/utils
sudo ./bb_eth_upload.py eth0 00:B3:F6:00:37:A9 /tmp/vmlinuz-

Rootfs installation on a USB device or an SD/MMC card

Once you have the kernel “installed” on the Bifferboard and ready to boot, you need to prepare a rootfs media. This is where your Debian installation is stored and booted from. Download one of the following pre-built rootfs images (default root password is “biffroot”):

The “developer” version adds the following packages: build-essential, perl, links, manpages, manpages-dev, man-db, mc, vim. Note that for each image you will need at least 100MB more free on the rootfs media.

In order to populate the rootfs media, you have to do the following:

  1. Create one primary partition, format it as “ext3” and then mount the USB device or SD/MMC card.
  2. Extract the archive in the mounted directory.
  3. Unmount the directory.

Example: Assuming that you have the Bifferboard SVN repository checked out in “~/biffer/svn“, you have downloaded the “minimal” rootfs image in “/tmp“, and you are using an SD/MMC card under the device name “/dev/mmcblk0“, here are the commands that you would need to use:

sudo bash
mkdir /mnt/rootfs
cd ~/biffer/svn/debian/rootfs
./format-and-mount.sh /dev/mmcblk0 /mnt/rootfs
tar -jxf /tmp/debian-lenny-bifferboard-rootfs-minimal.tar.bz2 -C /mnt/rootfs
umount /mnt/rootfs

When you have the USB device or SD/MMC card ready and populated with the customized Debian rootfs, plug it in Bifferboard, attach a serial cable to Bifferboard, if you have one, and boot it up.

That’s it. Enjoy your Bifferboard running Debian.

Update: As already mentioned in the comments below, you would probably need to set up swap too. Here is my recipe:

# change "128" (MBytes) below to a number which suits your needs
dd if=/dev/zero of=/swapfile bs=1M count=128
mkswap /swapfile
swapon /swapfile # enables swap right away; disable with "swapoff -a"
echo '/swapfile none swap sw 0 0' >> /etc/fstab # enables swap at system boot

Using a file for swap on a 2.6 Linux kernel has the same performances as using a separate swap partition as discussed at LKML.

Update 2: As announced by Debian, Debian 5.0 (lenny) has been superseded by Debian 6.0 (squeeze). Security updates have been discontinued as of February 6th, 2012. Thus by downloading and installing the images provided here, you’re using an obsolete Debian release. If that’s not a problem for you, read on. You need to change the file “/etc/apt/sources.list” to the following using your favorite text editor:

deb http://archive.debian.org/debian lenny main contrib non-free
deb-src http://archive.debian.org/debian lenny main contrib non-free
deb http://archive.debian.org/debian-security/ lenny/updates main contrib non-free
deb-src http://archive.debian.org/debian-security/ lenny/updates main contrib non-free

P.S. If you want to build your own customized Debian rootfs image for Bifferboard – checkout the Bifferboard SVN repository and review the instructions in “debian/rootfs/images.txt“.


Leave a comment

Debian rootfs installation customized for Bifferboard

Update: There are (more up-to-date) automated scripts which you can use for the below actions:

  1. You need to checkout the whole Bifferboard SVN repository.
  2. The scripts are located in the directory “/debian/rootfs“. Execute them from the checked out repository on your local computer.

First you have to mount a medium on which we are going to install the Debian system. Generally, you have two options:

  • Using a USB Flash drive:

    $ export ROOTDEV=/dev/sdc1
    $ sudo mkfs.ext3 $ROOTDEV
    $ sudo tune2fs -c 0 -i 0 $ROOTDEV
    $ export MNTPOINT=/mnt/diskimage
    $ sudo mount $ROOTDEV $MNTPOINT

  • Using a Qemu image:

    $ export MNTPOINT=/mnt/diskimage
    $ export IMGFILE=hd0.img
    $ sudo mount -o loop,offset=32256 "$IMGFILE" $MNTPOINT

Once we have the medium mounted at $MNTPOINT, we can proceed with installing Debian there and configuring it for Bifferboard:

$ export DBS_OS_VERSION=lenny
## replace "bg." with your local archive, or just omit it
$ export DBS_LOCAL_ARCHIVE=bg.
$ sudo debootstrap --arch i386 ${DBS_OS_VERSION} $MNTPOINT/ http://ftp.${DBS_LOCAL_ARCHIVE}debian.org/debian
## ... go grab a pizza or something ... this will take a while
$ sudo cp /etc/resolv.conf $MNTPOINT/etc/
$ sudo mount proc $MNTPOINT/proc -t proc
$ sudo chroot $MNTPOINT
## We are now in the "chroot" environment as root
/# apt-get -qq update && apt-get install wget
/# cd /root && wget http://bifferboard.svn.sourceforge.net/viewvc/bifferboard/debian/rootfs/include/debootstrap-postconfig.sh
/root# chmod +x debootstrap-postconfig.sh && ./debootstrap-postconfig.sh
/root# passwd root
/root# exit
## Back to our machine
$ sudo umount $MNTPOINT/proc
$ sudo umount $MNTPOINT

Now you have a minimum Debian installation customized for Bifferboard in the following way:

  • Custom kernel for Bifferboard installed by a .deb package.
  • Ethernet interface configured as DHCP client.
  • Temporary directories /tmp and /var/tmp mounted on a RAM-disk.
  • All APT sources “main contrib non-free” enabled.
  • Serial console on ttyS0 (115200 8N1).
  • RTC (real-time clock) kernel modules blacklisted – the Bifferboard has no RTC.
  • IPv6 disabled – takes a lot of resources and we won’t use it anyway, for now.

I may add any further customizations if needed. You can always review the debootstrap-postconfig.sh script for details on what is being configured.

You can use this image/disk as a rootfs which you can boot directly on Bifferboard or try in Qemu. Note that you have to install our Debian kernel on Bifferboard prior to booting this rootfs.

Used resources:

Leave a comment

Build a Debian Linux kernel for Bifferboard as .deb packages

In my previous article I explained why and how to build a very small Linux kernel with all possible modules enabled which would help us to run a standard Debian installation on Bifferboard.

You can download the already built .deb packages for Debian “lenny” at the following addresses:

On my Bifferboard, I use the following Kernel command line to boot this kernel:

rootwait root=/dev/sda1 console=uart,io,0x3f8

For Qemu, because of some USB mass-storage emulation issues, the line looks like:

rootwait root=/dev/sda1 console=uart,io,0x3f8 irqpoll

Update: There are (more up-to-date) automated scripts which you can use for the below actions:

  • You need to checkout the whole Bifferboard SVN repository.
  • The scripts are located in the directory “/debian/kernel“. Execute the “build.sh” script from the checked out repository on your local computer, on a Debian “lenny” system.

If you want to build the packages yourself, you need to execute the following commands on a Debian “lenny” machine (a virtual machine or a chroot()’ed installation work too):

famzah@FURNA:~$ sudo apt-get install kernel-package fakeroot build-essential ncurses-dev tar patch
famzah@FURNA:~$ export KVERSION=
famzah@FURNA:~$ rm -rf /tmp/tmpkern-$KVERSION
famzah@FURNA:~$ mkdir /tmp/tmpkern-$KVERSION
famzah@FURNA:~$ cd /tmp/tmpkern-$KVERSION && wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-$KVERSION.tar.bz2
famzah@FURNA:/tmp/tmpkern-$ tar -xjf linux-$KVERSION.tar.bz2
famzah@FURNA:/tmp/tmpkern-$ sudo mkdir -p /usr/src/bifferboard && sudo chown $USER /usr/src/bifferboard
famzah@FURNA:/tmp/tmpkern-$ mv linux-$KVERSION /usr/src/bifferboard/
famzah@FURNA:/tmp/tmpkern-$ cd /usr/src/bifferboard/linux-$KVERSION
famzah@FURNA:/usr/src/bifferboard/linux-$ wget 'http://www.famzah.net/download/bifferboard/obsolete/bifferboard-' -O bifferboard-
famzah@FURNA:/usr/src/bifferboard/linux-$ patch --quiet -p1 < bifferboard-
famzah@FURNA:/usr/src/bifferboard/linux-$ wget http://www.famzah.net/download/bifferboard/obsolete/build-biff-kernel-
famzah@FURNA:/usr/src/bifferboard/linux-$ chmod +x build-biff-kernel-
famzah@FURNA:/usr/src/bifferboard/linux-$ ./build-biff-kernel-
# When "make menuconfig" is displayed, just EXIT and SAVE the configuration.
# After the build, you can find the two .deb packages in "/usr/src/bifferboard".

Used resources:

Leave a comment

Build a very small Linux kernel with all possible modules enabled

…and still be able to mount a root file-system stored on a USB mass-storage.

The idea is to build a very small kernel with the bare minimum compiled-in and all the rest as modules which are stored on the “rootfs” device. Once the “rootfs” device has been mounted by the kernel, the kernel can load any additional modules from there. Therefore, our kernel has the following compiled-in features:

  • device drivers for the “rootfs”: USB mass-storage.
  • File-systems: ext3.
  • Misc: BSD process accounting, /proc support, inotify support, NO initrd (we do not need one as we can mount the “rootfs” device directly), NO compiled-in wireless support (only by modules, thus you cannot download a “rootfs” over-the-air by PXE, for example), NO swap support (Bifferboard I/O is too slow for swapping).
  • Size: very small, only 918224 bytes.

Why would someone need such a kernel?
The size of the bootable kernel image (+the initrd ramdisk, if any) on a Bifferboard single-chip-computer is limited to:

  • 974848 bytes with Biffboot v2.0
  • 983040 bytes with BiffBoot v1.X

Furthermore, some patches and special configuration is required for the RDC chip which is the heart of the system. The creator of Bifferboard has done this for us already – he developed the patch and created a minimal config for the Linux kernel.

In order to merge the Bifferboard minimal kernel config with a config where all modules are enabled, I do the following:

  • Make a kernel config with all possible modules enabled by executing “make allmodconfig“. The problem with this config is that it has every possible option selected as “Yes”, not only the modules. Therefore, I substitute every “Yes” (which is not a module) to “No” by executing “perl -pi -e ‘s/=y/=n/g’ .config“. This way I have only config entries which say “CONFIG_SOME_OPTION=m”.
  • Download the other minimal kernel config which I want to merge with priority over the “all modules config”. I make a “grep =y .config-biff > .config-biff-yes“. This way I leave only the “Yes” selected kernel config options, nothing more.
  • Finally, I can merge the config files into one by concatenating them. The file which is concatenated last has the most priority. This is how Kconfig merges the config lines and resolves conflicts or redefinitions of the same kernel option.
  • There is however a problem with this automatic way of generating and merging an all-modules kernel config – there are sections in the kernel config which add no additional code to the kernel (thus add no space either) but they “hide” their child sub-sections. One has to go through the kernel menu manually and select with “Yes” every menu option which has a sub-menu associated with it. You can easily recognize such menu options by the “—>” ending after their menu title. I’ve created a third config which is also being merged as last which selects all such options as “Yes” (multiple CONFIG_SUBMENU_EXAMPLE=y).
  • If you want to overwrite anything at the very end, you can create a fourth config file and merge it as very last.

Here is a Bash script which does what I’ve currently described: http://www.famzah.net/download/bifferboard/obsolete/build-biff-kernel-

Note that when you have no initrd and boot from a USB mass-storage device, you have to add “rootdelay=30” (or less) to your kernel command line. It takes some time for the USB mass-storage devices to get initialized. If there is no “rootdelay” option specified, the kernel tries to mount the “rootfs” device immediately which ends up in Kernel panic – not syncing: VFS: Unable to mount root fs. This very useful article describing the initial RAM disk (initd) in detail helped me to find out why the original Ubuntu kernel+initrd gave no kernel panic and was able to mount the root file-system from my USB stick, but at the very same time my custom kernel couldn’t do it. I did some initrd debugging and found out that it simulates the kernel command line option “rootdelay” – it polls if the “rootfs” device has been detected, every 0.1 seconds.

UPDATE: The option “rootwait” is what I was actually looking for. It is similar to “rootdelay=NN”, only that it waits forever for a root device and continues with the boot immediately after the root device is found, thus the kernel wastes no time in just waiting for “NN” seconds to elapse.

You can read my next article which gives detailed instructions on how to build a kernel suitable for Bifferboard and package it as .deb files.

Leave a comment

Qemu .deb package for the RDC Bifferboard hardware

Following the instructions found at these articles, I build a .deb package for Qemu which is suitable for the RDC processor which is used by Bifferboard. The instructions and patches can be found at the official Qemu Wiki page of Bifferboard.

There is nothing special I’ve done here, just packaged the qemu binary, so that you can easily try the “qemu-rdc” binary. The download link follows:

Here are some simple instructions on how to test your own “bzImage” kernel build:

# Installation instructions for the .deb package and for the Qemu setup
famzah@FURNA:~$ wget http://www.famzah.net/download/bifferboard/qemu-rdc_0.10.5-1_i386.deb
famzah@FURNA:~$ sudo dpkg -i qemu-rdc_0.10.5-1_i386.deb
famzah@FURNA:~$ mkdir test-kernel
famzah@FURNA:~$ cd test-kernel/
famzah@FURNA:~/test-kernel$ svn co https://bifferboard.svn.sourceforge.net/svnroot/bifferboard/qemu/
famzah@FURNA:~/test-kernel$ cd qemu/run
famzah@FURNA:~/test-kernel/qemu/run$ vi run-qemu.sh # at the last line, change "qemu" with "qemu-rdc"

# You can now test your kernel/rootfs build. For example:
famzah@FURNA:~/test-kernel/qemu/run$ cp /home/famzah/biffer/qemu/custom_bzImage ./bzImage
famzah@FURNA:~/test-kernel/qemu/run$ QEMU_BIN=qemu-rdc ./run-qemu.sh

If you want to attach a USB mass-storage device and try your rootfs build there, please follow the instructions at the official Qemu Wiki page of Bifferboard on which parameters to add to “qemu-rdc” in “run-qemu.sh”.

You can exit the emulator by pressing CTRL+a and “x”. You will get some help info by pressing CTRL+a and “?”. See the man or documentation pages of “qemu” for more information.

In a few days I’ll post an article and a .deb package for a kernel build with (almost) all possible modules, suitable for running a native i386 Debian rootfs installation on Bifferboard.

P.S. Today I got my serial USB RS232 @ 3.3V cable and can now start with some real tests 😀