Enthusiasm never stops


fork() gets slower as parent process uses more memory

Background information
Forking is an important, fundamental part of Unix, critical to the support of its design philosophy. For example, if a process wants to execute a command, it has to fork() a child process which then immediately calls exec(). And since the philosophy of Unix involves executing many small commands/programs, in order to achieve something meaningful, it turns out that fork() is called pretty often.

There are two main fork() patterns when a parent process wants to execute a command:

  • The parent does not need to communicate with the child process in any way – the child process executes, and the parent gets its exit code back. No input/output with the child is done at all, only the inital command line arguments are passed.
  • The parent needs to communicate with the child process – either it needs to supply something at the standard input or some other file descriptor of the child, or it wants to get some information from the standard output or some other file descriptor of the child, or both.

For the case when there is no communication involved, Unix guys developed the vfork() call. It is a very light-weight version of fork(), very close to threading. The gotcha here is that a child process which was created by vfork() cannot modify any variables in its space at all or do something else, because it has no own stack. The child process is only allowed to call exec() right after it was born, nothing more. This speeds up the usual fork()-then-exec() model, because often the parent does not need to communicate with the child process – the parent just wants the command executed with the given command line arguments.

For all other cases when the parent communicates with the child internally using file descriptors (anonymous pipes, etc.), the standard fork() system call is used.

Problem definition
It turns out that when the parent process allocates some memory, the fork() call takes longer to execute if a bigger amount of this allocated memory is being used, that is – if the parent process writes something there. Linux and probably the other Unix systems employ a copy-on-write feature and don’t physically copy the allocated memory from the parent into the child process initially on each fork(). Not until the child modifies it. Nevertheless, the fork() call gets slower and slower as more and more memory is being used (not just allocated) in the parent process. It seems that even though the data of the allocated/used memory itself is not being copied, thanks to the copy-on-write feature, the internal virtual memory structures in the kernel, which hold the information about how much and what memory the parent process has allocated, are being copied in an inefficient way while the child process is being created by fork().

Currently available options
So why don’t we then just vfork() always? It is very fast. And the answer is – because we cannot communicate with the child process when it was created by vfork().

Okay, so why don’t we use threads then? They are similar to vfork(), only that the child process (thread) has its own stack and shares the data segment (allocated memory) of the parent process. We can even use these shared data variables for inter-process communication. And the answer is – because a thread cannot invoke exec() by definition. This is not supported by the threading libraries, as required by POSIX.1.

Talk to me in numbers, how slower does fork() get in regards to memory allocation and usage
Here are some benchmark results, the forking is done a few thousand times, in order to accumulate an accountable CPU time. The program which is being exec()’ed is a very tiny binary which contains only two system calls – write(“Hello world”) and then _exit(0). The benchmark results follow:

System info Allocated memory Usage ratio vfork() + exec() fork() + exec()
Linux 2.6.28-15-generic i686 20MB 1:2 (10MB) 1.49 12.08
Linux 2.6.28-15-generic i686 20MB 1:1 (20MB) 1.53 21.60
Linux 2.6.28-15-generic i686 40MB 1:2 (20MB) 1.59 21.23
FreeBSD 7.1-RELEASE-p4 i386 20MB 1:2 (10MB) 2.26 20.22
FreeBSD 7.1-RELEASE-p4 i386 40MB 1:2 (20MB) 2.44 33.94

As we can see from the test results, the vfork() call is not affected by the amount of memory usage. This does not apply for fork() though. On Linux we observe almost two times more CPU usage as the memory usage is increased twice. On FreeBSD the results are similar, only a bit better – if the memory usage is increased twice, the CPU usage of fork() is increased with 50% (vs. 100% on Linux). Even though, the difference in CPU time between the vfork() and fork() calls is significant on both operating systems – fork() is more than 1000% slower.

You can read my next article which describes a solution for Linux which allows a parent process to communicate with its child, similar to fork(), but is as fast as vfork(). The article also contains more detailed information about the benchmark tests we did, in order to populate the above table.


Running Debian on Bifferboard

There are three major steps in installing Debian on your Bifferboard:

  1. Kernel boot command line.
  2. Kernel installation on the Bifferboard.
  3. Rootfs installation on a USB device or an SD/MMC card.

Kernel boot command line

Since Biffboot v3.3, dated 19.July.2010, the kernel boot command line no longer specifies an external block device for the root file system. As a result of this, you need to update the boot configuration before you can boot from a USB device or an SD/MMC card. You have two options to configure the boot command line:

You need to set the kernel boot command line (“Kernel cmndline”) to:

console=uart,io,0x3f8 root=/dev/sda1 rootwait

Kernel installation on the Bifferboard

Download a pre-built kernel binary image:

The kernel is compiled with (almost) all possible modules, so your Bifferboard should be able to easily use any device supported on Debian. Once you have downloaded the kernel image, you can then upload it to the Bifferboard, as advised at the Biffboot Wiki page. You have two options to upload the kernel – via the serial port or over the ethernet. Both work well.

Example: Assuming that you have the Bifferboard SVN repository checked out in “~/biffer/svn“, you have downloaded the “vmlinuz-” kernel image in “/tmp“, your Bifferboard has a MAC address of “00:B3:F6:00:37:A9“, and you have connected it on the Ethernet port “eth0” of your computer, here are the commands that you would need to use:

cd ~/biffer/svn/utils
sudo ./bb_eth_upload.py eth0 00:B3:F6:00:37:A9 /tmp/vmlinuz-

Rootfs installation on a USB device or an SD/MMC card

Once you have the kernel “installed” on the Bifferboard and ready to boot, you need to prepare a rootfs media. This is where your Debian installation is stored and booted from. Download one of the following pre-built rootfs images (default root password is “biffroot”):

The “developer” version adds the following packages: build-essential, perl, links, manpages, manpages-dev, man-db, mc, vim. Note that for each image you will need at least 100MB more free on the rootfs media.

In order to populate the rootfs media, you have to do the following:

  1. Create one primary partition, format it as “ext3” and then mount the USB device or SD/MMC card.
  2. Extract the archive in the mounted directory.
  3. Unmount the directory.

Example: Assuming that you have the Bifferboard SVN repository checked out in “~/biffer/svn“, you have downloaded the “minimal” rootfs image in “/tmp“, and you are using an SD/MMC card under the device name “/dev/mmcblk0“, here are the commands that you would need to use:

sudo bash
mkdir /mnt/rootfs
cd ~/biffer/svn/debian/rootfs
./format-and-mount.sh /dev/mmcblk0 /mnt/rootfs
tar -jxf /tmp/debian-lenny-bifferboard-rootfs-minimal.tar.bz2 -C /mnt/rootfs
umount /mnt/rootfs

When you have the USB device or SD/MMC card ready and populated with the customized Debian rootfs, plug it in Bifferboard, attach a serial cable to Bifferboard, if you have one, and boot it up.

That’s it. Enjoy your Bifferboard running Debian.

Update: As already mentioned in the comments below, you would probably need to set up swap too. Here is my recipe:

# change "128" (MBytes) below to a number which suits your needs
dd if=/dev/zero of=/swapfile bs=1M count=128
mkswap /swapfile
swapon /swapfile # enables swap right away; disable with "swapoff -a"
echo '/swapfile none swap sw 0 0' >> /etc/fstab # enables swap at system boot

Using a file for swap on a 2.6 Linux kernel has the same performances as using a separate swap partition as discussed at LKML.

Update 2: As announced by Debian, Debian 5.0 (lenny) has been superseded by Debian 6.0 (squeeze). Security updates have been discontinued as of February 6th, 2012. Thus by downloading and installing the images provided here, you’re using an obsolete Debian release. If that’s not a problem for you, read on. You need to change the file “/etc/apt/sources.list” to the following using your favorite text editor:

deb http://archive.debian.org/debian lenny main contrib non-free
deb-src http://archive.debian.org/debian lenny main contrib non-free
deb http://archive.debian.org/debian-security/ lenny/updates main contrib non-free
deb-src http://archive.debian.org/debian-security/ lenny/updates main contrib non-free

P.S. If you want to build your own customized Debian rootfs image for Bifferboard – checkout the Bifferboard SVN repository and review the instructions in “debian/rootfs/images.txt“.


Leave a comment

Debian rootfs installation customized for Bifferboard

Update: There are (more up-to-date) automated scripts which you can use for the below actions:

  1. You need to checkout the whole Bifferboard SVN repository.
  2. The scripts are located in the directory “/debian/rootfs“. Execute them from the checked out repository on your local computer.

First you have to mount a medium on which we are going to install the Debian system. Generally, you have two options:

  • Using a USB Flash drive:

    $ export ROOTDEV=/dev/sdc1
    $ sudo mkfs.ext3 $ROOTDEV
    $ sudo tune2fs -c 0 -i 0 $ROOTDEV
    $ export MNTPOINT=/mnt/diskimage
    $ sudo mount $ROOTDEV $MNTPOINT

  • Using a Qemu image:

    $ export MNTPOINT=/mnt/diskimage
    $ export IMGFILE=hd0.img
    $ sudo mount -o loop,offset=32256 "$IMGFILE" $MNTPOINT

Once we have the medium mounted at $MNTPOINT, we can proceed with installing Debian there and configuring it for Bifferboard:

$ export DBS_OS_VERSION=lenny
## replace "bg." with your local archive, or just omit it
$ export DBS_LOCAL_ARCHIVE=bg.
$ sudo debootstrap --arch i386 ${DBS_OS_VERSION} $MNTPOINT/ http://ftp.${DBS_LOCAL_ARCHIVE}debian.org/debian
## ... go grab a pizza or something ... this will take a while
$ sudo cp /etc/resolv.conf $MNTPOINT/etc/
$ sudo mount proc $MNTPOINT/proc -t proc
$ sudo chroot $MNTPOINT
## We are now in the "chroot" environment as root
/# apt-get -qq update && apt-get install wget
/# cd /root && wget http://bifferboard.svn.sourceforge.net/viewvc/bifferboard/debian/rootfs/include/debootstrap-postconfig.sh
/root# chmod +x debootstrap-postconfig.sh && ./debootstrap-postconfig.sh
/root# passwd root
/root# exit
## Back to our machine
$ sudo umount $MNTPOINT/proc
$ sudo umount $MNTPOINT

Now you have a minimum Debian installation customized for Bifferboard in the following way:

  • Custom kernel for Bifferboard installed by a .deb package.
  • Ethernet interface configured as DHCP client.
  • Temporary directories /tmp and /var/tmp mounted on a RAM-disk.
  • All APT sources “main contrib non-free” enabled.
  • Serial console on ttyS0 (115200 8N1).
  • RTC (real-time clock) kernel modules blacklisted – the Bifferboard has no RTC.
  • IPv6 disabled – takes a lot of resources and we won’t use it anyway, for now.

I may add any further customizations if needed. You can always review the debootstrap-postconfig.sh script for details on what is being configured.

You can use this image/disk as a rootfs which you can boot directly on Bifferboard or try in Qemu. Note that you have to install our Debian kernel on Bifferboard prior to booting this rootfs.

Used resources:

Leave a comment

Create a Qemu image file which you can mount in both Linux and Qemu

The idea is to be able to easily manage a Qemu image outside of Qemu, natively on Linux. This can help you to alter the files on the Qemu image easily on Linux and then test the modified Qemu image on a Qemu virtual machine.

You can download an empty, formatted with Ext3 Qemu raw image at the following URL address:

There is nothing special about how you can achieve this yourself:

  • Create an empty Qemu image file.
  • Run an installation CD of Debian (or any other Linux) under Qemu and use the empty image as an available hard disk.
  • Partition and format the Qemu hard disk (resp. the Qemu image file) using the Linux installer.
  • Interrupt the Linux installer, stop Qemu, mount the Qemu image on Linux and clean it up.

You can achieve the above using the following commands:

# make sure to update the .iso URL if needed
$ wget http://cdimage.debian.org/debian-cd/5.0.3/i386/iso-cd/debian-503-i386-netinst.iso
$ qemu-img create hd0.img 2G
$ qemu -hda hd0.img -cdrom debian-503-i386-netinst.iso -boot d
# continue with the installation to the point where you can set up the partitions
# set up a primary partition using the entire disk space, do not set up a swap partition; save changes to disk and continue
# interrupt the installation (for example from the second console by executing "halt"), stop the virtual machine, we will not need it any further
$ sudo mkdir -p /mnt/diskimage
$ sudo mount -o loop,offset=32256 hd0.img /mnt/diskimage
$ sudo rm -r /mnt/diskimage/*
$ sudo mkdir -m 0700 '/mnt/diskimage/lost+found'
$ sudo umount /mnt/diskimage

Now we have an empty Qemu image which we can mount in both Linux and Qemu.

Here is an example on how to mount this image in Qemu:

qemu -usb -usbdevice disk:hd0.img

Do not use the image simultaneously as Linux mount and Qemu hard disk.

Used resources:

Leave a comment

Build a Debian Linux kernel for Bifferboard as .deb packages

In my previous article I explained why and how to build a very small Linux kernel with all possible modules enabled which would help us to run a standard Debian installation on Bifferboard.

You can download the already built .deb packages for Debian “lenny” at the following addresses:

On my Bifferboard, I use the following Kernel command line to boot this kernel:

rootwait root=/dev/sda1 console=uart,io,0x3f8

For Qemu, because of some USB mass-storage emulation issues, the line looks like:

rootwait root=/dev/sda1 console=uart,io,0x3f8 irqpoll

Update: There are (more up-to-date) automated scripts which you can use for the below actions:

  • You need to checkout the whole Bifferboard SVN repository.
  • The scripts are located in the directory “/debian/kernel“. Execute the “build.sh” script from the checked out repository on your local computer, on a Debian “lenny” system.

If you want to build the packages yourself, you need to execute the following commands on a Debian “lenny” machine (a virtual machine or a chroot()’ed installation work too):

famzah@FURNA:~$ sudo apt-get install kernel-package fakeroot build-essential ncurses-dev tar patch
famzah@FURNA:~$ export KVERSION=
famzah@FURNA:~$ rm -rf /tmp/tmpkern-$KVERSION
famzah@FURNA:~$ mkdir /tmp/tmpkern-$KVERSION
famzah@FURNA:~$ cd /tmp/tmpkern-$KVERSION && wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-$KVERSION.tar.bz2
famzah@FURNA:/tmp/tmpkern-$ tar -xjf linux-$KVERSION.tar.bz2
famzah@FURNA:/tmp/tmpkern-$ sudo mkdir -p /usr/src/bifferboard && sudo chown $USER /usr/src/bifferboard
famzah@FURNA:/tmp/tmpkern-$ mv linux-$KVERSION /usr/src/bifferboard/
famzah@FURNA:/tmp/tmpkern-$ cd /usr/src/bifferboard/linux-$KVERSION
famzah@FURNA:/usr/src/bifferboard/linux-$ wget 'http://www.famzah.net/download/bifferboard/obsolete/bifferboard-' -O bifferboard-
famzah@FURNA:/usr/src/bifferboard/linux-$ patch --quiet -p1 < bifferboard-
famzah@FURNA:/usr/src/bifferboard/linux-$ wget http://www.famzah.net/download/bifferboard/obsolete/build-biff-kernel-
famzah@FURNA:/usr/src/bifferboard/linux-$ chmod +x build-biff-kernel-
famzah@FURNA:/usr/src/bifferboard/linux-$ ./build-biff-kernel-
# When "make menuconfig" is displayed, just EXIT and SAVE the configuration.
# After the build, you can find the two .deb packages in "/usr/src/bifferboard".

Used resources:

Leave a comment

Build a very small Linux kernel with all possible modules enabled

…and still be able to mount a root file-system stored on a USB mass-storage.

The idea is to build a very small kernel with the bare minimum compiled-in and all the rest as modules which are stored on the “rootfs” device. Once the “rootfs” device has been mounted by the kernel, the kernel can load any additional modules from there. Therefore, our kernel has the following compiled-in features:

  • device drivers for the “rootfs”: USB mass-storage.
  • File-systems: ext3.
  • Misc: BSD process accounting, /proc support, inotify support, NO initrd (we do not need one as we can mount the “rootfs” device directly), NO compiled-in wireless support (only by modules, thus you cannot download a “rootfs” over-the-air by PXE, for example), NO swap support (Bifferboard I/O is too slow for swapping).
  • Size: very small, only 918224 bytes.

Why would someone need such a kernel?
The size of the bootable kernel image (+the initrd ramdisk, if any) on a Bifferboard single-chip-computer is limited to:

  • 974848 bytes with Biffboot v2.0
  • 983040 bytes with BiffBoot v1.X

Furthermore, some patches and special configuration is required for the RDC chip which is the heart of the system. The creator of Bifferboard has done this for us already – he developed the patch and created a minimal config for the Linux kernel.

In order to merge the Bifferboard minimal kernel config with a config where all modules are enabled, I do the following:

  • Make a kernel config with all possible modules enabled by executing “make allmodconfig“. The problem with this config is that it has every possible option selected as “Yes”, not only the modules. Therefore, I substitute every “Yes” (which is not a module) to “No” by executing “perl -pi -e ‘s/=y/=n/g’ .config“. This way I have only config entries which say “CONFIG_SOME_OPTION=m”.
  • Download the other minimal kernel config which I want to merge with priority over the “all modules config”. I make a “grep =y .config-biff > .config-biff-yes“. This way I leave only the “Yes” selected kernel config options, nothing more.
  • Finally, I can merge the config files into one by concatenating them. The file which is concatenated last has the most priority. This is how Kconfig merges the config lines and resolves conflicts or redefinitions of the same kernel option.
  • There is however a problem with this automatic way of generating and merging an all-modules kernel config – there are sections in the kernel config which add no additional code to the kernel (thus add no space either) but they “hide” their child sub-sections. One has to go through the kernel menu manually and select with “Yes” every menu option which has a sub-menu associated with it. You can easily recognize such menu options by the “—>” ending after their menu title. I’ve created a third config which is also being merged as last which selects all such options as “Yes” (multiple CONFIG_SUBMENU_EXAMPLE=y).
  • If you want to overwrite anything at the very end, you can create a fourth config file and merge it as very last.

Here is a Bash script which does what I’ve currently described: http://www.famzah.net/download/bifferboard/obsolete/build-biff-kernel-

Note that when you have no initrd and boot from a USB mass-storage device, you have to add “rootdelay=30” (or less) to your kernel command line. It takes some time for the USB mass-storage devices to get initialized. If there is no “rootdelay” option specified, the kernel tries to mount the “rootfs” device immediately which ends up in Kernel panic – not syncing: VFS: Unable to mount root fs. This very useful article describing the initial RAM disk (initd) in detail helped me to find out why the original Ubuntu kernel+initrd gave no kernel panic and was able to mount the root file-system from my USB stick, but at the very same time my custom kernel couldn’t do it. I did some initrd debugging and found out that it simulates the kernel command line option “rootdelay” – it polls if the “rootfs” device has been detected, every 0.1 seconds.

UPDATE: The option “rootwait” is what I was actually looking for. It is similar to “rootdelay=NN”, only that it waits forever for a root device and continues with the boot immediately after the root device is found, thus the kernel wastes no time in just waiting for “NN” seconds to elapse.

You can read my next article which gives detailed instructions on how to build a kernel suitable for Bifferboard and package it as .deb files.

1 Comment

Record desktop activity by making regular screenshots on Ubuntu

If you want to capture your desktop regularly for accounting or other purposes, here is how I implemented this on my Kubuntu desktop machine.

I found the following packages in my Kubuntu repository:

  • scrot – easy batch mode, only console interface
  • deskscribe – just records in some text log files, no image screenshots
  • ksnapshot – dcop problems while trying to make it work in batch mode

The winner is scrot. The simple Bash scripts I developed do the following:

  • Makes a snapshot, suitable for running automatically by crontab (make-snapshot.sh)
  • Tests if there are recent snapshots in a specified folder; an error is issued otherwise (test-snapshot.sh)

Here is what I’ve put in my user’s crontab (“crontab -e”):

* * * * * ~/make-snapshot.sh :0 /no-backup/famzah/snapshots
* * * * * ~/test-snapshot.sh 5 /no-backup/famzah/snapshots

This way a snapshot is made every minute. Every five minutes a check is made if the snapshot utility works properly. In the case of an error, the output from the “test-snapshot.sh” script is sent via email by crontab. This is a standard feature of crontab.

Update: The snapshots are now automatically split into sub-directories according to the current date “%Y-%m-%d/”.

The scripts have a help message, in case any of the parameters are not very clear. You need to install the “scrot” package by the following command:

sudo apt-get install scrot

Tested with Kubuntu Karmic and Lucid.

An exercise left for the reader 🙂 – a crontab script to clean up very old directories with screenshots. Hint: A simple “find … -type d -mtime … | xargs rm …” should do the trick.