1. Datastore performance – lots of people have already written about it (see links #1, #2, and #3). Currently, when working with small datasets, it’s far from being comparable even with a slow MySQL database, and you may also occasionally get internal errors, as well as increased latencies. I contacted Google about this, and asked them if the Business customers of GAE who pay for it would get better Datastore performance. Here is what I got as an answer from Nick Johnson, a GAE developer:
   

   Business customers will receive paid support, which is prioritized, as well as the extra features we announced at I/O. System latency is not any different, however, as we try and make the system as fast as possible for all our users.

   So the bad news is that you cannot make the Datastore run faster, even if you pay.
   The good news is that we are all getting the same service in terms of speed, which is a good thing – when everybody is having difficulties, then the community will eventually find a solution.

2. Memcache fairness – what happens if another website (on the same server) uses the Memcache service extensively, thus making the Memcache entries of my website expire too quickly, due to the memory pressure. Here is what Nick Johnson from Google replied:
   

   Memcache is segmented by application. Although there is some variation (so that apps that don’t use any memcache don’t take up usable space), every app is guaranteed a fair share of memcache space.

   Excellent system design GAE engineers. Keep up the good work!

Update: Google App Engine engineers continue to do a very good work indeed! You should take a look at the new features announced with the 1.3.6 release of GAE.

usb 1-1.4: rejected 1 configuration due to insufficient available bus power

The “1-1.4″ ID may be different for your configuration.

If, and only if, you are absolutely sure that your USB hub and/or hardware configuration have a safe way to actually supply enough power, you can override this barrier and force the device to be activated despite of the error message. A possible situation is where you manually applied 5V external power on your USB device and/or USB hub, like I did on my Bifferboard.

Here is how you can override the power safety mechanism:

echo 1 > /sys/bus/usb/devices/1-1.4/bConfigurationValue

Replace “1-1.4″ with your USB device ID. Be careful and have fun!

Resources:

• Error insufficient available bus power RT2573.

I’m using an external hard disk via USB which is partitioned in two parts – /dev/sda1 (1GB) and the rest in /dev/sda2. Once you have installed Debian on Bifferboard, here are the commands which further transform your Bifferboard into a secure NAS:

apt-get update
apt-get -y install nfs-kernel-server

vi /etc/default/nfs-common
  # update: STATDOPTS='--port 2231'
vi /etc/default/nfs-kernel-server
  # update: RPCMOUNTDOPTS='-p 2233'

mkdir -m 700 /root/.ssh
  # add your public key for "root" in /root/.ssh/authorized_keys

echo '/mnt/storage 127.0.0.1(rw,no_root_squash,no_subtree_check,insecure)' >> /etc/exports
mkdir /mnt/storage
chattr +i /mnt/storage # so that we don't accidentally write there without a mounted volume

cat > /etc/rc.local <<EOF
#!/bin/bash

# allow only SSH access via the network
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT
/sbin/iptables -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT # TCP initiated by server
/sbin/iptables -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT # DNS traffic

# mount the storage volume here, so that any errors with it don't interfere with the system startup
/bin/mount /dev/sda2 /mnt/storage
/etc/init.d/nfs-kernel-server restart
EOF

# allow only public key authentication
fgrep -i -v PasswordAuthentication /etc/ssh/sshd_config > /tmp/sshd_config && \
  mv -f /tmp/sshd_config /etc/ssh/sshd_config && \
  echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config

reboot

There are two things you should consider with this setup:

1. You must trust the “root” user who mounts the directory! They have full shell access to your NAS.
2. A not-so-strong SSH encryption cipher is used, in order to improve the performance of the SSH transfer.

On the machine which is being backed up, I use the following script which mounts the NAS volume, starts the rsnapshot backup process and finally unmounts the NAS volume:

#!/bin/bash
set -u

HOST='192.168.100.102'
REMOTEPORT='22'
REMOTEDIR='/mnt/storage'
LOCALDIR='/mnt/storage'
SSHKEY='/home/famzah/.ssh/id_rsa-home-backups'

echo "Mounting NFS volume on $HOST:$REMOTEPORT (SSH-key='$SSHKEY')."
N=0
for port in 2049 2233 ; do
	N=$(($N + 1))
	LPORT=$((61000 + $N))
	ssh -f -i "$SSHKEY" -c arcfour128 -L 127.0.0.1:"$LPORT":127.0.0.1:"$port" -p "$REMOTEPORT" "root@$HOST" sleep 600d
	echo "Forwarding: $HOST: Local port: $LPORT -> Remote port: $port"
done
sudo mount -t nfs -o nfsvers=2,proto=tcp,rsize=8192,wsize=8192,intr,rw,bg,port=61001,mountport=61002 "127.0.0.1:$REMOTEDIR" "$LOCALDIR"

echo "Doing backup."
time sudo /usr/bin/rsnapshot weekly

echo "Unmounting NFS volume and closing SSH tunnels."
sudo umount "$LOCALDIR"
for pid in $(ps axuww|grep ssh|grep 6100|grep arcfour|grep -v grep|awk '{print $2}') ; do
	kill "$pid" # possibly dangerous...
done

Resources:

• Tunneling NFS over SSH « HowtoForge.

Until the numbers are below 008, and until you use the variable only in text interpolations, you’re safe. For example, the following works just fine:

N=016
echo "Value: $N"
# result is "016"

However…
If you start using this variable as a numeric variable in arithmetics, then you’re in trouble. Here is an example:

N=016
echo $((N + 2))
# result is 16, not 18, as expected!
printf %d "$N"
# result is 14, not 16, as expected!

You probably already see the pattern – “016″ is not treated as a decimal number, but as an octal one. Because of the leading zero. This is explained in the man page of bash, section “ARITHMETIC EVALUATION” (aka. “Shell Arithmetic”).

In order to force decimal representation and as a side effect also remove any leading zeros for a Bash variable, you need to treat it as follows:

N=016
N=$((10#$N)) # force decimal (base 10)
echo $((N + 2))
# result is 18, ok
printf %d "$N"
# result is 16, ok

Note also that there’s another caveat – forcing the number to decimal base 10 doesn’t actually validate that it contains only [0-9] characters. Read the very last paragraph of the man page of bash, section “ARITHMETIC EVALUATION” (aka. “Shell Arithmetic”), for more details on how digits can be represented by letters and symbols. My tests however show that you can’t operate with invalid numbers in base 10, though I’m no expert here. In order to be on the safe side, I would suggest that you validate your numbers with a strict regular expression, just in case, and if you don’t trust the data input.

Resources:

• Remove Leading 0’s in Bash « Mike Zupan’s Random Blog

While every Property can be validated automatically by specifying a “validator” function, there is no option for the Model key name to be automatically validated. Note that we can manually specify by our code the value of the key name, and therefore this key name can be considered user-data and must be validated. The key name is by the way the only unique index constraint, similar to the “primary key” in relational databases, which is supported by the Google datastore, and can be specified manually.

Here is my version for a validation function for the Model’s key name:

from google.appengine.ext import db
import re

def ModelKeyNameValidator(self, regexp_string, *args, **kwargs):
	gotKey = None
	className = self.__class__.__name__

	if len(args) >= 2:
		if gotKey: raise Exception('Found key for second time for Model ' + className)
		gotKey = 'args'
		k = args[1] # key_name given as an unnamed argument
	if 'key' in kwargs:
		if gotKey: raise Exception('Found key for second time for Model ' + className)
		gotKey = 'Key'
		k = kwargs['key'].name() # key_name given as Key instance
	if 'key_name' in kwargs:
		if gotKey: raise Exception('Found key for second time for Model ' + className)
		gotKey = 'key_name'
		k = kwargs['key_name'] # key_name given as a keyword argument

	if not gotKey:
		raise Exception('No key found for Model ' + className)

	id = '%s.key_name(%s)' % (self.__class__.__name__, gotKey)
	if (not re.search(regexp_string, k)):
		raise ValueError('(%s) Value "%s" is invalid. It must match the regexp "%s"' % (id, k, regexp_string))

class ClubDB(db.Model):
	# key = url
	def __init__(self, *args, **kwargs):
		ModelKeyNameValidator(self, '^[a-z0-9-]{2,32}$', *args, **kwargs)
		super(self.__class__, self).__init__(*args, **kwargs)

	name = db.StringProperty(required = True)

As you can see, the proposed solution is not versatile enough, and requires you to copy and alter the ModelKeyNameValidator() function again and again for every new validation type. I strictly follow the Don’t Repeat Yourself principle in programming, so after much Googling and struggling with Python, I got to the following solution which I actually use in my projects (click “show source” to see the code):

from google.appengine.ext import db
import re

def re_validator(id, regexp_string):
	def validator(v):
		string_type_validator(v)
		if (not re.search(regexp_string, v)):
			raise ValueError('(%s) Value "%s" is invalid. It must match the regexp "%s"' % (id, v, regexp_string))
	return validator

def length_validator(id, minlen, maxlen):
	def validator(v):
		string_type_validator(v)
		if minlen is not None and len(v) < minlen:
			raise ValueError('(%s) Value "%s" is invalid. It must be more than %s characters' % (id, v, minlen))
		if maxlen is not None and len(v) > maxlen:
			raise ValueError('(%s) Value "%s" is invalid. It must be less than %s characters' % (id, v, maxlen))
	return validator

def ModelKeyValidator(v, self, *args, **kwargs):
	gotKey = None

	if len(args) >= 2:
		if gotKey: raise Exception('Found key for second time for Model ' + self.__class__.__name__)
		gotKey = 'args'
		k = args[1] # key_name given as unnamed argument
	if 'key' in kwargs:
		if gotKey: raise Exception('Found key for second time for Model ' + self.__class__.__name__)
		gotKey = 'Key'
		k = kwargs['key'].name()
	if 'key_name' in kwargs:
		if gotKey: raise Exception('Found key for second time for Model ' + self.__class__.__name__)
		gotKey = 'key_name'
		k = kwargs['key_name']

	if not gotKey:
		raise Exception('No key found for Model ' + self.__class__.__name__)

	v.execute('%s.key_name(%s)' % (self.__class__.__name__, gotKey), k) # validate the key now

class DelayedValidator:
	''' Validator class which allows you to specify the "id" dynamically on validation call '''
	def __init__(self, v, *args): # specify the validation function and its arguments
		self.validatorArgs = args
		self.validatorFunction = v

	def execute(self, id, value):
		if not isinstance(id, basestring):
			raise Exception('No valid ID specified for the Validator object')
		func = self.validatorFunction(id, *(self.validatorArgs)) # get the validator function
		func(value) # do the validation

class ClubDB(db.Model):
	# key = url
	def __init__(self, *args, **kwargs):
		ModelKeyValidator(DelayedValidator(re_validator, '^[a-z0-9-]{2,32}$'), self, *args, **kwargs)
		super(self.__class__, self).__init__(*args, **kwargs)

	name = db.StringProperty(
		required = True,
		validator = length_validator('ClubDB.name', 1, None))

You probably noticed that in the second example I also added a validator for the “name” property too. Note that the re_validator() and length_validator() functions can be re-used. Furthermore, thanks to the DelayedValidator class which accepts a validator function and its arguments as constructor arguments, the ModelKeyValidator class can be re-used without any modifications too.

P.S. It seems that all “validator” functions are executed every time a Model class is being instantiated. This means that no matter if you are updating/creating the data object, or you are simply reading it from the datastore, the assigned values are always validated. This surely wastes some CPU cycles, but for now I have no idea how to easily circumvent this.

Disclaimer: I’m new to Python and Google App Engine. But they seem fun! Sorry for the long lines…

Resources:

• Google App Engine: Model integrity constraints? << Stack Overflow

The benchmarked scripts do only one thing – say “Hello, world” on the standard output. They do not include any additional modules in their source code – this may, or may not be your use-case. Though, very often the scripting languages have pretty many built-in functions, and for simple tasks you never need to include other modules.

Here are the benchmark results:

The clear winner among the script languages this time is… Perl.

All scripts were invoked 3000 times using the following Bash loop:

time ( i=3000 ; while [ "$i" -gt 0 ]; do $CMD >/dev/null ; i=$(($i-1)); done )

For complete information about the test environment, please review the previous article.

The C++ implementation follows, click “show source” below to see the full source:

#include <iostream>
using namespace std;

int main() {
	cout << "Hello, world!\n";
	return 0;
}

The Perl implementation follows, click “show source” below to see the full source:

use strict;
use warnings;

print "Hello, world!\n";

The PHP implementation follows, click “show source” below to see the full source:

<?php
echo "Hello, world!\n";

The Python implementation follows, click “show source” below to see the full source:

#import psyco
#psyco.full()

print 'Hello, world!'

The problem
If you are using RRDtool and Perl as described, you surely have noticed that running many of these processes wastes a lot of CPU resources. The question is – can we do some performance optimizations, and lessen the performance hit of loading the RRDs library into Perl? We know that launching often Perl itself is quite expensive, but after all, if we chose to work with Perl, this is a price we should be ready to pay.

The RRDtool shared library is a monolithic piece of code which provides ALL functions of the RRDtool suite – data manipulation, graphics and import/export tools. The last two components bring huge dependencies in regards to other shared libraries. The library from RRDtool version 1.4.4 depends on 34 other libraries on my Linux box! This must add up to the loading time of the RRDtool library into Perl.

Resolution and benchmarks
In order to prove my theory (actually, it was more a theory of zImage, and I just followed, enhanced and tried it), I commented out the implementation of the “graphics” and “import/export tools” modules from the source code of RRDtool. Then I re-compiled the library and did some performance benchmarks. I also re-implemented the RRDs.pm module by replacing the DynaLoader module with the XSLoader one. This made no difference in performance whatsoever. The re-compiled RRD library depends on only 4 other libraries – linux-gate.so.1, libm.so.6, libc.so.6, and /lib/ld-linux.so.2. I think this is the most we can cut down.

So here are the benchmark results. They show the accumulated time for 1000 invocations of the Perl interpreter with three different configurations:

• Only Perl (baseline): 5.454s.
• With RRDs, no graphics or import/export functions: 9.744s (+4.290s) +78%.
• With standard RRDs: 11.647s (+6.192s) +113%.

As you can see, you can make Perl + RRDs start 35% faster. The speed up for RRDs itself is 44%.

Here are the commands I used for the benchmarks:

• Only Perl (baseline): time ( i=1000 ; while [ "$i" -gt 0 ]; do perl -Mwarnings -Mstrict -e ” ; i=$(($i-1)); done )
• Perl + RRDs: time ( i=1000 ; while [ "$i" -gt 0 ]; do perl -Mwarnings -Mstrict -MRRDs -e ” ; i=$(($i-1)); done )

After much reading at their page, what people say was confirmed – StartSSL really issue SSL certificates for free, when they are about to be used by individuals on their websites. This means that your personal name stays in the SSL certificate information which can be reviewed if you click on the SSL bar in your web browser.

Business or other legal entities verify their company’s information once for an annual fee and can then issue an unlimited count of SSL certificates too, including wild-card ones. Once verified, a business customer can purchase EV certificates for US$ 49.90 per year.

You can compare these prices with any other SSL certificate authority and you’ll see it yourself that StartSSL are the most affordable one, and the only one which doesn’t charge you for what doesn’t cost them money either – that’s why they can offer “loosely verified” SSL certificates for personal websites for free. It’s unbelievable but true.

My IT brain immediately started to doubt the technical side. I had to check if web browsers accept these SSL certificates without issuing an SSL warning about the certificate being signed by an unknown SSL authority. The test results were successful and the SSL root authority of StartSSL was recognized by the latest version of:

• Internet Explorer 8 on Windows.
• Chrome on Windows.
• Firefox on Windows and Linux.
• Chromium on Linux.

Furthermore, the Debian “lenny”, “squeeze” and Ubuntu Lucid CA repositories also recognize the StartSSL root certificate. You can verify this yourself by the following command:
openssl s_client -CApath /etc/ssl/certs -connect startssl.com:443

StartSSL have a long list with platforms and browsers which recognize their certificates. You can review the list at the products comparison page.

No more self-signed SSL certificates for personal use, hurray!

Today, it’s the “Launch Console” button doing absolutely nothing on my Kubuntu desktop – no errors, no action after clicking it, no nothing. I (always) have a “backup option” – a Windows 7 virtual machine running on my desktop, as Java tends to work better for me on Windows (cross-platform, eh?). Same problem on the Windows too. As I’m a real paranoid about having a backup, I have a backup of the “backup option” – X over VNC, running on
some not-so-bleeding-edge Linux machines, in order to have a “stable” Java installation there. Though the Java failed on them today as well, as they are running Debian “lenny”, which seems to be having the latest Java version 1.6.20 too.

Well… sorry Java applets + Super Micro IPMI, you really disappoint me! :-/

The fix is to downgrade your Java to 1.6.19 (and disable automatic Java updates):
http://www.webhostingtalk.com/showthread.php?t=953055

Update #1: I downgraded to Java 1.6.19 on my Windows 7 by:

1. Uninstalling the Java 1.6.20 JRE update.
2. Installing the Java 1.6.19 JRE update which I downloaded from the “Archive: Java[tm] Technology Products Download” page.
3. Being able to get this working only with Chrome. Firefox and IE 8 failed to work.

Update #2: Linux doesn’t seem to be having any problems. Firefox 3.6.3 on Ubuntu and Gentoo with Sun Java 1.6.20 works fine.

Update #3: If you upgrade the IPMI firmware to version 2.02, the Windows problem is fixed.

Here is some debug info from the Debian “lenny” Iceweasel browser, the only one which issued an error:

Unable to launch ATEN Java iKVM Viewer.
An error occurred while launching/running the application.

Title: ATEN Java iKVM Viewer
Vendor: ATEN
Category: Download Error

Unable to load resource: (https://%IP%/iKVM.jar, 1.56.3.0×0)

Wrapped Exception: java.io.IOException: HTTP response 404.

At the same time, the Java test page works fine. The version on the Debian “lenny” “sun-java6-jre” package is “6-20-01lenny1″ (Java JRE 1.6.20).

The same problem is re-produced on:

• Windows 7, running Java 1.6.20, under IE 8, Firefox 3.6.3 and Chrome 5.0.375.99.
• Kubuntu Lucid, running OpenJDK 6 build b18, under Firefox 3.6.3.

The Firmware Revision of the IPMI interface on the X8DTL motherboard is 01.29, dated 2010-01-06. It’s not the latest one, but surely not a very old one. After all, you can’t reboot your production servers for every IPMI firmware release…

Anyway, I try not to write articles with negative attitude, but this time I just couldn’t resist.
Java, Java, Java…

This all began as a colleague of mine stated that Python was so damn slow for maths. Which really astonished me and made me check it out, as my father told me once that he was very satisfied with Python, as it was very maths oriented.

The benchmarks here do not try to be complete, as they are showing the performance of the languages in one aspect, and mainly: loops, arrays with numbers, basic math operations.

Update: Give your ideas and use-cases on what to benchmark, and I’ll try to implement it for you. I.e. “benchmark the languages for reading a file, then splitting it to tokens by white-space and finally outputting all unique elements and their count”.

Out of curiosity, Python was also benchmarked with and without the Psyco Python extension, which people say could greatly speed up the execution of any Python code without any modifications.

Here are the benchmark results:

The clear winner among the script languages is… Python.

The times include the interpretation/parsing phase for each language, but it’s so small that its significance is negligible. The math function is called 10 times, in order to have more reliable results. All scripts are using the very same algorithm to calculate the prime numbers in a given range. The correctness of the implementation is not so important, as we just want to check how fast the languages perform. The original Python algorithm was taken from http://www.daniweb.com/code/snippet216871.html.

All tests were done on a Kubuntu Lucid box. The versions of the used software packages follow:

• g++ (GNU project C and C++ compiler) 4.4.3
• Python 2.6.5
• Python Psyco 1.6 (1ubuntu2)
• Perl 5.10.1
• PHP 5.3.2 (1ubuntu4.2 with Suhosin-Patch), Zend Engine 2.3.0

The C++ implementation follows, click “show source” below to see the full source:

#include <cstdio>
#include <cmath>
#include <vector>

using namespace std;

vector<int> get_primes7(int n) { // ugly variable declarations but close to the other lang. syntaxes
	vector<int> res;

	if (n < 2) return res;
	if (n == 2) {
		res.push_back(2);
		return res;
	}
	vector<int> s;
	for (int i = 3; i < n + 1; i += 2) {
		s.push_back(i);
	}
	int mroot = sqrt(n);
	int half = (int)s.size();
	int i = 0;
	int m = 3;
	while (m <= mroot) {
		if (s[i]) {
			int j = (int)((m*m - 3)/2);
			s[j] = 0;
			while (j < half) {
				s[j] = 0;
				j += m;
			}
		}
		i = i + 1;
		m = 2*i + 3;
	}
	res.push_back(2);
	for (vector<int>::iterator it = s.begin() ; it < s.end(); ++it) {
		if (*it) {
			res.push_back(*it);
		}
	}

	return res;
}

int main() {
	vector<int> res;
	for (int i = 1; i <= 10; ++i) {
		res = get_primes7(10000000);
		printf("Found %d prime numbers.\n", (int)res.size());
	}

	return 0;
}

The Python implementation follows, click “show source” below to see the full source:

#import psyco
#psyco.full()

def get_primes7(n):
	"""
	standard optimized sieve algorithm to get a list of prime numbers
	--- this is the function to compare your functions against! ---
	"""
	if n < 2:  return []
	if n == 2: return [2]
	# do only odd numbers starting at 3
	s = range(3, n+1, 2)
	# n**0.5 simpler than math.sqr(n)
	mroot = n ** 0.5
	half = len(s)
	i = 0
	m = 3
	while m <= mroot:
		if s[i]:
			j = (m*m-3)//2  # int div
			s[j] = 0
			while j < half:
				s[j] = 0
				j += m
		i = i+1
		m = 2*i+3
	return [2]+[x for x in s if x]

for t in range(10):
	res = get_primes7(10000000)
	print "Found", len(res), "prime numbers."

The Perl implementation follows, click “show source” below to see the full source:

use strict;
use warnings;

sub get_primes7($) {
	my ($n) = @_;

	if ($n < 2) { return (); }
	if ($n == 2) { return (2); }
	# do only odd numbers starting at 3
	my @s = ();
	for (my $i = 3; $i < $n + 1; $i += 2) {
		push(@s, $i);
	}
	# n**0.5 simpler than math.sqr(n)
	my $mroot = $n ** 0.5;
	my $half = scalar @s;
	my $i = 0;
	my $m = 3;
	while ($m <= $mroot) {
		if ($s[$i]) {
			my $j = int(($m*$m - 3) / 2);
			$s[$j] = 0;
			while ($j < $half) {
				$s[$j] = 0;
				$j += $m;
			}
		}
		$i = $i + 1;
		$m = 2*$i + 3;
	}
	my @res = (2);
	foreach (@s) {
		push(@res, $_) if ($_);
	}
	return @res;
}

my @res;
for (1..10) {
	@res = get_primes7(10000000);
	print "Found ".(scalar @res)." prime numbers.\n";
}

The PHP implementation follows, click “show source” below to see the full source:

<?php
error_reporting(E_ALL);
ini_set('display_errors', '1');

function get_primes7($n) {
	if ($n < 2) return array();
	if ($n == 2) return array(2);
	$s = array();
	for ($i = 3; $i < $n + 1; $i += 2) {
		$s[] = $i;
	}
	$mroot = sqrt($n);
	$half = count($s);
	$i = 0;
	$m = 3;
	while ($m <= $mroot) {
		if ($s[$i]) {
			$j = (int)(($m*$m - 3) / 2);
			$s[$j] = 0;
			while ($j < $half) {
				$s[$j] = 0;
				$j += $m;
			}
		}
		$i = $i + 1;
		$m = 2*$i + 3;
	}
	$res = array(2);
	foreach ($s as $v) {
		if ($v) {
			$res[] = $v;
		}
	}
	return $res;
}

$res = array();
for ($i = 1; $i <= 10; ++$i) {
	$res = get_primes7(10000000);
	print "Found ".count($res)." prime numbers.\n";
}

Update (Jul/24/2010): Added the C++ optimized values.
Update (Aug/02/2010): Added a link to the benchmarks, part #2.