/contrib/famzah

Enthusiasm never stops


9 Comments

Secure chroot() remote file access via SFTP and SSH

If you want to replace the buggy and not-encrypted FTP protocol, and get rid of the FTP daemon on your system, the SFTP protocol comes to the rescue. Note that the SFTP protocol is something more than the SCP protocol (Secure copy), as it provides resuming interrupted transfers, directory listings, and remote file removal. This makes it more similar to the FTPS protocol (FTP over SSL) with the difference that it doesn’t require a separate FTP daemon, because the SSH daemon supports SFTP, which simplifies your network setup and lowers maintenance costs.

A standard security feature of the FTP servers is that logged in users are placed in a chroot jail directory, which restricts users from viewing and manipulating any other files but their own ones. Fortunately, the OpenSSH daemon supports chroot() too — see the sshd_config(5) man page.

Now that I’ve convinced you that SFTP is the right way to go for secure file transfers and remote file mounts, let’s see how we can configure it on a Debian or Ubuntu based server. All SFTP users will be kept in a root chroot() directory:

mkdir -m 751 /home/sftp-chroot /home/sftp-chroot/{dev,home}

The man page says that all components of the pathname must be root-owned directories that are not writable by any other user or group. Let’s verify this before going further:

ls -lad / /home /home/sftp-chroot /home/sftp-chroot/home
drwxr-xr-x 23 root root 4096 2011-01-31 17:15 /
drwxr-x--x  9 root root 4096 2011-01-31 18:05 /home
drwxr-x--x  4 root root 4096 2011-01-31 17:38 /home/sftp-chroot
drwxr-x--x  3 root root 4096 2011-02-01 08:46 /home/sftp-chroot/home

SFTP users’ home directories will be placed in “/home/sftp-chroot/home/$SFTPUSER” (after the chroot() the actual directory would be “/home/$SFTPUSER”). Note that this still gives full privacy, because users can’t list the root chroot() directory “/” nor the root home directory “/home”, as their permissions are 0751. Furthermore, even if one user guessed the username and home directory of another user, they cannot enter their home directory, because the users’ home directories give no permissions for the group “others”. The commands for managing users’ home directories are at the very end of this article.

In order for the chroot()’ed users and processes to be able to do logging, a listening UNIX socket of syslog must be created inside the root chroot() directory:

cat > /etc/rsyslog.d/sftp-chroot.conf <<'EOF' # the single quotes are important
# the following syslog socket will be used by all chrooted users
$AddUnixListenSocket /home/sftp-chroot/dev/log

# Log internal-sftp activity in a separate file
:programname, isequal, "internal-sftp" -/var/log/sftp.log
:programname, isequal, "internal-sftp" ~
EOF

/etc/init.d/rsyslog restart

We verify that the syslog UNIX socket was created:

ls -la /home/sftp-chroot/dev/log
srw-rw-rw- 1 root root 0 2011-01-31 16:42 /home/sftp-chroot/dev/log

We will also rotate the log files weekly:

cat > /etc/logrotate.d/sftp-chroot <<'EOF'
/var/log/sftp.log {
  weekly
  missingok
  rotate 52
  compress
  delaycompress
  postrotate
    invoke-rc.d rsyslog reload > /dev/null
  endscript
}
EOF

Finally, let’s set up the OpenSSH daemon accordingly:

# disable the standard "sftp" subsystem in the sshd config
perl -pi -e 's/^(\s*Subsystem\s+sftp\s+)/#$1/i' /etc/ssh/sshd_config

# enable the chroot "sftp" subsystem
cat >> /etc/ssh/sshd_config <<'EOF'

Subsystem sftp internal-sftp

Match group sftponly
  ChrootDirectory /home/sftp-chroot
  ForceCommand internal-sftp -l VERBOSE
  AllowTcpForwarding no
  PermitRootLogin no
  X11Forwarding no
  AllowAgentForwarding no
  # Change to yes to enable tunnelled clear text passwords, not recommended
  PasswordAuthentication no
EOF

groupadd sftponly
/etc/init.d/ssh restart

A few notes about the OpenSSH configuration:

  • This configuration requires OpenSSH 5.2 or later — discussion about this in the reference links below.
  • The SCP protocol is not supported, at least not very easily. If you really need SCP, you’d need to re-create a full binary environment, in order to provide standard SSH access in a chroot() way. This is not discussed here, nor tested by me. You are encouraged to use SFTP instead. [source article]
  • Some users report that the time-stamp of the messages in the syslog log file were wrong. This is indeed possible since there is no /etc/timezone file in the root chroot() directory. I haven’t encountered this bug, as my system is in GMT+0, or the bug no longer exists. [source article]

In the end, let’s take a look at Users management (sample Bash script below in the comments).

Adding an SFTP user:

export SFTPUSER='testsftp1'
useradd -d "/home/$SFTPUSER" --groups sftponly -s /bin/false -p '!' "$SFTPUSER"
passwd "$SFTPUSER" # if you enabled tunnelled clear text passwords, regardless of the security implications
mkdir -m 770 "/home/sftp-chroot/home/$SFTPUSER"
chown root:"$SFTPUSER" "/home/sftp-chroot/home/$SFTPUSER"

Enabling public/private key authentication, the more secure choice, for an SFTP user:

export SFTPUSER='testsftp1'
mkdir -m 750 "/home/$SFTPUSER" "/home/$SFTPUSER/.ssh"
chown root:"$SFTPUSER" "/home/$SFTPUSER" "/home/$SFTPUSER/.ssh"
vi "/home/$SFTPUSER/.ssh/authorized_keys" # add the public key

Deleting an SFTP user and its data:

export SFTPUSER='testsftp1'
userdel "$SFTPUSER"
rm -r "/home/sftp-chroot/home/$SFTPUSER"
rm -r "/home/$SFTPUSER" # if you authenticate using public keys

The end-result of all those efforts is that when an SFTP user logs in to your system, they see only their home directory and nothing else. This is because after the chroot, sshd(8) changes the working directory to the user’s home directory. Furthermore the user cannot list nor enter other home directories in the root chroot() directory, because of the directory setup we created.

Here is a sample log from the SFTP syslog file, click “show source” to view it:

Jan 31 17:55:10 m1 internal-sftp[13237]: session opened for local user testsftp1 from [127.0.0.1]
Jan 31 17:55:10 m1 internal-sftp[13237]: received client version 3
Jan 31 17:55:10 m1 internal-sftp[13237]: realpath "."
Jan 31 17:55:12 m1 internal-sftp[13237]: open "/home/testsftp1/fb.php" flags WRITE,CREATE,TRUNCATE mode 0644
Jan 31 17:55:13 m1 internal-sftp[13237]: close "/home/testsftp1/fb.php" bytes read 0 written 735
Jan 31 18:04:14 m1 internal-sftp[13237]: session closed for local user testsftp1 from [127.0.0.1]

References:


Leave a comment

ResellerClub technical problems with DNS and Domain Forwarding

Today I woke up and found out that many of my domains aren’t working. I’m using ResellerClub (aka. DirectI) free DNS and free Domain Forwarding. Well, they deleted all my DNS records. The Domain Forwarding service stopped working too.

I’ve filed a support ticket with an “Emergency” priority. Let’s see what happens now… I’ll keep you updated.

UPDATE: Instead of just fixing things, ResellerClub really disappointed me. I am their long-term and very faithful customer, but now I see that I shouldn’t have trusted them so much.

It took two days for ResellerClub’s support to respond with the following ridiculous statement:

DNS records are merely record entries made in the server, and not a space occupying entity for which a ‘backup’ would be available/generated.
Hence there is no backup for the records at our end.

ResellerClub keep no backup whatsoever of their DNS and Domain Forwarding configurations! The explanation of this fact is hilarious — because DNS records do not occupy space. 🙂

This really pissed me off, and I asked them how they could operate without a backup. What if the disks of their DNS or database servers fail? What if an operator deletes a record by mistake? The answer by ResellerClub’s support was that they monitor their servers, so this is not an issue. Ha-ha! Since when is monitoring a substitution of backups?? Furthermore, they added that the system was fully automated, so no operator’s mistake was possible. I replied that even if the system operates automatically, it is still being maintained by humans, who may delete data by mistake. How would they explain why my DNS and forwarding data was lost…

Anyway, no need to dig into this any further. I re-created the DNS records, and also learnt an important lesson — never trust an (IT) organization unless you really know that they operate in a professional way by following all well-established principles in the industry. ResellerClub ain’t one of them!


1 Comment

И с работа се изкарват пари…

…според късметите на Дядо Коледа за новата 2011 година. 🙂

За първа година ми се пада такова малко “количество” късмети — околосветско пътешествие и пари + малката монета. Пари, пари, ама как с толкова малко късмет — явно с повече работа.
От друга страна, парите купуват почти всичко. 😉

Успешна 2011 година на всички!


2 Comments

StartSSL customer care

A few days ago I had to go through the “Class 2 Identity Validation (StartSSL™ Verified)” process, so that I can issue an unlimited count of wild-card certificates.

Before even getting any deeper in the process, I first contacted StartSSL and asked them if they stand behind their words, that once verified (one time-fee $49.90/yr), a customer of theirs can issue an unlimited count of wild-card certificates. I got a pretty quick response by StartSSL stating that this is indeed true, there’s no catch.

After that I started the process of verification – registered, submitted documents, etc. There were many unknown things to me and I had to ask their support many questions (I won’t go into details as I can’t disclose all of it), and we’re almost at the end – they send me a verification postal mail. Once I get the code, we should be done.

But this post is not only about my success with StartSSL – it’s about their very quick, accurate and helpful Support service. You guys rock!


5 Comments

sudo hangs and leaves the executed program as “zombie”

Today I discovered a non-security bug in sudo – the executed program finishes successfully, but sudo hangs forever waiting for something. The executed program is left in a “zombie” process state. Here is how the process list looks like, for example:

root 6368 0.0 0.0 2808 1592 pts/6 Ss 18:39 0:00 | \_ -bash
root 1103 0.0 0.0 2200 1000 pts/6 S+ 21:45 0:00 | \_ ./sudo -u root sleep 5
root 1104 0.0 0.0 0 0 pts/6 Z+ 21:45 0:00 | \_ [sleep] <defunct>

If we try to trace the system calls of the sudo command, here is what we get:

[root@tester2 ~]# strace -fF -p 1103
select(4, [3], [], NULL, NULL

The sudo process waits endlessly in a select() system call which waits for file descriptor #3. So we quickly check what corresponds to file descriptor #3:

[root@tester2 ~]# ls -la /proc/1103/fd
total 0
dr-x—— 2 root root 0 Nov 1 21:45 .
dr-xr-xr-x 5 root root 0 Nov 1 21:45 ..
lrwx—— 1 root root 64 Nov 1 21:45 0 -> /dev/pts/6
lrwx—— 1 root root 64 Nov 1 21:46 1 -> /dev/pts/6
lrwx—— 1 root root 64 Nov 1 21:45 2 -> /dev/pts/6
lrwx—— 1 root root 64 Nov 1 21:46 3 -> socket:[1136261781]

Socket “socket:[1136261781]” is already closed tough (blinking in red on my console). Thus sudo is waiting in a blocked select() for a change on file descriptor #3, which is already closed and will never change its state. Sudo will therefore wait forever.

In order to understand why this happens, we will look at the source code of sudo. Here is snippet from “exec.c” – only the relevant code is left for clarity:

int
sudo_execve(path, argv, envp, uid, cstat, dowait, bgmode)
{
    int sv[2];

    if (socketpair(PF_UNIX, SOCK_DGRAM, 0, sv) != 0)
    error(1, "cannot create sockets");

    zero_bytes(&sa, sizeof(sa));
    sigemptyset(&sa.sa_mask);

    /* Note: HP-UX select() will not be interrupted if SA_RESTART set */
    sa.sa_flags = SA_INTERRUPT; /* do not restart syscalls */
    sa.sa_handler = handler;
    sigaction(SIGCHLD, &sa, NULL);
    sigaction(SIGHUP, &sa, NULL);
    sigaction(SIGINT, &sa, NULL);
    sigaction(SIGPIPE, &sa, NULL);
    sigaction(SIGQUIT, &sa, NULL);
    sigaction(SIGTERM, &sa, NULL);

    /* Max fd we will be selecting on. */
    maxfd = sv[0];

    child = fork()
    close(sv[1]);

    fdsr = (fd_set *)emalloc2(howmany(maxfd + 1, NFDBITS), sizeof(fd_mask));
    fdsw = (fd_set *)emalloc2(howmany(maxfd + 1, NFDBITS), sizeof(fd_mask));

    for (;;) {

    zero_bytes(fdsw, howmany(maxfd + 1, NFDBITS) * sizeof(fd_mask));
    zero_bytes(fdsr, howmany(maxfd + 1, NFDBITS) * sizeof(fd_mask));

    FD_SET(sv[0], fdsr);

    if (recvsig[SIGCHLD])
        continue;
    nready = select(maxfd + 1, fdsr, fdsw, NULL, NULL);
    if (nready == -1) {
        if (errno == EINTR)
        continue;
        error(1, "select failed");
    }

    }
}

/*
 * Generic handler for signals passed from parent -> child.
 * The recvsig[] array is checked in the main event loop.
 */
void
handler(s)
    int s;
{
    recvsig[s] = TRUE;
}

The race-condition happens right before the select() on line #40, and just after the “if” on lines #38 and #39. If the parent process gets re-scheduled after the “if” was executed, and at this very time the child process finishes and SIGCHLD is sent to the parent process, sudo gets in trouble. The SIGCHLD handler accounts in the variable “recvsig[]” that the signal was received, and then the parent process calls select(). This select will never be interrupted, as the author had it in mind. In 99% of the cases, the parent process will enter in the select() blocking state before the child process ended. The child would then send SIGCHLD, which will be accounted in the handler procedure, and will also interrupt select() which will return -1 in “nready”, and “errno” will be set to EINTR.

Here is an easy way to reproduce the bug. We add a sleep() of 10 seconds between the “if” and select(), thus simulating that the system was very busy and re-scheduled the parent sudo process right between these two operations. Here is the source diff:

--- sudo-orig/sudo-1.7.4p4/exec.c       Sat Sep  4 00:40:19 2010
+++ sudo-1.7.4p4/exec.c Mon Nov  1 21:48:24 2010
@@ -307,6 +307,10 @@
 
        if (recvsig[SIGCHLD])
            continue;
+       printf("debug: Missed the check for SIGCHLD, the child is still running. SIGCHLD status: %d\n", recvsig[SIGCHLD]);
+       sleep(10); // this will be interrupted by SIGCHLD, because the child exists at some time here (we run "sudo sleep 5")
+       printf("debug: We should have got SIGCHLD by now. SIGCHLD status: %d\n", recvsig[SIGCHLD]);
+       printf("debug: Entering the endless select()...\n");
        nready = select(maxfd + 1, fdsr, fdsw, NULL, NULL);
        if (nready == -1) {
            if (errno == EINTR)

After that we execute sudo, and observe the bug every time:

[root@tester2 sudo-1.7.4p4]# make >/dev/null && ./sudo -u root sleep 5
debug: Missed the check for SIGCHLD, the child is still running. SIGCHLD status: 0
debug: We should have got SIGCHLD by now. SIGCHLD status: 1
debug: Entering the endless select()…
…(this never finishes)…

The sudo author actually tried to avoid this potential race condition if SIGCHLD is received immediately
before we call select() – changeset 5334:99adc5ea7f0a. The proper way to fix this is to use a timeout in the select() call:

--- sudo-orig/sudo-1.7.4p4/exec.c       Sat Sep  4 00:40:19 2010
+++ sudo-1.7.4p4/exec.c Mon Nov  1 21:50:26 2010
@@ -307,7 +307,11 @@
 
        if (recvsig[SIGCHLD])
            continue;
-       nready = select(maxfd + 1, fdsr, fdsw, NULL, NULL);
+       struct timeval timeout;
+       timeout.tv_sec = 1; // Linux resets this, so set it everytime
+       timeout.tv_usec = 0;
+       nready = select(maxfd + 1, fdsr, fdsw, NULL, &timeout);
        if (nready == -1) {
            if (errno == EINTR)
                continue;

The select() mechanism and this bug were introduced somewhere between sudo versions 1.7.2 and 1.7.3. At least that is what I managed to see from the Changelog:

2010-06-29  Todd C. Miller  <Todd.Miller@courtesan.com>
	[72fd1f510a08] [SUDO_1_7_3] <1.7>
...
2009-11-15  Todd C. Miller  <Todd.Miller@courtesan.com>
	* script.c, sudo.c, sudo.h, sudoreplay.c, term.c, tgetpass.c:
	Use a socketpair to pass signals from parent to child.
...
2009-07-12  Todd C. Miller  <Todd.Miller@courtesan.com>
	[f5ad45f69f05] [SUDO_1_7_2]

I’ve reported this bug (#447) to the sudo maintainer, and I’m sure he will fix it when time permits. Because we all depend on sudo and love it. 🙂


Leave a comment

Google App Engine Performance Profiling

When developing under Google App Engine, developers need to pay attention to how fast their code works, and also how much resources their code uses. The first parameter is vital for user-experience, the second – for the hosting expenses, as resources usage costs money.

It turns out that there are quite a few suitable profilers for Google App Engine. At least these are the ones I could find:

  • Appstats – no doubt, the most advanced one, giving you information about the timing and costs of your RPC calls to the datastore, Memcache, etc.
    Works for both Python, and Java. Included in the official SDK as of version 1.3.1.
  • appengine-profiler – besides being an RPC profiler like Appstats, appengine-profiler gives you the option to profile the CPU usage of your code, thus you can easily identify hot-spots where your code wastes a lot of CPU resources and wall-clock time. You can define “tracepoints” which surround part of your code blocks, and you’ll easily know the resources usage of these blocks for each page load.
    Works for Python.
  • AppWrench – the Java profiler. I don’t code on Java, and I haven’t tested this profiler, but I’m including it here for all you Java gurus.

Resources:


10 Comments

Linux Cached/Buffers memory

I won’t try to explain in details what Linux Cached/Buffers memory is. In a nutshell, it shows how much of your memory is used for the read cache and for the write cache.

Usually when you look at your system memory usage and see that almost all of the unused memory is allocated for Cached/Buffers, you are happy, because this memory is used for file-system cache, thus your system is running faster.

Today however I observed quite an interesting fact – what I said above is still correct, however you don’t know how often these cache entries are used by the system. After all, it’s not the cache memory usage (or size) which makes the system run faster, but the cache hit ratio. If the file operations get satisfied by the cache (cache hit), then your system is running faster. If the system needs to make a physical disk I/O (cache miss), then you’d need to wait for a good few milliseconds.

What are your options, in order to know if your file cache is actually being used or is just sitting there allocated, giving you a false feeling that your system is running faster thanks to the used cache memory:

  1. (hard) In order to actually know the Cache Hit/Miss ratios for block devices, you’ll need to dig deep into the kernel, as I already explained in the “Is there a way to get Cache Hit/Miss ratios for block devices in Linux” article.
  2. (easy) You can clear the Cached/Buffers memory regularly, see how fast and how much the cache memory grows back, and draw some conclusions about the actual Cache Hit/Miss ratios.

The latter is not a perfect solution, but in all cases gives you a better idea of your file-system cache usage, than just watching the totally used memory in Cached/Buffers, and never actually knowing if it is used/accessed at all.

Therefore, you can run the following every hour in a cron job:

sync ; echo 3 > /proc/sys/vm/drop_caches
sync ; echo 0 > /proc/sys/vm/drop_caches

The commands are safe (see reference for “drop_caches“), and you won’t lose any data, just your caches will be zeroed. The disadvantage of this approach is that if the caches were very actively used indeed, Linux would need to read the data back from the disk.

A real-world example

 

Here is how the Cached/Buffers graphics of a server of mine looks for the following few days:

Linux memory usage

Pay attention to the points of interest which are marked. Here is the explanation and motivation to write this article:

  1. (Point A) The beginning of the graph shows my system after it just booted, and I did some small administrative tasks on it. After that, one script runs regularly on the machine, and as we see, it doesn’t use much file-system cache, as it doesn’t do many file operations.
  2. Then every day at 06:25 the “/etc/cron.daily” scripts are run and some of them read all files on the file-system. Such a script is the updatedb cron job. Because of the great disk activity, the Buffers/Cache usage gets maximal, as all possible files and meta data are being cached in memory.
  3. (see “Updatedb cron” markers on the graphics) After one hour, the scripts finish and no significant disk activity is done on the system any more.
  4. (Point B) But the Cached/Buffers usage never drops down. The file cache doesn’t seem to expire, and is therefore giving us the false feeling that it is being used by our system all the time, thus making it faster. But it isn’t!
  5. On Sat 15:08 the Cached/Buffers cache is cleared manually by the command I provided above, and I installed it as an hourly crontab too.
  6. As you can see, right after the cache was cleared, we see the sad reality – the Cached/Buffers cache was filled with data that nobody needed or accessed, and the high memory usage by Cached/Buffers actually didn’t speed up our system. I grieve for a while and accept the reality, and also understand why so many I/O requests are issued to my EBS storage, even though the cache was so huge.
  7. (Point C) That’s how the actual daily usage pattern of this machine looks like. The Cached/Buffers memory cache is heavily underused on my system, as it doesn’t do much I/O work. This wouldn’t be visible if I don’t clear the cache every hour.

References:


Leave a comment

PHP non-interactive usage in a cron job

Using a PHP script in a crontab is fairly easy, as stated in the “Using PHP from the command line” documentation… Until you start to get the following warning during the execution:

No entry for terminal type “unknown”;
using dumb terminal settings.

The script works, but this nasty warning really bothers you.

Here is a sample crontab entry:

* * * * * root sudo -u www-data php -r ‘echo “test”;’

When executed, it prints the warning on STDERR.

Yes, I know I don’t need “sudo” here, but this was my initial usage pattern as I discovered the problem, and at the first time I suspected that “sudo” got crazy. Well, it wasn’t “sudo” to blame, but PHP.

Here is the fixed crontab entry:

* * * * * root sudo -u www-data TERM=dumb php -r ‘echo “test”;’

The issue was encountered on an Ubuntu 10.04 server. I though crond usually sets $TERM to something… Anyway, problem solved.


Leave a comment

Getting “500 Line too long (limit is 4096)” error in Perl

The error may also be “500 Line too long (limit is 8192)” but the problem is still the same – LWP or SOAP::Lite return this error when you try to POST or GET something very long.

The one to blame is actually Net::HTTP::Methods, included somewhere by something.

It took me a few hours to get this resolved:

use LWP::Protocol::http; # to suppress the warning "possible typo" in the next statement
push(@LWP::Protocol::http::EXTRA_SOCK_OPTS, MaxLineLength => 0); # to remove the limit

Put the above code in your Perl HTTP client and you’re good to go!

References: