When I received the phone call with the message that the server was unreachable, I immediately logged into the server. The Apache daemon had crashed, and the error log was filled with

[emerg] (43)Identifier removed: couldn't grab the accept mutex
...
[emerg] (22)Invalid argument: couldn't grab the accept mutex

After fiddling about a bit with AcceptMutex (to no avail), I noticed the following lines, preceding the ones about mutexes:

[alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 33
[alert] Child $p returned a Fatal error... Apache is exiting!

A resource exhaustion! The Apache server tried to use system resources, but ran into a limitation. Had I set my limits.conf wrong? Turns out, kind of.

To protect the server agains processing scripts that go haywire, there is a limit of 25 processes for all users that are part of the Indianio platform (group indianio). As it happens, the account that I use to administer server, is also a member of this group. But that poses no problem, because everything happens through sudo, which resets the rlimits, right?

Well, on my server, it didn't:

davy@indianio:~$ ulimit -u
25
davy@indianio:~$ sudo bash -c "ulimit -u"
25

This limitation was also imposed on the Apache daemon and its subprocesses, after restarting it with sudo /etc/init.d/apache2 restart. The output of the CGI-script I installed to verify my suspicion:

id
+ id
uid=33(www-data) gid=33(www-data) groups=33(www-data)
ulimit -u
+ ulimit -u
25

Oops...

Note that this doesn't happen when restaring Apache with apachectl, because that doesn't create a new process.

I feel like a bug digging throught an onion!

Instead of trying to improve my probably poor attempt at eliminating noise, I ordered a usb2ethernet thingy from DealExtreme. The idea was that this would suffice until I could replace the NS-K330 with a Raspberry Pi. But alas, delivery from Hong Kong seems to take more time than usual. Luckily, my ADSL-router also has a USB network connection: snake is now has an eth1 device, provided by usbnet and cdc_ether. The connection is only USB1.1,  so the speed is not really that high, but it'll do for now.

Extra content! I found an old picture on my cellphone from when I first screwed around with the NS-K330. ''Twas the time that I bricked the board by flashing the wrong SnakeOS image (the one without bootloader, damn!). The only option was desoldering the flash chip and reflashing it with the proper firmware. Luckily, I had a Seeeduino lying around to reflash the 3.3v SPI chip. Instead of resoldering the flash chip in its original place, I did what you can see in the picture. I didn't want to risk a second tricky desolder operation, since the first already destroyed a trace...

To compile a module (e.g. fuse.ko) for a kernel of which you only have the headers (e.g. some linaro build for a Pandaboard), do the following:

make -C /lib/modules/3.1.1-8-linaro-lt-omap/build/ CONFIG_FUSE_FS=m SUBDIRS=$PWD/fs/fuse modules

The argument for -C compiles the module in the proper environment: the target kernel. The "CONFIG_FUSE_FS" enables building fuse as module (linaro default is n). What to build is restricted by SUBDIRS, in which $PWD is the location of the stock 3.1.1 source (similar enough), and the target "modules".

As noted in my previous post on this project, the drives can be triggered from a local shell or by knocking on port 5002. However, neither is easily done from the XMBC interface. The most ideal solution would be activating on demand: when mountd receives a request for a volume that is not mounted nor present, it fires up the PSU to start all drives. Eventually, the requested volume shows up, it gets mounted and all is well.

I decided to settle -for now- for an intermediate solution. When mountd receives a request for a volume with id "activate", it starts the drives. Since we know in advance there actually is no volume with this name, we don't have to worry about waiting for the device, etc. This would be necessary for true on-demand activation.

With this set-up, I can add a location to XBMC with path /path/to/rfs/mount/.mounts/activate with name "ChAssNAS activate". If the XMBC user tries to access this location, (s)he gets an error message but knows the drives are being started. After a few seconds, other locations become available, as the volumes are mounted on the ChAssNAS side.

While testing these modifications to mountd (060-activate.patch) I ran into some odd behaviour. Both remotefs and mountd are started at boot time, through the init mechanism of OpenWRT. Whenever remotefs requests a non-existing path on the autofs mount created by mountd, the latter leaps into action and does whatever a mountd does. However, it didn't: mountd did not receive any autofs packages, until either was restarted from shell. After spending almost an entire day trying to find out what's wrong, I called MacGyver. He suggested starting remotefs right after mountd, through an SSH session. The current version of /etc/init.d/mountd now contains the following line:

/usr/bin/dbclient -i /path/to/private.key localhost
"PATH=/bin:/sbin:/usr/bin:/usr/sbin /etc/init.d/rfsd start"

For shame. But it works now.

In this post, I would like to present my custom, cheap-ass NAS-like device. It is an embedded Linux device running OpenWRT with a bunch of (spare) hard disk attached to it through IDE/SATA-USB convertors. The device exposes the disk to my internal network through Samba and remotefs.

The base of the project is a NS-K330 I bought from DealExtreme already quite some time ago. It features a CNS2132/str8132/FA526 (it depends on who you ask) ARMv4 core with Ethernet and two USB 2.0 ports. I didn't like the idea of having the disk(s) running continuously, so I wanted to make something with an on-demand activation. The entire contraption is powered by an old ATX-PSU and the disks can be activated from across the network.

Hardware

In the first picture, on the right, you can see the custom-made male 24pin power supply connector: a bunch of nails fixed with hot glue and some wires attached to it. The NS-K330 board is connected to the +5V standby power pin, so if the PSU is connected to the grid, the device runs. The yellow wire is connected to PS_ON: an ATX power supply is activated by pulling that pin to ground. The NS-K330 board has 4 leds that can be controlled from software. In hardware, this is also done by connecting something to ground. By connecting the yellow wire to the the cathode pin of one of the leds, the PSU can be started by triggering that led. The hard drives are powered through the molex connectors of the PSU, which means they only run when the led is lit.

The PSU was recovered from an old Dell PC. The pin-out of the connector, I discovered, is different from a standard ATX power plug. Wtf, Dell? The board is attached to the back plate of an old PC case. I also recycled an old case fan (or perhaps it was a PSU fan) to create a bit of airflow. The extra heatsink on the CPU (a quarter of a Pentium heatsink) hopefully reduces the temperature a bit.

The third picture shows a test setup with one disk and a USB flash drive. I'm still waiting for the USB hubs I ordered from DealExtreme. The idea is that they will also be powered by the PSU (I just noticed there is no port for external power, damn! Ah well, some more DIY I guess), which means that the IDE-USB convertors would also be powered down when not in use.

Software

The default firmware was at first replaced with snake-os, a environment developed for devices based on this chip, but now it runs the cns21xx flavour of OpenWRT backfire. This port is still work in progress (although it works just fine) and has to be built from source. OpenWRT contains a package called mountd, that uses AutoFS to mount and umount attached volumes on demand. I did modify the code a bit: a device is now mounted on a directory named after its serial instead of the block name (050-chassnas.patch), and the serial is computed differently (040-serial.patch).

Activation

For now, the hard drives can be activated from on the device itself (it runs a Dropbear SSH server, included in OpenWRT) and by port-knocking on port 5002 (that's remotefs + 1). That last bit was surprisingly easy to set up: there is an iptables LED target to trigger leds on network events. Niceee.

Later, I will try to integrate the activation (and perhaps deactivation) into mountd. In a way, the patches above are already working in that direction.