Defcon:Blog Keklkakl blog blag

The task here is: at work, we have these cute little Vesa-mount-sized computers originally manufactured by DMP Electronics as the eBOX 4310, rebranded as NorhTec MicroClient Sr, that we are going to use backpack-mounted on large-screen HDTV's for our internal digital signage project.

These little boxes are to have no spinning disk, and an as reliable as possible system. To get no moving parts, the boxes skip using a hard drive, and we are setting them up with Compact Flash (CF) as the main storage. But, as any documentation will tell you, CF has a limited number of write-cycles, and as a result of that, it is desirable to have the system running with its filesystem read-only once booted.

Internally we normally standardize on CentOS for servers, and Ubuntu+Fedora as supported desktop Linux'es. But none of these are really “dead simple” to make read-only-rootfs, and to be honest comes with too much bloat in my opinion. So I am trying to do this using Debian Lenny. Debian is stock, standard, known tech, easily modified (ref. Pebble, LEAF, DSL, Ubuntu, Mint), and supportable. The base install is also fairly easy to make small.

I hooked up one of the eBOX'es, PXE-booted the Debian Lenny installer, and installed onto the CF.

My magic during install was close to none:

• Partitioning: Manual, one partition, no swap
• Tasksel Install: Base system only.
• Non-root user: kitteh

Post install:

apt-get -y install less bzip2 screen vim-nox openssh-server
halt

After installing, I removed the card from the eBOX, put it in a CF-reader on my workstation, created an image of it, and made a backup.

sudo cat /dev/sdb > sdb_dump1     # Raw disk-dump of the CF card..
cp sdb_dump1 lenny_testimg1.img   # This copy is what I will be working on,
sudo bzip2 sdb_dump1              # while this file is my backup.

So, should I ever need to restore the original Lenny install, all I'll have to do is bunzip the sdb_dump1.bz2 file back onto a CF card of the same size (or larger).

After waiting for a few ages (dumping 8GB, copying 8GB and fibnally bzipping 8GB down to 586MB takes a SOLID time), I mounted up the image using loopback mounting, and continued customizing the install.

As the CF dump is a dump of the complete CF card, and Debian Installer has treated it as a harddisk, it contains a partition table. So a simple “mount -o loop imagefile” will not work.

First, get a hold of where the partition to mount starts:

fdisk -lu lenny_testimg1.img

You must set cylinders.
You can do this from the extra functions menu.

Disk lenny_testimg1.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000bdb20

             Device Boot      Start         End      Blocks   Id  System
lenny_testimg1.img1              63    15615179     7807558+  83  Linux

This image opnly contains one partition, and it starts at sector number 63. Each sector is 512 bytes, so the mount command becomes:

sudo mount -o loop,offset=$((63*512))  lenny_testimg1.img /mnt/

This mounts the file system that starts at sector 63 onto /mnt. If you get the mount command wrong, it will tell you:

mount: you must specify the filesystem type

If you get this, adjust your command and try again …. You correct info is given in the “fdisk -lu” output.

Jump into the file-system from the host computer:

sudo mount -t proc proc /mnt/proc/
sudo mount -o bind /dev/ /mnt/dev/
sudo LC_ALL=C chroot /mnt /bin/bash

Remove /etc/mtab and symlink it to /proc/mounts. This special file also describes the mounted filesystems, and can replace mtab without needing a write access on it.

rm /etc/mtab
ln -s /proc/mounts /etc/mtab

Setting up /etc/fstab so that all locations where we will write data is a tempfs, in other words a memory filesystem. If the directories that are set up as tempfs here need to contain data after boot, those data will either be copied across by /etc/rcS.d/S36rosystem-fix or /etc/rc.local

# <file system> <mount point>    <type>  <options>          <dump>  <pass>
/dev/hdc1       /                ext3    defaults,noatime   0       0
proc            /proc            proc    defaults 0 0
tmpfs           /var/run         tmpfs   defaults 0 0
tmpfs           /var/lock        tmpfs   defaults 0 0
tmpfs           /var/local/data  tmpfs   defaults 0 0
tmpfs           /var/log         tmpfs   defaults 0 0
tmpfs           /tmp             tmpfs   defaults 0 0
tmpfs           /var/lib/urandom tmpfs   defaults 0 0
tmpfs           /var/lib/dhcp3   tmpfs   defaults 0 0
tmpfs           /home            tmpfs   defaults 0 0

Now, we need resolv.conf to be writable for dhclient, so it needs to be located somewhere outside of /etc

mkdir /var/local/data   
mv resolv.conf /var/local/data/
ln -s /var/local/data/resolv.conf /etc/resolv.conf

Now, we need to have the dhclient-script (runs while/after dhclient is run) update the writable resolv.conf, instead of trying to replace our link on the read-only part of the system. This is sed-magic, to really see what this does, make a copy of /sbin/dhclient-script, and diff it afterwards…

sed \
    -e '/local new_/s/=.*/=\/tmp\/resolv.conf.dhclient-new/' \
    -e '/mv -f/s/mv -f.*/cat $new_resolv_conf > \/etc\/resolv.conf/' \
    -i /sbin/dhclient-script

Just verify that network settings are appropriate for our/your/the network. I am using DHCP to configure, so i end up with:

cat /etc/network/interfaces

auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp

ifupdown insists on writing a status file to /etc/network/run, so that location needs to be redirected to a tmpfs location. Note that the directory /var/local/data/network_run is created on boot by /etc/rcS.d/S36rosystem-fix, which will be created later on..

rm /etc/network/run
mkdir /var/local/data/network_run
ln -s /var/local/data/network_run /etc/network/run
rmdir /var/local/data/network_run

Debian uses udev and hotplug since, well, a long time now. It is set up to assign each ethernet adaper a separate device name, based on the MAC-address and/or the kernel-driver used. We do not want to save any persistent ethernet device data, and we want to use eth0 as the networking device. So, we have to remove the udev-definitions of “persistent devicenames”. The definition for persistent net device names are located in /etc/udev/rules.d/70-persistent-net.rules. Edit this file by hand, or use the following sed line:

 sed -e '/^SUBSYSTEM=="net".*eth/D' -e '/^$/D' \
     -e '/^# PCI device/D' \
     -e '/^$/D' \
     -i /etc/udev/rules.d/70-persistent-net.rules

We install syslogd for logging, and remove all references to xconsole from the configuration. At a later stage, it will be configured to send its log data to a central logging server, and not buffer any logging locally.

apt-get install syslogd
sed -e '/xconsole/s/^/#/' -i /etc/syslog.conf

In the fstab listed above, the root filesystem is mounted with “defaults,noatime”. That normally implies that the file system is mounted read-write. But we want read-only. This is fixed by changing the default “rootmode” in /etc/init.d/checkroot.sh from “rw” to “ro”. It _could_ also be done by setting “ro” on the kernel boot lines in /boot/grub/menu.lst, but previous experience shows that this may sometimes not be honored. So, forcing it is perhaps not the best way, but still the most “stable” way((Most stable as long as we do not upgrade to a new release of Debian, that is…)).

sed -e 's/rootmode=rw/rootmode=ro/' -i /etc/init.d/checkroot.sh

I have referenced the file /etc/rcS.d/S36rosystem-fix a few times now. This file is a init-script-hack that sets up a few locations and files that are needed right after the mount of local filesystems. I'm not listing the entire file here, to examine the content, and what it does, check out the current version at http://dilbert.hig.no/jonl/rosystem-fix.txt (((FIX ME))) oops- broken link...

The file is placed in /etc/init.d as normal, set executable, and symlinked into /etc/rcS.d. The number is chosen so that it is run right after mountlocal-bootmisc.sh.

wget http://dilbert.hig.no/jonl/rosystem-fix.txt -O /etc/init.d/rosystem-fix
chmod 755 /etc/init.d/rosystem-fix
ln -s /etc/init.d/rosystem-fix /etc/rcS.d/S36rosystem-fix

The final changes done to the prototype image, is to set up a pair of handy aliases to remount the root filesystem in read-write and read-only setups, respectively.

echo "alias ro='mount -o remount,ro /'" >> /root/.bashrc
echo "alias rw='mount -o remount,rw /'" >> /root/.bashrc

Transferring the image back is simply a task of unmounting the loopmount, and dumping the resulting image back onto the CF card.

Unmounting first…

exit
cd ~
sudo umount -l /mnt/proc
sudo umount -l /mnt/dev/
sudo umount /mnt/

And dumping the image back. Yes, I am using 'cat'. I should have used 'dd'. But both tools do the same job, so, why bother with complexity when you can have simplicity…

sudo cat lenny_testimg1.img > /dev/sdb

After tanking up the CF image, the card is moved back to the eBOX, and the eBOX is booted up. While the system is booting, check closely for stuff that fails, and verify that all essential systems start up OK.

In the first iteration, I had a whole lot of errors. I went though them and corrected all the critical ones, and updated this document to reflect the corrections. The corrections that I was not able to fix by cleanly modifying existing startup scripts, or doing bind-mounts or tempfs mounts, are fixed in the rosystem-fix startup script. So, if you follow this walkthrough, and have errors, the likely places to put your fixes will be in /etc/fstab and /etc/init.d/rosystem-fix.

After booting up, the next task is to make this a kind of Kiosk; i.e. starting a web browser in full screen…

I decided to try to keep the X Server installation small, but not super-small 😉 By using apt, a bunch of dependencies that are not really needed are pulled in, but at least it is not as bloated as it would be, had I pulled in a complete desktop meta-package.

First: set the filesystem writable….

rw

apt-get install pci-utils dbus dbus-x11 defoma x11-apps x11-session-utils x11-utils  \
                x11-xserver-utils xfonts-base  xserver-xorg-input-kbd \
                xserver-xorg-input-mouse xserver-xorg-video-dummy xserver-xorg-video-openchrome \
                xserver-xorg-video-vesa xserver-xorg-video-vga xinit cpp cpp-4.3 xserver-xorg \
                xserver-xorg-core xfs xfonts-100dpi xfonts-75dpi xfonts-scalable ttf-dejavu \
                ttf-freefont gtk2-engines xterm blackbox iceweasel unclutter

We need some additional fonts for our setup. Note that we add the “contrib” package of Microsoft Core fonts. No religious views here, we allow non-free stuff in… Klavika is a font that is used extensively in the GUC visual profile.

echo "deb http://ftp.no.debian.org/debian/ lenny contrib" >> /etc/apt/sources.list
echo "deb http://ftp.no.debian.org/debian/ lenny non-free" >> /etc/apt/sources.list
apt-get update
apt-get install msttcorefonts
cd /tmp
wget http://dilbert.hig.no/jonl/klavika.tar.bz2
cd /usr/share/fonts
tar jxvf /tmp/klavika.tar.bz2
fc-cache -f -v
fc-list | grep Kl # to confirm the install

The Xorg configuration will need to be tweaked to the display the Kiosk is to run with.

Finally, the filesystem is set read-only again.

ro

With X up and running, and the filesystem back in read-only mode, make sure the non-root user crated during install is available for use, by setting up the home directory:

mkdir /home/kitteh
chown kitteh.kittteh /home/kitteh

Now, either switch to a different VT, or log out root. Log in as the non-root user created during install, create a simple ~/.xinitrc:

# File: /home/kitteh/.xinitrc
xterm &
blackbox

Start up X:

startx

Now, start up iceweasel (yeah, firefox is not called firefox, but iceweasel on Debian, due to licensing issues). Install the Autohide extension from http://www.krickelkrackel.de/autohide/, and configure it for the local user. By enabling this extension, and exiting iceweasel while it is displayed fullscreen, iceweasel will “always” start in full-screen mode.

After setting iceweasel up, update the .xinitrc:

# File: /home/kitteh/.xinitrc
xsetroot -solid white &
xset -dpms s off &
unclutter &
xterm -iconic &
iceweasel http://the.url.of.the.kiosk.example.com &
blackbox

After this, exit X11, test that it works OK, and exit the non-root login completely.

Now, we need to retain the settings made from boot to boot. /home is mounted as a tmpfs, so we need to clone the kitteh home directory somewhere else on the file system, and push it back in place on boot. Log in as root..

rw
cp -pRv /home/kitteh /usr/local/kitteh

Add the following to /etc/rc.local to clone the saved homedir content on boot:

cp -R /usr/local/kitteh /home/kitteh
chown -R kitteh.kitteh /home/kitteh

I have opted to force X to start on boot by running it from /etc/rc.local as well, by adding this line just above “exit 0”:

su -c "startx" kitteh

Now, X will normally not start unless the user starting it is also controlling the console. As I am starting X from an rc-script, that is not the case. So, to allow rc.local starting X as the user “kitteh”, put the following content in /etc/X11/Xwrapper.config

# Keeping the old setting as a comment, for clarity...
# allowed_users=console
#
# This is not really seen as "safe", but in our case, it is needed..
allowed_users=anybody
#
# Niceness value? Nah....
nice_value=0

That should really be enough for starters. Close everything up, set the file system as read only again, and reboot.

ro
reboot