tech notes

Web applications can be traced or debugged on many layers. The highest layer would be what some frameworks like symfony provide in their development consoles. Another layer would be to look at output from php-xdebug in kcachegrind to do profiling. But the lowest possible level is to look at actual system calls used by a running application.

FreeBSD has a base system utility called ktrace. It allows administrator to attach to a running process and log all system calls used by the process. How to use it to trace a web application running under apache?

First apache has to be started in debug mode, with only one worked running. This will make finding the apache process running our application easier to find.

This is how to start apache in debug mode:

httpd -X &
[1] 34702

Running this in the background will return the ID of the process: 34702. Now, all we have to do is to attach ktrace to this process:

ktrace -dip 34702

option “-d” means that all descendants (current child processes) of the process will also be traced and option “-i” means that all process spawned by our process will also be traced. “-p” option is used to give the PID of the traced process.

at this point we can run our application from the browser and after doing so take a look at system calls used. Ktrace saves all system calls to a ‘ktrace.out” file in the current directory. kdump utility is used to display contents of this file:

kdump -R

“-R” option will display time taken between entries so we can estimate how long a syscall took to finish.

we can detach ktrace and stop tracing the process by:

ktrace -C

nothing new or special but certainly worth joting down What’s done in MySQL with auto_increment statement in PosgteSQL is a bit more complicated. It requires creating a sequence first:

CREATE SEQUENCE exp_seq;

and then using it in table definition:

CREATE TABLE exp (id INT NOT NULL PRIMARY KEY DEFAULT nextval(‘exp_seq’), other varchar(30), created_at date);

Sequences are actually more powerfull than MySQL’s auto_increment as they have some interesting options. One of them is INCREMENT BY which can be very useful with some multiple master replication or multiple servers scenarios.

Operations allowed on sequences are: nextval, currval and setval.

This is a follow-up to the previous post. Instead of having your disks mounted read-write and have them broken by an unexpected reboot why not run them in read-only mode instead? Especially if your file systems are stored on a CompactFlash card and shouldn’t be written to too often anyway. If mounted read-only your partitions are never marked dirty and don’t even require fsck to be run during boot (so having fastboot enabled is fine in this case).

First of all, lets just assume that there is only one partition on the system (the root partition – /). It’s not impossible to change it’s mount options to ro and boot your system this way. OpenBSD will actually boot fine, but it will complain a bit.. First of all, it will complain about some parts of the /var subtree not being writeable. Syslog won’t certainly like not being able to write to /var/log/*, and other daemons might not like read-only /var/run and /tmp directories. Additionally, some daemons really need write access to some devices in /dev. So this is not ideal solution.

A slightly better one is to have your / partition mounted read-only and have /var /tmp and /dev mounted read-write from memory based file systems. A sample /etc/fstab doing this would look like this:

/dev/wd0a / ffs ro,noatime 1 1
swap /tmp mfs rw,noatime,nodev,nosuid,-s=20000 0 0
swap /var mfs rw,noatime,nodev,nosuid,-s=40000 0 0
swap /dev mfs rw,noatime,nosuid,noexec,-s=20000 0 0

This creates 10MB /tmp and /dev file systems and 20MB /var. The only problem now is that /var and /dev partitions are empty and we need to recreate their structure in order to make OpenBSD happy.

First, the /var parition needs all subdirectories created, this is easily done with mtree:

mtree -qdef /etc/mtree/4.4BSD.dist -p / -u

The /dev partition needs all devices created. This is usually done by running the MAKEDEV script located in /dev. Since we’re creating a new, blank /dev directory we need to make a copy of the MAKEDEV script:

cp /dev/MAKEDEV /root/

now, after mounting our memory based /dev we can do:

cd /dev; sh MAKEDEV all

and all devices should be created for us.

This all should be done during the boot process, this patch does that:

— rc.orig    2008-10-06 17:01:58.000000000 +0100
+++ rc    2008-10-06 17:03:18.000000000 +0100
@@ -212,6 +212,8 @@
mount -a -t nonfs,vnd
mount -uw /        # root on nfs requires this, others aren’t hurt
rm -f /fastboot        # XXX (root now writeable)
+mtree -qdef /etc/mtree/4.4BSD.dist -p / -u
+cp /root/MAKEDEV /dev; cd /dev; sh MAKEDEV all

random_seed

after applying the patch (don’t forget to copy the MAKEDEV file!!!) and rebooting we should have a working OpenBSD server which can survive random power loses without disk. What if you need to upgrade your system or change some config files? Easy, just remount your root file system:

mount -u -o rw /