tech notes

This is something new I’ve just learned that only exists on OpenBSD. Up until today I thought that the only way to manually failover a carp setup was to down the carp interface on the master.

It looks like there is an easier way of doing it on OpenBSD. In fact OpenBSD uses this feature itself during the boot process. Just before setting up all interfaces it “demotes” all carp interfaces so they won’t become master interfaces for their ip addresses until all enabled system daemons, pf, ipsec etc have been configured and started. After that the whole carp group of interfaces is put back to the neutral state and they can become master interfaces (if there is no advskew set on them).

How is it done?

OpenBSD has this concept of groups of interfaces. It’s easy to spot it when you do ifconfig:

# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0×3
vic0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0c:29:9c:5e:57
groups: egress
media: Ethernet autoselect
status: active
inet 172.21.33.5 netmask 0xffffff00 broadcast 172.21.33.255
inet6 fe80::20c:29ff:fe9c:5e57%vic0 prefixlen 64 scopeid 0×1
enc0: flags=0<> mtu 1536
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp

Each interface has its own default group (or groups). The default group for all carp interfaces is… the carp group! You can create your own groups and add interfaces to them. An interface can belong to multiple groups. Here’s how to create a new group and add carp0 to it:

# ifconfig carp0 group mygroup
# ifconfig carp0
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp mygroup

and here is how to remove it

# ifconfig carp0 -group mygroup
# ifconfig carp0
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp

All groups have this additional property called the demote count which is used by carp during the master election process. Using this property you can demote a group of interfaces:

# ifconfig -g carp carpdemote 128

and promote it back:

# ifconfig -g carp -carpdemote 128

and you can see the current value:

# ifconfig -g carp
carp: carp demote count 0

So how is this better than downing all your carp interfaces by doing something like this:

for i in `ls /etc/hostname.carp*`; do echo $i | awk -F. ‘{print $2}’ | xargs -I% ifconfig % down; done

When you down your carp interface they no longer take part in the whole “carp process”. Basically since they are down they no longer advertise their presence and cannot be elected as masters. So if your backup server dies and all carp interfaces on your master are down you loose your connectivity.

Carp demote counter acts in a bit similar way to advskew but has higher precendence over it. So a carp interface with advskew set to 0 and demote counter set to 10 will be ranked lower (and become slave) than another carp interface with advskew 100 and demote counter set to 0.

Plus, by logically groupping carp interfaces you can failover only one group at a time, and when you have a lot of interfaces this is certainly easier then using ifconfig down.

Flash clients don’t comply with the XMPP protocol. This is due to some limitation/security features in the flash itself. The problems are:

• flash sends null terminated stanzas and expects the same from the server
• opening stream header is a bit different than expected. In case of a flash client it’s:

  <flash:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='' from='' />

There are currently only two jabber servers I could find with support for this:

• openfire, which is obvious since they also provide a flash based jabber client with their jabber server solution
• ejabber with this patch applied. After patching ejabber needs to be recompiled with the –enable-flash-plugin option passed to the configure script.

Although FreeBSD runs under VMware  and there’s even an option for it in the GUI management tool, it doesn’t run perfectly.

timekeeping

The first problem with FreeBSD under VMware is time synchronisation. After running FreeBSD for a few minutes you’ll notice that its timer is completely out of sync comparing with the host’s timer. This is mainly due to the nature of virtualization provided by VMware connected with FreeBSD’s kernel tick value.

By default the kernel ticks every 1/1000 second which is too frequently for VMware based virtualized environment. The number of ticks per second can be adjusted either by adding HZ=100 to your kernel config file and recompiling it or by adding kern.hz=100 to /boot/loader.conf and rebooting your server.

100 is just a suggested value (it actually used to be the default value in older versions of FreeBSD) but something smaller than that, like 50 or 20, might work better for you.

on top of these changes you should also run ntpd!

network performance

Network performance can be tuned a bit by using a different ethernet card emulation in your VMware. So instead of using a driver for an AMD card, change it to the em driver for Intel cards. This requires editing the .vmx file describing your virtual server and adding:

You have to restart your virtual server for these changes to take place.

Imagine you have a FTP server behind your NAT/firewall server and you want to (or rather have to, because why would you really _want_ to do it?) give access to this FTP from outside world.

As FTP is a really crazy old protocol it’s not as easy as with HTTP or POP3 where you only have to redirect one port. With FTP there are actually two connections established, one on port 21 which is used to control the FTP session and is used to send commands, and the other connection is used to transfer data. The problem is with the other connection, which doesn’t usually have one set port that it would use, instead it’s port is negotiated everytime you want to download something. So what you need is a tool that will open and redirect ports needed for data session based on what’s being negotiated in the control session.

Here’s how to do it on OpenBSD using ftp-proxy and a few simple pf rules.

My internal network is 172.21.33.0/24 and my gateway’s IP on this network is 172.21.33.1. FTP server’s IP is 172.21.33.5. First, we have to set up ftp-proxy daemon in reverse-proxy mode. It’s option -R with FTP server’s IP, so add

ftpprooxy_flags="-R 172.21.33.5"

to /etc/rc.conf.local and starting the deamon manually (it will start automatically after the next reboot)

# /usr/sbin/ftp-proxy -R 172.21.33.5

Now all you have to do is to edit your /etc/pf.conf file and add

rdr-anchor "ftp-proxy/*"
rdr pass on $ext_if proto tcp from any to $ext_if port 21 -> 127.0.0.1 port 8021

before your redirection rules

nat-anchor "ftp-proxy/*"

before your nat rules

and finally, before your filtering rules:

anchor "ftp-proxy/*"

pf will use these anchors to dynamically create rules needed for new FTP sessions. That’s it, reload pf and test everything.

Update (2008-06-04): This patch also works with the latest (1.3.1) version of wsf/php.

WSO is a web services framework I’ve been recently playing with. There are versions of it for C, php, ruby, perl, and some other languages… The php version comes as a php module that needs to be compiled from source (or installed as a package if you’re running a linux distribution that has it) and added to your php configuration.

The source code compiles under windows, linux and OS X but needs some patching to work under FreeBSD.

you’ll need libxml2, libiconv, zlib and sqlite (or mysql) libraries installed from ports.

Here is how to do it:

# fetch http://dist.wso2.org/products/wsf/php/1.2.1/wso2-wsf-php-src-1.2.1.tar.gz
# fetch http://bsd.dischaos.com/files/wso2-1.2.1-freebsd.patch
# tar xvfz wso2-wsf-php-src-1.2.1.tar.gz
# patch -p0 < wso2-1.2.1-freebsd.patch
# setenv CPATH /usr/local/include
# setenv LD_LIBRARY_PATH /usr/local/lib
# cd wso2-wsf-php-src-1.2.1
# ./configure LDFLAGS="-lcompat" && make && make install

now just add wo.so extension to your /usr/local/etc/php/extensions. You’ll probably also need xml and xsl php extensions to have everything in wo working. Also, I copied the scripts directory from the wso2 package to /usr/local/share/wso2 and added this directory to include_path.

One of my servers has recently died. When I was installed the server around 7 years ago, qmail was a very trendy alternative to the “bad and insecure” sendmail, FreeBSD’s default MTA. Now, after a few years have passed, qmail is no longer so trendy and chicks don’t dig it as they used to… So while reinstalling the server I knew I had to choose something different than qmail. It’s not that I don’t like qmail anymore but after spending some time with postfix, especially after integrating it with things like clamav, dspam and other different extensions I think it’s just easier to manage. So the real reason was - i wanted all these nice toys I had with other postfix installations and I wanted them quickly and without all that hacking around and patching patches I would have to do with qmail.

Moving email accounts between qmail and postfix is not a hard task. They both support Maildir format so getting this part working was quick and painless. The only problem I had with moving this mail installation was what to do with my ezmlm based mailing list. So this is how to make qmail and postfix live happily under one roof.

Install all required ports

I installed these ports (my local versions in brackets, note that it’s a bit old installation now):

• mail/qmail (1.03_6)
• mail/qmail-contrib (0.1_1)
• mail/ezmlm-idx (0.40_4)
• mail/postfix (2.4.6,1)

Configure postfix

add qmail transport to master.cf:

qmail unix - n n - - pipe flags=R user=qmailq argv=/var/qmail/bin/sendmail ${recipient}

and in main.cf define transport configuration file with perl regex syntax:

transport_maps = pcre:/usr/local/etc/postfix/pcre_transport

now, add all your mailing lists to pcre_transport file so they get delivered via the qmail transport. An example for lists eztest@server.com and otherlist@server.com would look like this:

/eztest(-[a-z]+)?@server.com/ qmail:
/otherlist(-[a-z]+)?@server.com/ qmail:

Set up qmail

Just make a standard port installation but do not enable qmail as your default MTA, after all, you want it to be postfix, right?

After starting everything you can test it

create a new mailing list:

# ezmlm-make ~alias/eztest ~alias/.qmail-eztest eztest server.com
# chown -R alias ~alias/eztest

add a test user:

# ezmlm-sub ~alias/eztest user@example.com

And that’s it. Of course your list probably needs some changes in the configuration files (~alias/eztest/) but this is all well documented in ezmlm(5)

OpenBSD 4.3 has just been released. I’ve just tested installing it on a net 5501. The installation procedure is exactly the same as  for OpenBSD 4.2 described here. The only exception is that I couldn’t find pxeboot on the install43.iso file downloaded from ftp.openbsd.org (I’ll check if it’s on the original CDs when they arrive). It’s not a problem as one from a previous release works fine.

Currently I can’t really find any improvements in 4.3 that would be especially useful on net 5501 as it was the case with 4.2 and net 4501 (performance improvements to pf and many updates to sis(4)). However one nice thing is a new snmp daemon in the base system. This could we quite useful when tracking usage statistics on routers with, for example network weathermap

Why OpenBSD?

A bit of explanation first. I did some test installations of FreeBSD 6.x on an older Soekris net 4801 box and found out, to my surprise, that it wasn’t running as fast and stable as OpenBSD. Additionally, comparing to OpenBSD, installing FreeBSD over PXE on Soekris was a bit of a nightmare.

Instructions

Console connection

As Soekris boxes don’t come with VGA cards the only way to access and configure them is through a console cable. I’m using a standard console cable connected to my FreeBSD box. The only non standard thing is the default console speed which in Soekris is set up to 19200. So to get it working under FreeBSD you need to do:

# cu -s 19200 -l /dev/cuad0

If you reboot your Soekris now you should be able to see something like:

comBIOS ver. 1.33  20070103  Copyright (C) 2000-2007 Soekris Engineering.
net5501
0512 Mbyte Memory                        CPU Geode LX 500 Mhz
Pri Mas  CF CARD 2GB                     LBA Xlt 983-64-63  1982 Mbyte

Slot   Vend Dev  ClassRev Cmd  Stat CL LT HT  Base1    Base2   Int

-------------------------------------------------------------------
0:01:2 1022 2082 10100000 0006 0220 08 00 00 A0000000 00000000 10
0:06:0 1106 3053 02000096 0117 0210 08 40 00 0000E101 A0004000 11
0:07:0 1106 3053 02000096 0117 0210 08 40 00 0000E201 A0004100 05
0:08:0 1106 3053 02000096 0117 0210 08 40 00 0000E301 A0004200 09
0:09:0 1106 3053 02000096 0117 0210 08 40 00 0000E401 A0004300 12
0:20:0 1022 2090 06010003 0009 02A0 08 40 80 00006001 00006101
0:20:2 1022 209A 01018001 0005 02A0 08 00 00 00000000 00000000
0:21:0 1022 2094 0C031002 0006 0230 08 00 80 A0005000 00000000 15
0:21:1 1022 2095 0C032002 0006 0230 08 00 00 A0006000 00000000 15

1 Seconds to automatic boot.   Press Ctrl-P for entering Monitor.

Intel UNDI, PXE-2.0 (build 082)

Copyright (C) 1997,1998,1999  Intel Corporation

VIA Rhine III Management Adapter v2.43 (2005/12/15)

CLIENT MAC ADDR: 00 00 24 DE AD AA

And your system will start a PXE boot. At this point, copy your MAC address (in this case 00:00:24:DE:AD:AA) as you may need it in the next step.

DHCP

I’m using a ports based installation of isc-dhcp3-server running on FreeBSD.

To boot your soekris box over PXE you need to set up a DHCP server to send tftp server details. I’m doing it by creating a separate host configuration for my soekris.

host soekris {
hardware ethernet 00:00:24:DE:AD:AA;
option tftp-server-name "10.0.0.1";
option root-path "10.0.0.1:/tftpboot";
next-server 10.0.0.1;
filename "pxeboot";
option    routers 10.0.0.1;
}

What it does is it tells the Soekris box that it should use a tftp server at 10.0.0.1 and fetch and run the pxeboot file from it.

TFTP

I’m running a tftp server on my FreeBSD on 10.0.0.1. To set it up just comment out this line in your /etc/inetd.conf:

tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /tftpboot

Now you have to enable inetd in your /etc/rc.conf and start it (/etc/rc.d/inetd start)

At this point you have your tftp ready, now it’s time to prepare all the files needed to install OpenBSD. Mount your OpenBSD installation CD (or an ISO of it) and do the following on your TFTP server:

# cd /tftpboot
# mkdir 4.2 etc
# cp /mnt/cdrom/4.2/i386/bsd.rd /mnt/cdrom/4.2/i386/pxeboot .
# cp -Rv /mnt/cdrom/4.2/i386 4.2/

and finally create a /tftpboot/etc/boot.conf file with the following content:

set tty com0
stty com0 19200
set image /4.2/i386/bsd.rd

Installation

At this moment, after rebooting your Soekris it should fetch OpenBSD kernel image and ram disk and start the installation process. Follow it as you would do normally ( http://openbsd.org/faq/faq4.html ) with only these exceptions:

• do not create a swap partition if you are using a CF card, unless you really know what you’re doing.
• answer yes when asked “change the default console to com0″
• answer 19200 when asked “Which one should com0 use?” about the speed of your com0 console

after installing all required packages and finishing the whole process you should have a working OpenBSD installation on your Soekris.

There is only one last thing to be done. By default Soekris boots via PXE first and then uses its primary IDE drive (your CF card in this case) . You can either comment out the soekris section in your DHCP configuration or change the default boot order in Soekris BIOS. To do that reboot the Soekris, hit ctrl+p to enter Monitor and type:

set BootDrive=80 81 F0 FF

The goal is to have two IPsec hubs, one master and one slave, where the slave can take over when the master goes down and it can do it without losing all established IPsec tunnels.

With OpenBSD it’s really straight forward, as all features needed to do it are in the base system. They are: carp, pfsync and sasyncd.

We start with two hosts - master with IP 192.168.1.1 and slave with 192.168.1.2.

First we set up carp so both servers will share one IP address. This IP will be used as the end point for establishing all IPsec tunnels.

and

We also need pfsync, not only to synchronize pf states but we will also need it to synchronize SA replay counters. Ideally pf states should be synchronized over a dedicated interface and a crossover cable (in case of 2 failover boxes).

And now it’s time to set up sasyncd. In this simple configuration sasyncd only needs information about peer name, carp interface which state it will follow (acting as the master sasyncd if the carp interface is in MASTER state or as a slave when the carp interface is in BACKUP state) and a shared key which will be used to encrypt SA updates.

Edit /etc/sasyncd.conf , on master:

use openssl rand -hex 32 to generate your shared key.

and on slave:

Now on both servers enable sasyncd by adding sasyncd_flags=”" to your /etc/rc.conf.local files.

At this moment you can either reboot your servers to get everything started or do

Test it on your slave box, after a second you should be able to see exactly the same flows and SAD entries with ipsecctl -sa

I had some problems with my jabber server. It was just a plain jabberd+mysql installation from ports. For some reason it didn’t want to authenticate with SASL using digest-md5 mechanism. The only working option was plaintext without SASL, which wasn’t really perfect…

After searching in google it turned out to be a problem with GSASL library, which is the default SASL library used by jabberd.

It looks like jabberd also works with other SASL libraries, cyrus-sasl for example. Even that it’s marked as being experimental it seems to be working better than the default GSASL library.

Here’s how to install jabberd with cyrus-sasl:

1. Install cyrus-sasl (/usr/ports/security/cyrus-sasl2)

2. edit jabberd’s Makefile

 --- Makefile.orig       2008-02-19 22:11:37.000000000 +0000
+++ Makefile    2008-02-19 22:11:39.000000000 +0000
@@ -15,7 +15,6 @@
 COMMENT=       Online presence and instant messaging server

 LIB_DEPENDS=   expat.6:${PORTSDIR}/textproc/expat2 \
-               gsasl.11:${PORTSDIR}/security/gsasl \
                idn.16:${PORTSDIR}/dns/libidn

 OPTIONS=       MYSQL "Support MySQL (storage/auth/reg)" on \
@@ -37,7 +36,7 @@
 USE_LDCONFIG=  ${PREFIX}/lib/jabberd
 CONFIGURE_ARGS+=       --localstatedir=/var \
                --sysconfdir=${PREFIX}/etc/jabberd \
-               --enable-ssl --enable-mio=poll --enable-sasl=gsasl \
+               --enable-ssl --enable-mio=poll --enable-sasl=cyrus \
                --with-extra-include-path="${LOCALBASE}/include ${EIP}" \
                --with-extra-library-path="${LOCALBASE}/lib ${ELP}"

3. do make patch, and edit work/jabberd-2.1.21/sx/sasl_cyrus.c

--- sasl_cyrus.c.orig   2008-02-19 22:13:53.000000000 +0000
+++ sasl_cyrus.c        2008-02-19 22:14:05.000000000 +0000
@@ -20,7 +20,6 @@

 /* SASL authentication handler */

-#error Cyrus SASL implementation is not supported! It is included here only for the brave ones,\
 that do know what they are doing. You need to remove this line to compile it.

 #include "sx.h"
 #include "sasl.h"

4. continue with make install