tech notes

This is something new I’ve just learned that only exists on OpenBSD. Up until today I thought that the only way to manually failover a carp setup was to down the carp interface on the master.

It looks like there is an easier way of doing it on OpenBSD. In fact OpenBSD uses this feature itself during the boot process. Just before setting up all interfaces it “demotes” all carp interfaces so they won’t become master interfaces for their ip addresses until all enabled system daemons, pf, ipsec etc have been configured and started. After that the whole carp group of interfaces is put back to the neutral state and they can become master interfaces (if there is no advskew set on them).

How is it done?

OpenBSD has this concept of groups of interfaces. It’s easy to spot it when you do ifconfig:

# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0×3
vic0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0c:29:9c:5e:57
groups: egress
media: Ethernet autoselect
status: active
inet 172.21.33.5 netmask 0xffffff00 broadcast 172.21.33.255
inet6 fe80::20c:29ff:fe9c:5e57%vic0 prefixlen 64 scopeid 0×1
enc0: flags=0<> mtu 1536
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp

Each interface has its own default group (or groups). The default group for all carp interfaces is… the carp group! You can create your own groups and add interfaces to them. An interface can belong to multiple groups. Here’s how to create a new group and add carp0 to it:

# ifconfig carp0 group mygroup
# ifconfig carp0
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp mygroup

and here is how to remove it

# ifconfig carp0 -group mygroup
# ifconfig carp0
carp0: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:00:00:00:00
groups: carp

All groups have this additional property called the demote count which is used by carp during the master election process. Using this property you can demote a group of interfaces:

# ifconfig -g carp carpdemote 128

and promote it back:

# ifconfig -g carp -carpdemote 128

and you can see the current value:

# ifconfig -g carp
carp: carp demote count 0

So how is this better than downing all your carp interfaces by doing something like this:

for i in `ls /etc/hostname.carp*`; do echo $i | awk -F. ‘{print $2}’ | xargs -I% ifconfig % down; done

When you down your carp interface they no longer take part in the whole “carp process”. Basically since they are down they no longer advertise their presence and cannot be elected as masters. So if your backup server dies and all carp interfaces on your master are down you loose your connectivity.

Carp demote counter acts in a bit similar way to advskew but has higher precendence over it. So a carp interface with advskew set to 0 and demote counter set to 10 will be ranked lower (and become slave) than another carp interface with advskew 100 and demote counter set to 0.

Plus, by logically groupping carp interfaces you can failover only one group at a time, and when you have a lot of interfaces this is certainly easier then using ifconfig down.

Flash clients don’t comply with the XMPP protocol. This is due to some limitation/security features in the flash itself. The problems are:

• flash sends null terminated stanzas and expects the same from the server
• opening stream header is a bit different than expected. In case of a flash client it’s:

  <flash:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='' from='' />

There are currently only two jabber servers I could find with support for this:

• openfire, which is obvious since they also provide a flash based jabber client with their jabber server solution
• ejabber with this patch applied. After patching ejabber needs to be recompiled with the –enable-flash-plugin option passed to the configure script.

Although FreeBSD runs under VMware  and there’s even an option for it in the GUI management tool, it doesn’t run perfectly.

timekeeping

The first problem with FreeBSD under VMware is time synchronisation. After running FreeBSD for a few minutes you’ll notice that its timer is completely out of sync comparing with the host’s timer. This is mainly due to the nature of virtualization provided by VMware connected with FreeBSD’s kernel tick value.

By default the kernel ticks every 1/1000 second which is too frequently for VMware based virtualized environment. The number of ticks per second can be adjusted either by adding HZ=100 to your kernel config file and recompiling it or by adding kern.hz=100 to /boot/loader.conf and rebooting your server.

100 is just a suggested value (it actually used to be the default value in older versions of FreeBSD) but something smaller than that, like 50 or 20, might work better for you.

on top of these changes you should also run ntpd!

network performance

Network performance can be tuned a bit by using a different ethernet card emulation in your VMware. So instead of using a driver for an AMD card, change it to the em driver for Intel cards. This requires editing the .vmx file describing your virtual server and adding:

You have to restart your virtual server for these changes to take place.

Imagine you have a FTP server behind your NAT/firewall server and you want to (or rather have to, because why would you really _want_ to do it?) give access to this FTP from outside world.

As FTP is a really crazy old protocol it’s not as easy as with HTTP or POP3 where you only have to redirect one port. With FTP there are actually two connections established, one on port 21 which is used to control the FTP session and is used to send commands, and the other connection is used to transfer data. The problem is with the other connection, which doesn’t usually have one set port that it would use, instead it’s port is negotiated everytime you want to download something. So what you need is a tool that will open and redirect ports needed for data session based on what’s being negotiated in the control session.

Here’s how to do it on OpenBSD using ftp-proxy and a few simple pf rules.

My internal network is 172.21.33.0/24 and my gateway’s IP on this network is 172.21.33.1. FTP server’s IP is 172.21.33.5. First, we have to set up ftp-proxy daemon in reverse-proxy mode. It’s option -R with FTP server’s IP, so add

ftpprooxy_flags="-R 172.21.33.5"

to /etc/rc.conf.local and starting the deamon manually (it will start automatically after the next reboot)

# /usr/sbin/ftp-proxy -R 172.21.33.5

Now all you have to do is to edit your /etc/pf.conf file and add

rdr-anchor "ftp-proxy/*"
rdr pass on $ext_if proto tcp from any to $ext_if port 21 -> 127.0.0.1 port 8021

before your redirection rules

nat-anchor "ftp-proxy/*"

before your nat rules

and finally, before your filtering rules:

anchor "ftp-proxy/*"

pf will use these anchors to dynamically create rules needed for new FTP sessions. That’s it, reload pf and test everything.