tech notes

A simple munin plugin for graphing Ironport statistics. All it graphs is CPU, memory, disk IO and queue utilisation. It requires SNMP to be enabled on the Ironport (and assumes you’re using SNMP v3) and snmpwalk to be installed on the munin server. The plugin supports the following environment variables:

* snmp_community – by default set to v3get (default Ironport setting)
* snmp_password – you have to set it yourself
* snmp_oidbase – set to 1.3.6.1.4.1.15497.1.1.1 which points at Ironport’s MIBS.

Save it as ipstatus_ and link to it via ipstatus_yourironporthostname

Looks like there’s a hidden command you can use to fix queue problems similar to these:

Critical: Queue: Your queue has been corrupted; UNABLE TO REPAIR: unable to
mount queue: ‘(\’qstore/gcq.py get_time_sorted_gens|919\’, “ \'exceptions.OSError\'>“, “[Errno 2] No such file or directory:
\’/var/db/godspeed/gen/gen063.chk\’”, \’[qstore/gcq.py mount|1387]
[qstore/gcq.py load|996] [qstore/gcq.py get_time_sorted_gens|919]\’)’

Critical: Error while sending alert: Unable to send System/Critical alert to xxx@xxx.com with subject “Critical ironport: Queue: Your queue has been corrupted; UNABLE TO REPAIR: unab…”.

What it means is basically that the workqueue is corrupted and the ironport is unable to accept/deliver emails. Rebooting doesn’t help (doesn’t really change anything). There’s however a way of recovering your ironport from this problem. The hidden command is:

It deletes the broken queue, creates a new one, removes all messages in the system quarantines and reboots the ironport. After this clean up operation your ironport should be as new.

What’s interesting is that by looking at various error messages thrown by Ironports from time to time (especially when something breaks more seriously – doesn’t happen too often Ironports are quite solid) you can actually see what’s running under the hood. Other than it running on something derived from FreeBSD (can’t be that far off it as they actually contribute some code back to the OS) it looks like it’s mostly run by python scripts. That’s interesting from the performance perspective as even the queue management seems to be written in python. Also the database used internally seems to be some version of PostgreSQL. A very nice choice of software…

Haven’t seen anything yet that would suggest what MTA (if it’s not something created by Ironport) is used there. Sure the fact it can write qmail compatible log files doesn’t mean anything )

If you happen to use Nortel’s Alteon load balancers and need something to help you visualise your configuration here’s a script that produces graphs of connections and dependencies between groups, virtual IPs, real IPs and services in the Alteon configuration. It’s not very useful, the code is ugly, but the graphs are cool to look at Especially for huge config files with lots of services and VIPs. The script uses two great python libraries – pygraph and graphviz.

This simple script displays all prefixes advertised by an AS. It uses RIPE’s looking glass to get their local BGP table and parses it for the given AS number.

For example, to get all the other networks advertised by the AS dischaos.com is in, first I need to get Layered tech’s AS number:

whois -h www.ris.ripe.net 216.32.74.94

% This is RIPE NCC’s Routing Information Service
% whois gateway to collected BGP Routing Tables
% IPv4 or IPv6 address to origin prefix match
%
% For more information visit http://www.ripe.net/ris/riswhois.html

route: 216.0.0.0/8
origin: AS3303
descr: SWISSCOM Swisscom (Switzerland) Ltd
lastupd-frst: 2009-06-08 11:31Z 192.65.185.243@rrc04
lastupd-last: 2009-06-08 11:31Z 192.65.185.243@rrc04
seen-at: rrc04
num-rispeers: 1
source: RISWHOIS

route: 216.32.0.0/14
origin: AS3561
descr: SAVVIS – Savvis
lastupd-frst: 2009-06-08 12:56Z 198.32.160.22@rrc11
lastupd-last: 2009-07-22 12:06Z 195.69.145.49@rrc03
seen-at: rrc00,rrc01,rrc03,rrc04,rrc06,rrc07,rrc10,rrc11,rrc12,rrc13,rrc15,rrc16
num-rispeers: 86
source: RISWHOIS

route: 216.32.64.0/19
origin: AS22576
descr: LAYER3-ASN – Layered Technologies, Inc.
lastupd-frst: 2009-06-08 12:56Z 198.32.160.22@rrc11
lastupd-last: 2009-07-22 14:55Z 193.232.244.147@rrc13
seen-at: rrc00,rrc01,rrc03,rrc04,rrc06,rrc07,rrc10,rrc11,rrc12,rrc13,rrc15,rrc16
num-rispeers: 85
source: RISWHOIS

Their AS number is 22576. Now lets see what other networks they advertise:

$ ./getprefixes.py 22576
networks advertised by AS22576:
64.92.160.0/20
72.21.32.0/19
72.36.128.0/17
72.232.0.0/17
72.232.128.0/19
72.232.160.0/21
72.232.168.0/22
72.232.172.0/23
72.232.174.0/23
72.232.176.0/20
72.232.192.0/18
72.233.0.0/19
72.233.28.0/22
72.233.32.0/19
72.233.64.0/18
72.233.64.0/20
72.233.80.0/23
72.233.82.0/23
72.233.84.0/23
72.233.86.0/23
72.233.88.0/21
72.233.96.0/22
72.233.100.0/22
72.233.104.0/21
72.233.112.0/21
72.233.120.0/21
72.233.127.0/24
208.95.152.0/22
209.67.208.0/20
216.32.64.0/19

This helps tracking down bottlenecks in DNS response from recursive queries.

First you need a DNS query payload (contents of a valid udp DNS A type query).

and now use this payload with hping2 to send udp packets:

our cisco router is at 10.0.0.1 and our freebsd box is at 10.0.0.20.

first cisco configuration:

!adds router’s local time to messages

service timestamps log datetime localtime

!this works on ios 12.4, other versions might use different syntax

logging trap debugging

!our syslog server

logging 10.0.0.20

logging on

now on the freebsd box. first enable syslog to accept messages from external sources, in /etc/rc.conf:

syslogd_flags=”-a 10.0.0.1/32:*”

the “:*” at the end is quite important as it tells syslogd to accept all messages sent from 10.0.0.1 from any source port. Without it it only accepts messages sent from port 514 (syslog)

next create your log file: touch /var/log/router.log and add something similar to the top of your /etc/syslog.conf:

#enter your router’s host name here:

+10.0.0.1

#in fact local7.* should be enough here, as it’s cisco’s default facility

*.*  /var/log/router.log

#this resets the previous +host definition

+*

now restart syslogd:

# /etc/rc.d/syslogd restart

if you can’t see anything in /var/log/router.log (and it’s not because your router has nothing to report), start your syslog in the debugging mode:

# /etc/rc.d/syslogd stop

# syslogd -d -v -a ‘10.0.0.1/32:*’