This is a time-lapse video of the Don Roberts ice rink going from the final stages of the 2008 Nobel Conference to the newly repaired ice rink. Earlier this year the cooling system failed and had to be replaced. In this video you can see the removal of the old cooling system followed by the installation of the new cooling system. If you ever wanted to see what is down there below the ice, here is your chance.

The video was created by taking a photo every minute from October 9th through November 30th and stitching them together. To try and shorten the video I removed most of the images where not much was happening such as each night and some of the weekends.

I tried to diagnose the bottleneck and it appears as though the bayes_token table ( now over 3GB ) is frequently locked causing other access to it to stall for a bit. So, my plan was to move the table from MyISAM to InnoDB and hope that row level locking would help. Well it turns out that even after days of trying, I just could not get our current database altered to be InnoDB in a reasonable amount of time. As a result I created new tables on a different database server and pointed spamd at the now empty but InnoDB based tables. For now performance is great, hopefully InnoDB will scale much better than MyISAM did otherwise I will be doing this again in a few months.

If anyone else is using MySQL to house their SpamAssassin database I would like to hear how you do it. Perhaps there is something simple that I missed.

In the past when using fc4 the database would sometimes get corrupted and I would have to run db_recover -h /var/lib/ldap.  Well after installing the db4-utils rpm and trying to run db_recover -h /var/lib/ldap I would get this error:

db_recover: Program version 4.3 doesn’t match environment version
db_recover: Unacceptable log file /var/lib/ldap/log.0000000043: unsupported log version 11
db_recover: Invalid log file: log.0000000043: Invalid argument
db_recover: PANIC: Invalid argument
db_recover: PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery

After a little hunting I discovered that openldap now comes with its own db_recover program called slapd_db_recover.

In order to keep the db log files in check, I would run  db_archive -h /var/lib/ldap -d daily, now I run slapd_db_recover -h /var/lib/ldap -d daily.
I also ran into some trouble while trying to import our data.  I used slapcat on our FC4 box to generate an ldif file and tried to use slapadd on the fc5 box to import it.  Then I got an error that said  “Lock table is out of available locks”.  To fix that problem I copied the DB_CONFIG.example that came with openldap in /etc/openldap to /var/lib/ldap and renamed it just DB_CONFIG.  Then I added these lines to the file:

set_lk_max_locks 8000
set_lk_max_lockers 2000
set_lk_max_objects 2000

And that fixed all my importing errors. Then I needed to change the ownership of all the files in /var/lib/ldap so they were owned by ldap and fire it up.

For some reason it tries CRAM-MD5 first, that fails, then it does not try anything else. To solve that problem I edited /usr/lib/sasl2/smtpd.conf. I added “mech_list: login plain” to the end of that file. Now if I look at the AUTH line that postfix sends to clients, I see this:


250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN


Now all of our email clients can happily authenticate and send email again.

In the short term, here are a couple of things we have done to help reduce the load. First, we have cut back the maximum size of a person’s inbox. We have a script that runs each night and if your inbox is more than 50MB, we move the oldest messages out of the inbox until it is down to about 30MB.

Second, we have installed a newer version of imapproxy on our webmail servers. This new version supports an IMAP SELECT cache which should help when reloading the inbox display because it will have the message headers already cached. Another nice feature of this version is that it now supprts encrypted IMAP connections by using TLS. So now your messages and password are being encrypted while they are being transmitted from the mailserver to the webmail server.

option WPAD code 252 = string;
option WPAD “http://www.gac.edu/cgi-bin/proxies”;

In our old config we had option option-252 “http://www.gac.edu/cgi-bin/proxies”; but that just gave us an error about option-252 unknown.

To make Firefox’s auto config work, we had to make sure we had wpad.domain defined for all of our subdomains such as wpad.it.gac.edu, wpad.res-hall.gac.edu, etc. all pointing to our real, wpad.gac.edu.

Hopefully this will help our bandwith usage a bit.