Wednesday, May 25, 2005


I rebooted a bunch of our servers earlier today to pick up new kernels. We have our own build of Perl, Apache and other software like that mounted via NFS and on a couple of the boxes the NFS directory didn't get mounted when it booted.

Also on one of the cpansearch boxes squid didn't start up on boot, and it wasn't in our monitoring setup so I only noticed just now because a user wrote in! Doh. (It worked half the time, depending on which server you got).

Our colobus server also didn't start up properly (or rather logging didn't), so the server has been flaky through the afternoon.

Anyway, it should all be better now. If not, send us mail.

Saturday, May 14, 2005


There's a barebones website for pgeodns now.

Robert moved our fancy DNS server out to a public repository, so it's availble from subversion if you are interested.

eww-tee-fff eight

Today some complaints came in about all the Perl Monger mailing lists switching to UTF-8. (As part of a fix for the encoding issues our Chinese group was having.)

So... I switched everything back to us-ascii, and created a special English (UTF-8) variant to use where necessary.

Thursday, May 12, 2005

EU mirror live

I flipped the switch, so is now using the EU mirror for European users.

(see the mirror post from yesterday)

Thanks again to Digital Craftsmen. We're also running one of our "geodns" thingies on their server now.

Wednesday, May 11, 2005

Subversive Ponies!

Ponie has moved from CVS to Subversion. You can now find it here. This leaves only a few small projects running off of our CVS server. mirror

Over the last week Graham and I have been setting up the box provided by Digital Craftsmen; our first mirror not hosted on the infrastructure. We are planning a mirror in Japan and another US mirror too (but haven't gotten the hardware for it yet).

It's not on the name yet, but (Very Alpha-ish) at If you are in Europe you should reach the new mirror, if you are anywhere else you should reach one of the hosted boxes (with the "hardware by bizrate" logo). The name will go away when we are done testing.

Please try it out and let me know if you are having trouble with it (email

Our geography-aware DNS server, pgeodns, is a bit naively assuming that all of Europe will have better access to the UK mirror than to the West Coast US mirror.

In the process of setting it up, I updated pgeodns a bit, so if I made a mistake or two it might impact the and names too, so please let me know if you are seeing any abnormalities with the DNS for those too.

pgeodns is in a private svn repository, but Robert says he'll move it to another one for me any day now. :-)

Tuesday, May 10, 2005

More disk!

More information and details later, but I wanted to share our new terabyte RAID with you:

[root@x4 root]# df -h /mnt/raid
Filesystem Size Used Avail Use% Mounted on
/dev/md0 1.4T 33M 1.3T 1% /mnt/raid

Monday, May 9, 2005

Data center day / new hardware

Tuesday afternoon is going to be datacenter day™ for Robert and I.

Our colocation facility asked us to get off the building UPS as the building apparently is running out of UPS power.  Since they are incredibly nice to us (they just gave us the neighbor rack so we have two racks there now!) we could hardly say no.  Last week we figured out what we need (I hope!) and ordered a APC SUA3000RM2U UPS that should have enough capacity.  The building has a generator that'll kick in very quickly so we only need a minimal run-time.  If it doesn't have enough capacity we'll get another one of those most likely.   The higher capacity ones gets a lot more expensive per watt as it quicky turns into some 208V+transformer mess.

Monday morning I heard the doorbell/barking dog duo and Vani came upstairs telling me there was a DHL guy with a heavy package that he was going to try getting down to the house.  I rushed and caught him going down the stairs so he could help me bring it up again and put it in the car.  The darn thing is 120lb!

Later in the day UPS showed up with a Dell 5324 gigabit switch (I mentioned that before) and Fedex showed up with more drives for the file/NFS/database/backup server (more on that later).

Tomorrow I have to go by the post office and pick up a 48-port 100Mbit switch I found on eBay and then we'll be off to the datacenter to setup the new rack.

Many or most of our services may be down briefly Tuesday afternoon (local time in California) as we shut down equipment to move it over to the UPS and install the new disks.

Sunday, May 8, 2005

Challenge Response for NNTP

A few weeks ago we enabled posting to mailing lists via Google Groups. In looking to expand that to the rest of the NNTP enabled world, we've settled on implementing a Challenge/Response system for verifying email addresses. (It really isn't fun when you reply on a mailing list and find out the email address you're replying to is fake.) Because we generally require posting from the subscribed address to the mailing lists, we know those are valid addresses.

So... Last night I attempted to implement TMDA. It is very easy to setup... but... I hit a roadblock. The envelope from/return path on NNTP gated postings is generally the gateway, not the original poster. So, TMDA had to be modified to use the From address instead. I had it mostly working after a few hours, alternately tweaking the scripts (something I didn't want to do -- makes it hard to upgrade) and writing procmail wrappers. Finally, I gave up, after TMDA made it clear that it wasn't happy if I sent a message from address A, and then sent the verification message from address B.

Next step is either to:

a) write my own Challenge/Response system just for this.

b) find another system that works

c) give up, letting the 3 or 4 people a wek who post via a NNTP server that is not send their emails directly to the mailing list.

Saturday, May 7, 2005


I upgraded mailman tonight to better support Chinese, as our new PM group was having issues with their mailing list archives.

While I was working on mailman, I also fixed some issues with periodic processes not being run. So, now you'll get reminders for your mailing list passwords, if you haven't turned it off in your account.