Friday, December 28, 2007

Upgrades mostly done

We finished the upgrades around 8pm (PST). It was surprisingly easy to upgrade from RHEL3 to RHEL5. The only thing we've run into so far were some of our custom built things we had compiled with the system OpenSSL that needed recompiling.

Most services are up again (and have been for some hours). We're upgrading our monitoring and system management tools and we're still tweaking and configuring, so do let us know if you see anything broken.

Thursday, December 27, 2007

"Annual" Maintenance - Dec 28th, 2007

Friday, December 8th, 2007 will be our "annual" datacenter day.  This means that Ask and I will go down to the colocation facility, swap drives, rewire things, upgrade BIOSes, upgrade operating systems, and reaffirm our frustration with hardware. (Our Cyclades Console Server means we hardly ever have to physically visit the machines.)

Starting at around 12pm PST (20:00 UTC) some services may be unavailable.  Email to,, and other hosted domains may be delayed.  Some websites including PerlBug, CPAN Ratings,,,,, may be unavailable.  We have a plan to minimize downtime for each service.

We expect the maintenance period to last less than six hours.  Otherwise, we'll be ordering some pizza.   

One of the biggest projects on our list for tomorrow is an upgrade to Redhat Enterprise Linux 5.  Many of our boxes are currently running RHEL3, which is getting a little long in the tooth.  We're looking forward to the newer kernel and more modern utilities.

We'll post here if there are any outstanding issues. 

Wednesday, December 26, 2007

Smething broke (at least it's the day after a holiday)

Our main internal fileserver has gone funky.  Due to the joy of NFS, this is making other servers unhappy too.    We suspect bad ram, but don't know yet.  The machine is currently rebooting and fscking.  We'll update this post as we learn more.

Update 8:54pm: The machine is back up, nothing appears to be horribly wrong, although we lost one disk from a RAID set.  We're moving some data around, but things should be returning to normal shortly.

Update 10:16pm: Everything seems to be fine now.  I expect those to be famous last words.

Tuesday, December 25, 2007

Monkey Spam

As our holiday gift to you, we provide you with some amusing monkey spam that made it through our spam filters at  This one was sent to a CPAN related email address, from a hosting provider's machine in Texas.  The phone number has an Ohio area code, but is missing the last digit, so even if you were interested in a monkey, you'd need to write back to pastor patrick's Yahoo Mail account. 

At we do our best to keep spam out of your mailboxes and off of our mailing lists.  It's not easy, as the only thing worse than letting too much spam through -- is blocking legit email.  Over the past month, we've made some changes to our mail system to ensure that mail continues to flow fast and furious... and we have some changes planned for the near future to continue to improve our systems.

And now... the monkey spam:


Complement of the season,
MY NAME IS joe helderberg, I need a responsible someone to adopt my  baby capuchin , My baby is still available for  christmas adoption and his 7weeks old now,weight 7lbs,Akc regis and am asking for $800.presently am in our church camp with my pets in Ohio.just tell me if you are still interest in my baby monkey so that i can tell pastor patrick about the baby shipment.
I will be waiting to hear from you if you are still interest asap.
God bless
more info.(614)xxx-xxx

Tuesday, November 27, 2007

From the Mailbag: 1 Unit Edition

I love when we get weird emails at  Here's one that came in recently...

22.11.2007 15:51

        Thema:  Enquiry  Perl

Hello Joanna,

we got an inquiry from our customer about the following products:

1 Unit of Perl

Please send a formal quotation with specify terms and conditions for
delivery as well as availability, anticipated delivery time and delievery
charges/proccesing fees and dealer discounts.
We also need Information on which operating system the software is
running. Also we need informations on which language
do you´ll deliever the software. Please also let us know on which form
you´ll deliever the software: Boxproduct or electronicaly delievery.

ATTENTION: We need your fax-number, because we don´t order online (
Internet security problems)!!

Please send your quote to my attention at

For more informations about our company please have a look on our homepage
( here you´ll find
an english company profil.

Thank you in advance for quick response.

best regards

Purchase department

Sunday, October 14, 2007

Goblins come early this year

Apparently nasty car crashes isn't the only news this weekend.  We seem to have lost a drive in one of the machines serving  Of course, it's the drive that everything else depends on. and email and some websites may be non functional while we're repairing it.  Check back here for more updates.

Update 6:40pm: Looks like things recovered with some remote percussive maintenance combined with patience.  (Have we shared how much we love our Cyclades console servers and power management recently?)

Monday, October 8, 2007 down (not hosted here)

We got a few mails asking about being down. It's not actually hosted here (anymore). is hosting it and they were informed about it earlier today (~5-6 hours ago I think). I'm sure they'll get it back as quickly as possible, but emailing us doesn't help. :-)

Our CPAN mirror is at

Wednesday, September 26, 2007 password/account management

For a long time we had a confusing thing where to get a subversion (and webdav) account you had to go to and set it up there.

We've improved it a bit now so resetting your password (or just setting up the account initially) you just go to Whee!

Let me know ( if you have any trouble with the new site.

Tuesday, September 18, 2007

Week of failures!

Geez! This week it seems like everything is falling apart over here. Servers that have been running for hundreds of days are acting up. Configuration files last changed 3 years ago needs tweaking. Programs not changed for 4 years have bugs popping up.

We had an outage of mail (mail bounced) early this morning and later in the afternoon many of the sites were unavailable for a few hours. One of our database servers got "stuck" and it had a cascading effect on some of the web services.

Apologies! IM me at if you see anything else amiss and we'll get it fixed in a hurry.

Monday, September 10, 2007

CPAN Search adds Gravatar icons

At the request of Michael G Schwern gravatar icons have been added to author pages on CPAN search.

Any author can get a gravatar icon on their page by registering at using their CPAN email alias.

Wednesday, July 18, 2007 outage

The server went kaboom. The kernel killed itself with lots of Out Of Memory errors.

I can't figure out the magic incantation to get into the Lights Out Management on that box (it's an old IBM server) and it appears we set it up to use that rather than the Cyclades console and power management.

Anyway - the datacenter staff has been sent to our cabinet to get it booted, so hopefully it won't be too long before it's back. Worst case it'll be up tomorrow morning (PST) when Robert or I go down there and find out what's going on (sorry, you will have to do without the svn server for 7 hours).

Thursday, July 5, 2007

Thanks Leon!

We just wanted to thank Leon for his assistance in providing a copy of BackPan we could use to validate our local copy against.  Be sure to check out his recipies!

Sunday, June 10, 2007

~All is well

Almost everything is back up. Apologies for the super-extended outage. I suppose it's bound to happen every few years, but we'll get to work on making sure this particular failure won't happen again.

Roberts been nudging the MySQL databases and no data should be lost. (A few databases did disappear, but we should be able to find a backup or otherwise restore it from exported data for all of them). Cross your fingers for us!

We're still working on making sure all the RT data is in good shape before putting it online, but it should be back one of the next few days.

Let us know at ask@ and robert@ if you see anything strange in the next few days. (Or when RT is back then at webmaster@ ...)

Thank you for your patience!

Friday, June 8, 2007

RAID went boom part 2

The system is still recovering from the failure. Details are long and boring, but computers suck. :-)

Ironically we've just been talking (again) about getting the services running on the failed server made redundant, but haven't actually done it yet. It's our biggest SPOF by far. Grrh.

Right now the ETA is "in the morning" (PST). Robert is sleeping and will check on it when he gets up. I'll keep an eye on it for a little bit longer (and then go to bed because tomorrow it's my birthday and hopefully I'll have to eat a big cake or something!)

Something Go Boom

Something's broken on our main fileserver.  It's related to our disk array.  We're working on fixing it.  More news as it happens.  Lots of stuff may be broken until we get it back up.

Update 7:10pm:  We've had a disk failure, and the raid volume got unhappy.  We don't think we've lost any important data, but are verifying.  Things are going slow because it takes a while to check half a terabyte of data.\

Update 11:40pm: This is taking a while to resync and fsck.  (Plus we accidentally aborted it halfway through.)   We're going to go to bed, we'll try and have services back in the morning.

Thursday, May 24, 2007

May is for Maintenance

Friday, May 25th at around 10:30am PDT we're going to begin performing maintenance on some of our servers. For a few hours you may notice some oddness or unavailability. We'll post here with any updates and when we're done.


Maintenance is over, it went off without a hitch. RAM upgraded, bad disks replaced.

Monday, May 21, 2007

subversion sidegrade

You may have noticed we had about two hours of subversion downtime tonight on You shouldn't (we hope!) notice anything different, but we've converted our repositories from the BerkeleyDB backing store to FSFS. (Design Doc). BDB has served us well with no major issues for years, but it was time to change.

Wednesday, April 4, 2007


We've started upgrading some of our boxes to Red Hat Enterprise Linux 5 which includes virtualization. We haven't entirely figured out how we are going to use it, but it'll likely make it easier for us to host more things, which is good...

Thursday, March 22, 2007

Bad Spam Day, But Nice Weather

<me> Today is a bad spam day.

<someone> nice weather. all the spammers are coming out.

<someone else> no comment

<someone> crap, are you spamming today?

There's a lot of email spam slipping through our filters over the past few days, and that stinks! (It's getting through other people's filters too - so we're not the only ones having trouble.) We're working on tightening things up - but we have day jobs too - so please bear with us and don't report us to spamcop.

Monday, March 19, 2007

New CPAN Search mirror(s)

One of the boxes serving CPAN Search to the US and the world has been about to fall over for a while (one of the disks hangs the system for a minute once in a while with scsi errors...).   We are working on setting up new a pair of load balanced (for redundancy) mirrors provided by YellowBot.

Ideally we'll get it setup so the DNS servers will check and notice when one of the mirrors are out too... (patches to pgeodns are welcome - we even have a wiki for it and other infrastructure things, send an email to get invited to it).

We could still use a well-connected box for CPAN Search in Asia -- it probably wouldn't get much traffic, but it'd decrease latency for users there.  Minimal specs required (it can be in a virtual box, vmware or xen): ~2.5GHz CPU, ~2GB ram, not much disk space, ability to install RHEL (we'll provide a license).

Wednesday, March 14, 2007

Oops - several services down!

Oops - while doing something else in the datacenter earlier today it seems like I managed to unplug one of our switches! The friendly people there are looking into it. If they don't find the problem I'll go down there shortly.

update: Everything is back!

Tuesday, March 6, 2007

perl infrastructure news

Many visitors come here from the CPAN Search Site that we run with Graham Barr. They all end up on the CPAN Search News page, which doesn't get the regular updates that often does affect that site. Oops.

We do have more redundant mirrors of the search site than the rest of the infrastructure, but unplanned outages still affect it. So, if you are here for operational news regarding, be sure to check out the main weblog. :-)

Thursday, February 22, 2007

"Planned" Power Outage

All hosted services will be taken offline shortly before 12:01AM PST on February 25th. (That's late Saturday night.) Our building is having a scheduled power outage because "a conductor cable has been compromised to the point that immediate action to repair it is necessary or a ground fault will occur to the building systems". (The memo from the building goes on to explain that if the cable fails, it will be worse than crossing the streams from proton packs.)

Ask and I will be keeping an eye on things, but don't be surprised if things are down well into Sunday the 26th California time. No email should be lost, but it may be greatly delayed.

You can also read about this on the Dreamhost Status Blog. One of their datacenters is in the same building as we are.

update: All services should be back (they started coming back between 2 and 3am PST and I got the last ones going around 6am or so. I also got our console server fixed so I don't have to spend half the night here too soon again (I hope, knock on wood...). It's 7am now and Robert should be up soon and can fix things that are broken if you email webmaster@obvious.domain -- Ask)

Saturday, February 3, 2007

Feeds on the new list archive site

The new list archive now has RSS and Atom feeds - both of individual postings and just of new threads.

The server that's running the old site is very much on its last legs, so I'm planning to switch over the main url over to the new site shortly unless someone points out something terribly wrong. :-)