Friday, December 28, 2007

Upgrades mostly done

We finished the upgrades around 8pm (PST). It was surprisingly easy to upgrade from RHEL3 to RHEL5. The only thing we've run into so far were some of our custom built things we had compiled with the system OpenSSL that needed recompiling.

Most services are up again (and have been for some hours). We're upgrading our monitoring and system management tools and we're still tweaking and configuring, so do let us know if you see anything broken.

Thursday, December 27, 2007

"Annual" Maintenance - Dec 28th, 2007

Friday, December 8th, 2007 will be our "annual" datacenter day.  This means that Ask and I will go down to the colocation facility, swap drives, rewire things, upgrade BIOSes, upgrade operating systems, and reaffirm our frustration with hardware. (Our Cyclades Console Server means we hardly ever have to physically visit the machines.)

Starting at around 12pm PST (20:00 UTC) some services may be unavailable.  Email to,, and other hosted domains may be delayed.  Some websites including PerlBug, CPAN Ratings,,,,, may be unavailable.  We have a plan to minimize downtime for each service.

We expect the maintenance period to last less than six hours.  Otherwise, we'll be ordering some pizza.   

One of the biggest projects on our list for tomorrow is an upgrade to Redhat Enterprise Linux 5.  Many of our boxes are currently running RHEL3, which is getting a little long in the tooth.  We're looking forward to the newer kernel and more modern utilities.

We'll post here if there are any outstanding issues. 

Wednesday, December 26, 2007

Smething broke (at least it's the day after a holiday)

Our main internal fileserver has gone funky.  Due to the joy of NFS, this is making other servers unhappy too.    We suspect bad ram, but don't know yet.  The machine is currently rebooting and fscking.  We'll update this post as we learn more.

Update 8:54pm: The machine is back up, nothing appears to be horribly wrong, although we lost one disk from a RAID set.  We're moving some data around, but things should be returning to normal shortly.

Update 10:16pm: Everything seems to be fine now.  I expect those to be famous last words.

Tuesday, December 25, 2007

Monkey Spam

As our holiday gift to you, we provide you with some amusing monkey spam that made it through our spam filters at  This one was sent to a CPAN related email address, from a hosting provider's machine in Texas.  The phone number has an Ohio area code, but is missing the last digit, so even if you were interested in a monkey, you'd need to write back to pastor patrick's Yahoo Mail account. 

At we do our best to keep spam out of your mailboxes and off of our mailing lists.  It's not easy, as the only thing worse than letting too much spam through -- is blocking legit email.  Over the past month, we've made some changes to our mail system to ensure that mail continues to flow fast and furious... and we have some changes planned for the near future to continue to improve our systems.

And now... the monkey spam:


Complement of the season,
MY NAME IS joe helderberg, I need a responsible someone to adopt my  baby capuchin , My baby is still available for  christmas adoption and his 7weeks old now,weight 7lbs,Akc regis and am asking for $800.presently am in our church camp with my pets in Ohio.just tell me if you are still interest in my baby monkey so that i can tell pastor patrick about the baby shipment.
I will be waiting to hear from you if you are still interest asap.
God bless
more info.(614)xxx-xxx