We finished the upgrades around 8pm (PST). It was surprisingly easy to upgrade from RHEL3 to RHEL5. The only thing we've run into so far were some of our custom built things we had compiled with the system OpenSSL that needed recompiling.
Most services are up again (and have been for some hours). We're upgrading our monitoring and system management tools and we're still tweaking and configuring, so do let us know if you see anything broken.
Friday, December 8th, 2007 will be our "annual" datacenter day. This means that Ask and I will go down to the colocation facility, swap drives, rewire things, upgrade BIOSes, upgrade operating systems, and reaffirm our frustration with hardware. (Our Cyclades Console Server means we hardly ever have to physically visit the machines.)
Starting at around 12pm PST (20:00 UTC) some perl.org services may be unavailable. Email to perl.org, cpan.org, and other hosted domains may be delayed. Some websites including PerlBug, CPAN Ratings, www.perl.org, www.parrotcode.org, www.perldoc.org, search.cpan.org, may be unavailable. We have a plan to minimize downtime for each service.
We expect the maintenance period to last less than six hours. Otherwise, we'll be ordering some pizza.
One of the biggest projects on our list for tomorrow is an upgrade to Redhat Enterprise Linux 5. Many of our boxes are currently running RHEL3, which is getting a little long in the tooth. We're looking forward to the newer kernel and more modern utilities.
We'll post here if there are any outstanding issues.
Our main internal fileserver has gone funky. Due to the joy of NFS, this is making other servers unhappy too. We suspect bad ram, but don't know yet. The machine is currently rebooting and fscking. We'll update this post as we learn more.
Update 8:54pm: The machine is back up, nothing appears to be horribly wrong, although we lost one disk from a RAID set. We're moving some data around, but things should be returning to normal shortly.
Update 10:16pm: Everything seems to be fine now. I expect those to be famous last words.
As our holiday gift to you, we provide you with some amusing monkey spam that made it through our spam filters at perl.org. This one was sent to a CPAN related email address, from a hosting provider's machine in Texas. The phone number has an Ohio area code, but is missing the last digit, so even if you were interested in a monkey, you'd need to write back to pastor patrick's Yahoo Mail account.
At perl.org we do our best to keep spam out of your mailboxes and off of our mailing lists. It's not easy, as the only thing worse than letting too much spam through -- is blocking legit email. Over the past month, we've made some changes to our mail system to ensure that mail continues to flow fast and furious... and we have some changes planned for the near future to continue to improve our systems.
And now... the monkey spam:
hello,
Complement of the season,
MY NAME IS joe helderberg, I need a responsible someone to adopt my baby capuchin , My baby is still available for christmas adoption and his 7weeks old now,weight 7lbs,Akc regis and am asking for $800.presently am in our church camp with my pets in Ohio.just tell me if you are still interest in my baby monkey so that i can tell pastor patrick about the baby shipment.
I will be waiting to hear from you if you are still interest asap.
God bless
more info.(614)xxx-xxx
