We finished the upgrades around 8pm (PST). It was surprisingly easy to upgrade from RHEL3 to RHEL5. The only thing we've run into so far were some of our custom built things we had compiled with the system OpenSSL that needed recompiling.
Most services are up again (and have been for some hours). We're upgrading our monitoring and system management tools and we're still tweaking and configuring, so do let us know if you see anything broken.
Friday, December 8th, 2007 will be our "annual" datacenter day. This means that Ask and I will go down to the colocation facility, swap drives, rewire things, upgrade BIOSes, upgrade operating systems, and reaffirm our frustration with hardware. (Our Cyclades Console Server means we hardly ever have to physically visit the machines.)
Starting at around 12pm PST (20:00 UTC) some perl.org services may be unavailable. Email to perl.org, cpan.org, and other hosted domains may be delayed. Some websites including PerlBug, CPAN Ratings, www.perl.org, www.parrotcode.org, www.perldoc.org, search.cpan.org, may be unavailable. We have a plan to minimize downtime for each service.
We expect the maintenance period to last less than six hours. Otherwise, we'll be ordering some pizza.
One of the biggest projects on our list for tomorrow is an upgrade to Redhat Enterprise Linux 5. Many of our boxes are currently running RHEL3, which is getting a little long in the tooth. We're looking forward to the newer kernel and more modern utilities.
We'll post here if there are any outstanding issues.
Our main internal fileserver has gone funky. Due to the joy of NFS, this is making other servers unhappy too. We suspect bad ram, but don't know yet. The machine is currently rebooting and fscking. We'll update this post as we learn more.
Update 8:54pm: The machine is back up, nothing appears to be horribly wrong, although we lost one disk from a RAID set. We're moving some data around, but things should be returning to normal shortly.
Update 10:16pm: Everything seems to be fine now. I expect those to be famous last words.
As our holiday gift to you, we provide you with some amusing monkey spam that made it through our spam filters at perl.org. This one was sent to a CPAN related email address, from a hosting provider's machine in Texas. The phone number has an Ohio area code, but is missing the last digit, so even if you were interested in a monkey, you'd need to write back to pastor patrick's Yahoo Mail account.
At perl.org we do our best to keep spam out of your mailboxes and off of our mailing lists. It's not easy, as the only thing worse than letting too much spam through -- is blocking legit email. Over the past month, we've made some changes to our mail system to ensure that mail continues to flow fast and furious... and we have some changes planned for the near future to continue to improve our systems.
And now... the monkey spam:
hello,
Complement of the season,
MY NAME IS joe helderberg, I need a responsible someone to adopt my baby capuchin , My baby is still available for christmas adoption and his 7weeks old now,weight 7lbs,Akc regis and am asking for $800.presently am in our church camp with my pets in Ohio.just tell me if you are still interest in my baby monkey so that i can tell pastor patrick about the baby shipment.
I will be waiting to hear from you if you are still interest asap.
God bless
more info.(614)xxx-xxx
I love when we get weird emails at perl.org. Here's one that came in recently...
Dirk
22.11.2007 15:51Kopie:
Thema: Enquiry PerlHello Joanna,
we got an inquiry from our customer about the following products:
1 Unit of Perl
Please send a formal quotation with specify terms and conditions for
delivery as well as availability, anticipated delivery time and delievery
charges/proccesing fees and dealer discounts.
We also need Information on which operating system the software is
running. Also we need informations on which language
do you´ll deliever the software. Please also let us know on which form
you´ll deliever the software: Boxproduct or electronicaly delievery.ATTENTION: We need your fax-number, because we don´t order online (
Internet security problems)!!Please send your quote to my attention at removed@removed.de.
For more informations about our company please have a look on our homepage
(http://www.removed.de) here you´ll find
an english company profil.Thank you in advance for quick response.
best regards
Dirk
Purchase department
Apparently nasty car crashes isn't the only news this weekend. We seem to have lost a drive in one of the machines serving perl.org. Of course, it's the drive that everything else depends on.
perl.org and cpan.org email and some websites may be non functional while we're repairing it. Check back here for more updates.
Update 6:40pm: Looks like things recovered with some remote percussive maintenance combined with patience. (Have we shared how much we love our Cyclades console servers and power management recently?)
We got a few mails asking about www.cpan.org being down. It's not actually hosted here (anymore). Pair.com is hosting it and they were informed about it earlier today (~5-6 hours ago I think). I'm sure they'll get it back as quickly as possible, but emailing us doesn't help. :-)
Our CPAN mirror is at cpan.perl.org.
For a long time we had a confusing thing where to get a svn.perl.org subversion (and groups.pm.org webdav) account you had to go to auth.perl.org and set it up there.
We've improved it a bit now so resetting your password (or just setting up the account initially) you just go to svn.perl.org. Whee!
Let me know (ask@perl.org) if you have any trouble with the new site.
Geez! This week it seems like everything is falling apart over here. Servers that have been running for hundreds of days are acting up. Configuration files last changed 3 years ago needs tweaking. Programs not changed for 4 years have bugs popping up.
We had an outage of cpan.org mail (mail bounced) early this morning and later in the afternoon many of the perl.org sites were unavailable for a few hours. One of our database servers got "stuck" and it had a cascading effect on some of the web services.
Apologies! IM me at xmpp:ask@plys.net if you see anything else amiss and we'll get it fixed in a hurry.
At the request of Michael G Schwern gravatar icons have been added to author pages on CPAN search.
Any author can get a gravatar icon on their page by registering at gravatar.com using their CPAN email alias.
The svn.perl.org server went kaboom. The kernel killed itself with lots of Out Of Memory errors.
I can't figure out the magic incantation to get into the Lights Out Management on that box (it's an old IBM server) and it appears we set it up to use that rather than the Cyclades console and power management.
Anyway - the datacenter staff has been sent to our cabinet to get it booted, so hopefully it won't be too long before it's back. Worst case it'll be up tomorrow morning (PST) when Robert or I go down there and find out what's going on (sorry, you will have to do without the svn server for 7 hours).
Almost everything is back up. Apologies for the super-extended outage. I suppose it's bound to happen every few years, but we'll get to work on making sure this particular failure won't happen again.
Roberts been nudging the MySQL databases and no data should be lost. (A few databases did disappear, but we should be able to find a backup or otherwise restore it from exported data for all of them). Cross your fingers for us!
We're still working on making sure all the RT data is in good shape before putting it online, but it should be back one of the next few days.
Let us know at ask@ and robert@ if you see anything strange in the next few days. (Or when RT is back then at webmaster@ ...)
Thank you for your patience!
The system is still recovering from the failure. Details are long and boring, but computers suck. :-)
Ironically we've just been talking (again) about getting the services running on the failed server made redundant, but haven't actually done it yet. It's our biggest SPOF by far. Grrh.
Right now the ETA is "in the morning" (PST). Robert is sleeping and will check on it when he gets up. I'll keep an eye on it for a little bit longer (and then go to bed because tomorrow it's my birthday and hopefully I'll have to eat a big cake or something!)
Something's broken on our main fileserver. It's related to our disk array. We're working on fixing it. More news as it happens. Lots of stuff may be broken until we get it back up.
Update 7:10pm: We've had a disk failure, and the raid volume got unhappy. We don't think we've lost any important data, but are verifying. Things are going slow because it takes a while to check half a terabyte of data.\
Update 11:40pm: This is taking a while to resync and fsck. (Plus we accidentally aborted it halfway through.) We're going to go to bed, we'll try and have services back in the morning.
Friday, May 25th at around 10:30am PDT we're going to begin performing maintenance on some of our servers. For a few hours you may notice some oddness or unavailability. We'll post here with any updates and when we're done.
Update:
Maintenance is over, it went off without a hitch. RAM upgraded, bad disks replaced.
You may have noticed we had about two hours of subversion downtime tonight on svn.perl.org. You shouldn't (we hope!) notice anything different, but we've converted our repositories from the BerkeleyDB backing store to FSFS. (Design Doc). BDB has served us well with no major issues for years, but it was time to change.
We've started upgrading some of our boxes to Red Hat Enterprise Linux 5 which includes virtualization. We haven't entirely figured out how we are going to use it, but it'll likely make it easier for us to host more things, which is good...
<me> Today is a bad spam day.
<someone> nice weather. all the spammers are coming out.
<someone else> no comment
<someone> crap, are you spamming today?
There's a lot of email spam slipping through our filters over the past few days, and that stinks! (It's getting through other people's filters too - so we're not the only ones having trouble.) We're working on tightening things up - but we have day jobs too - so please bear with us and don't report us to spamcop.
One of the boxes serving CPAN Search to the US and the world has been about to fall over for a while (one of the disks hangs the system for a minute once in a while with scsi errors...). We are working on setting up new a pair of load balanced (for redundancy) mirrors provided by YellowBot.
Ideally we'll get it setup so the DNS servers will check and notice when one of the mirrors are out too... (patches to pgeodns are welcome - we even have a wiki for it and other perl.org infrastructure things, send an email to get invited to it).
We could still use a well-connected box for CPAN Search in Asia -- it probably wouldn't get much traffic, but it'd decrease latency for users there. Minimal specs required (it can be in a virtual box, vmware or xen): ~2.5GHz CPU, ~2GB ram, not much disk space, ability to install RHEL (we'll provide a license).
Oops - while doing something else in the datacenter earlier today it seems like I managed to unplug one of our switches! The friendly people there are looking into it. If they don't find the problem I'll go down there shortly.
update: Everything is back!
Many visitors come here from the CPAN Search Site that we run with Graham Barr. They all end up on the CPAN Search News page, which doesn't get the regular updates that often does affect that site. Oops.
We do have more redundant mirrors of the search site than the rest of the infrastructure, but unplanned outages still affect it. So, if you are here for operational news regarding perl.org, be sure to check out the main weblog. :-)
All perl.org hosted services will be taken offline shortly before 12:01AM PST on February 25th. (That's late Saturday night.) Our building is having a scheduled power outage because "a conductor cable has been compromised to the point that immediate action to repair it is necessary or a ground fault will occur to the building systems". (The memo from the building goes on to explain that if the cable fails, it will be worse than crossing the streams from proton packs.)
Ask and I will be keeping an eye on things, but don't be surprised if things are down well into Sunday the 26th California time. No email should be lost, but it may be greatly delayed.
You can also read about this on the Dreamhost Status Blog. One of their datacenters is in the same building as we are.
update: All services should be back (they started coming back between 2 and 3am PST and I got the last ones going around 6am or so. I also got our console server fixed so I don't have to spend half the night here too soon again (I hope, knock on wood...). It's 7am now and Robert should be up soon and can fix things that are broken if you email webmaster@obvious.domain -- Ask)
The new list archive now has RSS and Atom feeds - both of individual postings and just of new threads.
The server that's running the old site is very much on its last legs, so I'm planning to switch over the main url over to the new site shortly unless someone points out something terribly wrong. :-)
