As you've may have already noticed, perl.org is back up. The move is complete (all 0.7 miles of it.) At this point, we believe that everything is working again (at least as well as it was before the move), but it's possible that we missed something. If you find something, please let us know at webmaster at perl.org, and we'll fix it when we wake up.
Special thanks to IX2 (our awesome colo provider) and Phyber (our new transit provider)!
We are now beginning the move of the rest of our boxes to our new datacenter. See you in a few hours. Watch this space for updates.
The perl.org datacenter move (that was going to happen today) has been postponed while we stabilize our network connection at the new site. The move may happen tomorrow, Sunday, March 22nd, or later this week. We'll try and remember to post here before we take things down.
perl.org is moving... down the street.
Our wonderful colocation provider, IX2 Networks, is closing down one of their Los Angeles facilities, and we're moving to another one. We're going to be sharing a cage with YellowBot. We're looking forward to this move, as the new facility is above ground, our cell phones work, and you can even kind of see daylight if you try.
Since we're going to be physically moving our boxes, this means there will be some downtime and service unavailability. For a few hours between Thursday, March 19th and Saturday, March 21st, some perl.org (and related) services may not function for you. (We are being vague because we haven't yet decided when to actually move the hardware.) We are going to try and minimize downtime as much as we can. No email will be lost, although it may be delayed. DNS services will still work fine.
We will update http://log.perl.org as we go.
Tonight, we moved svn.perl.org (and a few other svn vhosts) from real hardware to a virtual Xen box. You shouldn't notice anything different, but if you do, please let us know.
The box hosting rt.cpan.org (yes, right now it's just on one box) was rebooted to get a bit more memory. It took a bit longer than usual because the hardware it's running on was rebuilding its raid.
Today search.cpan.org got a tiny slightly bit faster. We're serving the CSS and images via the PantherExpress CDN, so in particular if you are in Asia/Australia or thereabout it'll all load a fraction or two of a second faster.
For the main site and search currently users in North America are served from Montréal (hosted by Weblocal), European users are served from London (hosted by Digital Craftsmen) and everyone else (most notably South America and Asia) are served from the perl.org servers in Los Angeles.
Whoops. We had one outage earlier that dragged down a server (OOM killer went nuts). I rebooted it and of course it won't come back up. Either the (raid'ed) disk got corrupted, the bios ran out of battery and lost some critical settings or some variation of that. I can't get into the BIOS to make it netboot remotely.
In any case, I'm in Europe and Robert might have the day off (imagine!), so all the services (including www.pool.ntp.org and www.perl.org) running on that box might be down for the day.
Graham (who runs search.cpan.org) let us know that two of the machines appeared to be running very slowly, and he thought it might be NFS related, as everything else looked fine.
Here's what we found looking at where requests to our NFS server were coming from...
oops.
Turns out Apache::Reload had been accidentally turned on on one of the production servers for geourl and *every* page hit was causing a stat of a few hundred .pm files. Turned it off, and NFS load dropped by 80%, and everything went back to normal.
Tune in next week for "Don't use NFS in production".
(And yes, this entire post exists to show off the pretty Google Chart.)
Twenty miles to the East, fires burn.
But here at perl.org headquarters, it's a slow day.
Not that we're not busy as usual, but today our primary webserver was running veeeeeeeeery veeeery slooooooooowww..
I think it's just one of those days.
Today's issues:
But everything's back to fast and normal now.....
Today's message came in to one of the CPAN support addresses. We could tell him about Bundle::Everything (out of date as it may be), although I suspect he doesn't have a comfortable enough chair to read them all.
Hello dear Hack-o-holics !
Is this not possible ???
I want to download all available perl-modules in one bundle !?It is crucial, to download every useful module separated.
One bundle and then I can read them in peace at home.
Thank you
