Yikes, the perl.org data center unbelievably had another power outage. Lots of services didn't start up; I'm working on it.
The list server is slightly broken and can't get going without a kick (we are getting a replacement), so it won't be back until I've been by to, well, kick it. I should be there in a couple of hours ...
Update: All should be well again. Email me at ask@perl.org if you find a service that's still down...
I did a bit of work on CPAN Ratings today.
The RSS feeds for reviewers and distributions should work (better) now.
I added "alternate" headers for the RSS feeds so your tools can find the RSS feeds more easily.
I also implemented a proper API for the helpful votes. This was just to make it easier to make other API things in the future (and to maybe make the site support non-javascript browser some day ;-) )
If the site doesn't work properly, be sure to clear your cache, shift-reload etc etc. (The .js and .css files aren't versioned so your browser or ISP proxy might have cached the old versions).
I've been working on a new list archive. There's still work to be done, but you can test the Work In Progress at the beta site.
As part of an effort to translate the perl documentation to other languages Joergen W Lang graphed the perldoc.perl.org logs from the first week of October.
The perlfoundation.org DNS has temporarily reverted back to ancient settings. We expect to have it sorted tomorrow (Friday).
Emails to @perlfoundation.org might not work until then.
The websites can be accessed with the "-" version, www.perl-foundation.org and news.perl-foundation.org (the non "-" version is usually what we normalize them to).
perl.org and pm.org services are not affected.
Some months ago our european search.cpan.org mirror went down with some hardware trouble.
Our friends at Digital Craftsmen kindly replaced the server some time ago and Graham and I finished the setup just now and put it "back in rotation".
If you are in Europe you should see a "hosted by digital craftsmen" thing at the bottom of the search.cpan.org page now.
If you notice any trouble with the service, please let us know.
The building is doing some sort of upgrade to the power system this weekend; hopefully it won't impact us in a negative way. (Crossed fingers, knock on wood).
We have been informed by the building management that on Friday, August 11, 2006, the Department of Water and Power, City of Los Angeles will be upgrading one existing 2500 KVA transformer to a new 3750 KVA transformer on the MST Grid here at the Garland Building. This project will commence at 6:00pm PST on Friday, August 11, 2006 and conclude on Sunday, August 13, 2006 at 9:00pm PST.
During the installation of this transformer the normal electrical power for the MST Grid will be transferred from the DWP utility grid to the Building’s emergency diesel generator plant. ABM Engineering has informed Morlin Asset Management that this transformer upgrade will affect the Garland Building in the following ways:
At exactly 6:00pm PST on Friday, August 11, 2006 on the MST Grid only, a two (2) minute outage will occur.
When the DWP utility grid is restored at 9:00pm PST on Sunday evening, another two (2) minute outage will take place.
We have been assured by building engineering that the service that is being upgraded will not have any impact to the power that provides service to IX2's facilities including any of the cooling equipment for the building. This work is the first step in the building's plan to increase the UPS capacity of the building. Again, this outage will not impact IX2 and our clients according to building management.
The SSL certificate for svn.perl.org expired yesterday. We'll get it replaced ASAP.
Error validating server certificate for 'https://svn.perl.org:443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
- The certificate has expired.
Certificate information:
- Hostname: svn.perl.org
- Valid: from Aug 4 06:20:36 2004 GMT until Aug 4 06:20:36 2006 GMT
- Issuer: Certificate Authority, Develooper, Los Angeles, CA, US
- Fingerprint: 67:09:93:f3:3b:41:f2:7e:0f:fe:6c:1b:fd:b4:2a:fb:65:f0:29:e7
Dreamhost's datacenter is in the same building that the perl.org rack is in. They put together a wonderful blog post that's a very good summary of last week's troubles. (Of cousre, their troubles are an order of magnitude or seven greater than ours.)
svn is running again. I'm very glad I didn't try and repair it last night, because all of the repositories would have ended up vaporized, as I slumped over the keyboard and triggered the rm -rf / macro I have bound to Ctrl-Alt-2oSDLFHq.
we had another "power event" today, and at least one of the repositories on svn.perl.org ended up with some minor corruption. I need sleep, so I'm going to bed. I'll fix things in the morning. (I don't want to do a rush job tonight and mess something else up.)
Apparently MySpace is hosted in the same building we are. When they go down, they get press coverage and more press coverage and international press coverage.
It has not been a good weekend for our datacenter. It all started on Saturday, when Los Angeles experienced record breaking temperatures. (I spent the afternoon outside, and it was a scorcher... 110 degrees plus.) There was a power failure in the building that hosts our datacenter...
the backup power kicked in, but failed after a time period when it got too hot. Our machines lost power briefly, and all but one came back up. Because Murphy's law always takes effect when you least want it to - the machine that didn't come back up was the one that hosts all the perl.org mailing lists. The datacenter personel attempted to reset it, but as they were dealing with many other customers (much more important than us - I don't blame them), they didn't have time to hook a monitor up to our system and see what was going on. So, at 11pm, I drove down to the datacenter to find out that all it wanted was "press F1 to continue". Further diagnosis showed that the bios battery was gone, and the case open sensor kept tripping. Even if the bios was set to not prompt, it would "conveniently" forget that fact. (Did I mention that I was leaving to go to Portland for OSCON in the morning?)
Today, we recieved a note that we may lose power due to some emergency maintenance the building was going to perform to repair electrical damage caused by yesterday's outage. So, instead of having to deal with fscking and rapid power loss, we shut down all of the systems. Severla hours later, we attempted to turn them back on - but only 50% came up! The datacenter staff helped reset the rest, and gave the ornery list box from above the 'f1' treatment. Everything is back up and happy now.
I know that several other companies hosted in the same building lost power, and not just in our datacenter. One, a large perl shop, is still down -- going on six hours. For larger deployments, they are concerned more with heat dissipation - so need to wait for things to cool down. I'm very happy with our hosting arrangements - they've been very helpful with getting boxes reset - and I know things are worse for them than they are for us.
This weekend has identified some weaknesses in our architechture, and we're going to be working over the next few months to solve them. While it doesn't make sense for us to have a fully distributed system, we could definitely use more redundancy in some core systems. We'll probably be posting here with an updated "wish list" soon.
Fingers crossed that the rest of this week goes smoothly. It's no fun having to deal with a datacenter from hundreds of miles away.
