Rainy Day Outage (Update One)

Ask and I have confirmed that we've had a hardware failure and are planning to go to the datacenter tomorrow night California time to attempt to repair it.

We've been able to restore most services, but there are a few which were dependent on the failed machine.  It'll be much easier for us to get the machine fixed before bringing these services back. 

We do not believe any data has been lost, it's just temporarily inaccessible.  All email and mailing lists are working (although some mail from earlier this afternoon may be delayed.)

As of now, the following services are unavailable or degraded:

• rt.perl.org


• www.pm.org and hosted *.pm.org


• historical cpan-testers archives


• svn.perl.org

We'll post another update tomorrow.

Rainy Day Outage

It's raining (again) here in Los Angeles...

Some perl.org services may be experiencing an outage now.  It looks like we've got to go down to the colo to diagnose a possible power issue.  More details later.

List Tags Launched

Last week, we asked for help with tagging the lists.perl.org data.  I'd like to publicly thank Arthur Barrett for providing a patch that contained tag data.  We've now got a pretty tag cloud and more flexible categorization.   

List Tagging

When we redesigned lists.perl.org last month we launched with a hard-coded list of categories maintained in static HTML files.  All the rest of the data is stored in a well structured JSON file.  We're looking for a volunteer who is interested in spending a few hours going through the JSON file and adding tag based categorization to all the lists.  This will make it easier for people to find the perl related list they want, plus that modern invention, a tag cloud!

If you're interested, drop us a line at webmaster at perl.org, or even better, a patch that adds a new "tag" element to every list containing a comma separated list of tags.

Link Checking

Once a month we get an email from Nodeworks, the link checking service we use.  

On really good months, the subject looks like this:

[NodeWorks] 0 dead of 3164 links for http://www.perl.org 

(This particular one came after our recent redesign.  Not one dead link slipped in.)

Mailing list upgrades

We've been doing some (long overdue!) mailing list upgrades and in the process we managed to make the pm.org lists not work for a couple of hours and the perl.org lists for about 5 minutes earlier in the evening. All should be well now, but do let us know if you notice something amiss.



As an added bonus we now have lists on lists.perlfoundation.org. Subscribe to the first of them by emailing events-subscribe at that domain.

perl.org outage

One of our remaining single-point-of-failure-servers is doing it's duty demonstrating to us why we really should get rid of the last single-point-of-failures.  Our console server is down, too (arrgh!), so no ETA yet but we'll get it fixed as soon as possible.

Update 8.30 PST - the troubled server is coming back now and we'll prioritize our plans to get things off this box.  (It is among other things our shared NFS server from back at a time when saving a couple of gigabytes was worth the pain of NFS; obviously that's not true anymore).

A new look for perl.org

To match the massive advances in Perl over the last few years, www.perl.org has been brought into the modern era. www.perl.org has been completely redesigned, making it clearer and easier to use.  All the content has been reviewed and brought up-to-date to provide links and other helpful resources for both new and experienced Perl programmers.

Thanks to www.foxtons.co.uk for donating time from Leo Lapworth, Stephen Morgan, and Cameron Richmond!

All is well again

We got most things up pretty quickly and I've beaten our console server into submission again so it's ready next time we need it (going to the data center sucks).  If there are any of our services still missing, please let us know.

Fuel Pump Fail

The building where the perl.org datacenter is hosted was performing safety tests today that involved running on generator power for a few hours.  No problems were expected, as our UPS would have (and did) covered the transition.  And then a fuel pump failed in one of the generators, requiring it to be shut down, resulting in us losing power.

Several machines didn't come back up.  So if your favorite perl.org service isn't available or isn't working right today, that's why. 

Ask is on his way to the datacenter to get things sorted.  We'll update this blog as we have more information.

Blackholes!

This morning, one of our upstream sources of "is this ip a possible source of spam or not" that we use for scoring spam stopped serving real data and started returning "YES!!!!!!" to everything.  This may have caused email that already contained a weak spam signal to be marked as spam and rejected.  We've removed the bad data source, and things should be back to normal now.

Still notice less spam?

A few weeks ago we posted that we were using a new source of anti-spam data.  Today we're happy to share that the new source was the invaluement Anti-Spam DNSBL.  It has resulted in almost a 10% reduction in spam that makes it through to our mailing list moderator, and zero complaints about improperly blocked mail. Thanks to Rob McEwen for developing and maintaining these lists.

Looking for servers in Asia

We'd like to make some of the perl.org services (Search CPAN for example) faster for users in Asia; so we're looking for servers (Xen or KVM based servers are fine as long as we can run RHEL) in for example Japan, Taiwan or Singapore.

If you think you might be able to help, please email ask@perl.org.