perl.org outage

One of our remaining single-point-of-failure-servers is doing it's duty demonstrating to us why we really should get rid of the last single-point-of-failures.  Our console server is down, too (arrgh!), so no ETA yet but we'll get it fixed as soon as possible.

Update 8.30 PST - the troubled server is coming back now and we'll prioritize our plans to get things off this box.  (It is among other things our shared NFS server from back at a time when saving a couple of gigabytes was worth the pain of NFS; obviously that's not true anymore).

A new look for perl.org

To match the massive advances in Perl over the last few years, www.perl.org has been brought into the modern era. www.perl.org has been completely redesigned, making it clearer and easier to use.  All the content has been reviewed and brought up-to-date to provide links and other helpful resources for both new and experienced Perl programmers.

Thanks to www.foxtons.co.uk for donating time from Leo Lapworth, Stephen Morgan, and Cameron Richmond!

All is well again

We got most things up pretty quickly and I've beaten our console server into submission again so it's ready next time we need it (going to the data center sucks).  If there are any of our services still missing, please let us know.

Fuel Pump Fail

The building where the perl.org datacenter is hosted was performing safety tests today that involved running on generator power for a few hours.  No problems were expected, as our UPS would have (and did) covered the transition.  And then a fuel pump failed in one of the generators, requiring it to be shut down, resulting in us losing power.

Several machines didn't come back up.  So if your favorite perl.org service isn't available or isn't working right today, that's why. 

Ask is on his way to the datacenter to get things sorted.  We'll update this blog as we have more information.

Blackholes!

This morning, one of our upstream sources of "is this ip a possible source of spam or not" that we use for scoring spam stopped serving real data and started returning "YES!!!!!!" to everything.  This may have caused email that already contained a weak spam signal to be marked as spam and rejected.  We've removed the bad data source, and things should be back to normal now.

Still notice less spam?

A few weeks ago we posted that we were using a new source of anti-spam data.  Today we're happy to share that the new source was the invaluement Anti-Spam DNSBL.  It has resulted in almost a 10% reduction in spam that makes it through to our mailing list moderator, and zero complaints about improperly blocked mail. Thanks to Rob McEwen for developing and maintaining these lists.

Looking for servers in Asia

We'd like to make some of the perl.org services (Search CPAN for example) faster for users in Asia; so we're looking for servers (Xen or KVM based servers are fine as long as we can run RHEL) in for example Japan, Taiwan or Singapore.

If you think you might be able to help, please email ask@perl.org.

PerlMonks compromised, some PAUSE accounts potentially at risk

If you're a CPAN author with a PAUSE account (or just a PerlMonks user) you may be interested in the below. (And you should already have received it as an email.) The same perpetrators have been getting press for also hacking Dan Kaminksy and Kevin Mitnick. Details on PerlMonks here and here.

Dear CPAN author,

This email is being sent to inform you that all passwords on the popular Perl Monks website were compromised. Many CPAN authors have accounts there and in some cases have used the same password for PAUSE. 

If you have any reason to suspect that your PAUSE account password is no longer secure, please visit https://pause.cpan.org/ and change it.

 

If your PAUSE account is not affected, please disregard this message and
accept apologies for the unsolicited email.

 

Regards,


PAUSE Administrators

perldoc.perl.org updated

JJ has updated perldoc.perl.org to a new look and feel:

The main change is a complete new visual design, bringing a fresh, modern look to the site. Additionally there are a number of new features to aid navigation and usability - a floating page index window, recently read pages list, improved Pod rendering, and many more.

Notice less spam?

I just (~11:30PM PDT, July 15th 2009) put into place a new source of data for our spam filtering.  Hopefully this will result in less spam getting through to @cpan.org addresses, @pm.org addresses, and into the moderation queue for @perl.org mailing lists.

As with any spam filtering change, there is a risk that we've inadvertently caused legitimate mail to be blocked.  If you notice this, please let us know.

In a few weeks, if all has gone well, I'll give more details on the new spam filtering data.  If things haven't gone well, we'll revert the changes and pretend this never happened.

Whoops!

Accidentally posted something to the wrong blog.  Now posting this to see if I can get the RSS feed to regenerate.  Ignore this post.  And ignore the one before it too.

email issues: .org blocked, now fixed

For about the past 18 hours, you may have experienced issues sending emails to perl.org/cpan.org/pm.org addresses, if those emails contained a reference to any .org URI.  One of the uribl providers we use accidentally added all of .org to their list of things to block.  

This has likely resulted in mail being permanantly rejected, and it will not be resent.   If you sent such an email, you should have received a bounce message from your mail provider. If you are subscribed to any other .org email lists via your cpan.org address, you may wish to double check that you haven't been unsubscribed due to bouncing.

We have disabled the errant uribl provider, and informed them of the issue.  We may not re-enable them after they fix the problem.  (Although, historically, their data has been a very good signal for blocking spam.) 

Attentive network services

One of the great things with Internap was that they were extremely proactive; the few times there was trouble in their network we'd have their mail telling about it at the same time as we got the monitoring alerts.  When we unplugged all the equipment to move it a few months ago we had a voicemail from them when we came up above ground to cell phone coverage.

I'm happy to say that getting ip transit from Phyber is much of the same.  Last week the MSN web crawler did ~10Mbit of traffic crawling one of the perl.org servers for a few days.  When it stopped and the traffic dropped we had an email from Phyber checking in if everything was okay!  We hadn't noticed that the traffic came and went, but they did.  :-)