Mailing list upgrades

We've been doing some (long overdue!) mailing list upgrades and in the process we managed to make the pm.org lists not work for a couple of hours and the perl.org lists for about 5 minutes earlier in the evening. All should be well now, but do let us know if you notice something amiss.



As an added bonus we now have lists on lists.perlfoundation.org. Subscribe to the first of them by emailing events-subscribe at that domain.

perl.org outage

One of our remaining single-point-of-failure-servers is doing it's duty demonstrating to us why we really should get rid of the last single-point-of-failures.  Our console server is down, too (arrgh!), so no ETA yet but we'll get it fixed as soon as possible.

Update 8.30 PST - the troubled server is coming back now and we'll prioritize our plans to get things off this box.  (It is among other things our shared NFS server from back at a time when saving a couple of gigabytes was worth the pain of NFS; obviously that's not true anymore).

A new look for perl.org

To match the massive advances in Perl over the last few years, www.perl.org has been brought into the modern era. www.perl.org has been completely redesigned, making it clearer and easier to use.  All the content has been reviewed and brought up-to-date to provide links and other helpful resources for both new and experienced Perl programmers.

Thanks to www.foxtons.co.uk for donating time from Leo Lapworth, Stephen Morgan, and Cameron Richmond!

All is well again

We got most things up pretty quickly and I've beaten our console server into submission again so it's ready next time we need it (going to the data center sucks).  If there are any of our services still missing, please let us know.

Fuel Pump Fail

The building where the perl.org datacenter is hosted was performing safety tests today that involved running on generator power for a few hours.  No problems were expected, as our UPS would have (and did) covered the transition.  And then a fuel pump failed in one of the generators, requiring it to be shut down, resulting in us losing power.

Several machines didn't come back up.  So if your favorite perl.org service isn't available or isn't working right today, that's why. 

Ask is on his way to the datacenter to get things sorted.  We'll update this blog as we have more information.

Blackholes!

This morning, one of our upstream sources of "is this ip a possible source of spam or not" that we use for scoring spam stopped serving real data and started returning "YES!!!!!!" to everything.  This may have caused email that already contained a weak spam signal to be marked as spam and rejected.  We've removed the bad data source, and things should be back to normal now.

Still notice less spam?

A few weeks ago we posted that we were using a new source of anti-spam data.  Today we're happy to share that the new source was the invaluement Anti-Spam DNSBL.  It has resulted in almost a 10% reduction in spam that makes it through to our mailing list moderator, and zero complaints about improperly blocked mail. Thanks to Rob McEwen for developing and maintaining these lists.

Looking for servers in Asia

We'd like to make some of the perl.org services (Search CPAN for example) faster for users in Asia; so we're looking for servers (Xen or KVM based servers are fine as long as we can run RHEL) in for example Japan, Taiwan or Singapore.

If you think you might be able to help, please email ask@perl.org.

PerlMonks compromised, some PAUSE accounts potentially at risk

If you're a CPAN author with a PAUSE account (or just a PerlMonks user) you may be interested in the below. (And you should already have received it as an email.) The same perpetrators have been getting press for also hacking Dan Kaminksy and Kevin Mitnick. Details on PerlMonks here and here.

Dear CPAN author,

This email is being sent to inform you that all passwords on the popular Perl Monks website were compromised. Many CPAN authors have accounts there and in some cases have used the same password for PAUSE. 

If you have any reason to suspect that your PAUSE account password is no longer secure, please visit https://pause.cpan.org/ and change it.

 

If your PAUSE account is not affected, please disregard this message and
accept apologies for the unsolicited email.

 

Regards,


PAUSE Administrators

perldoc.perl.org updated

JJ has updated perldoc.perl.org to a new look and feel:

The main change is a complete new visual design, bringing a fresh, modern look to the site. Additionally there are a number of new features to aid navigation and usability - a floating page index window, recently read pages list, improved Pod rendering, and many more.

Notice less spam?

I just (~11:30PM PDT, July 15th 2009) put into place a new source of data for our spam filtering.  Hopefully this will result in less spam getting through to @cpan.org addresses, @pm.org addresses, and into the moderation queue for @perl.org mailing lists.

As with any spam filtering change, there is a risk that we've inadvertently caused legitimate mail to be blocked.  If you notice this, please let us know.

In a few weeks, if all has gone well, I'll give more details on the new spam filtering data.  If things haven't gone well, we'll revert the changes and pretend this never happened.

Whoops!

Accidentally posted something to the wrong blog.  Now posting this to see if I can get the RSS feed to regenerate.  Ignore this post.  And ignore the one before it too.

email issues: .org blocked, now fixed

For about the past 18 hours, you may have experienced issues sending emails to perl.org/cpan.org/pm.org addresses, if those emails contained a reference to any .org URI.  One of the uribl providers we use accidentally added all of .org to their list of things to block.  

This has likely resulted in mail being permanantly rejected, and it will not be resent.   If you sent such an email, you should have received a bounce message from your mail provider. If you are subscribed to any other .org email lists via your cpan.org address, you may wish to double check that you haven't been unsubscribed due to bouncing.

We have disabled the errant uribl provider, and informed them of the issue.  We may not re-enable them after they fix the problem.  (Although, historically, their data has been a very good signal for blocking spam.) 

Attentive network services

One of the great things with Internap was that they were extremely proactive; the few times there was trouble in their network we'd have their mail telling about it at the same time as we got the monitoring alerts.  When we unplugged all the equipment to move it a few months ago we had a voicemail from them when we came up above ground to cell phone coverage.

I'm happy to say that getting ip transit from Phyber is much of the same.  Last week the MSN web crawler did ~10Mbit of traffic crawling one of the perl.org servers for a few days.  When it stopped and the traffic dropped we had an email from Phyber checking in if everything was okay!  We hadn't noticed that the traffic came and went, but they did.  :-)

google347cd5ef81c7809c

google verify

svn.perl.org briefly down

svn.perl.org will be unavailable for the next 15-25 minutes while we move it to faster storage.

Minor network issue resolved

Some of the switches between us and our uplink didn't play nice together and it made the link effectively only allow each connection to transfer at a couple megabits a second (or less!)

We got it resolved yesterday and now you can download Perl in just a few seconds again.

IP transit from Phyber Communications

As part of the move a couple weeks ago the perl.org internet connection got upgraded from an often somewhat over-saturated 10Mbit connection (hey, that was pretty fast 8 years ago!) to a 100Mbit link generously provided by Phyber Communications.

We can't say enough good things about the insanely effective and proactive Internap NOC, but after years of getting free connectivity from them (it was setup by our friends at Ticketmaster who called in a few favors), it was time to try something new.  We're very grateful that Max Clark and Phyber got us hooked up.

One of the exciting things is that we now have an IPv6 connection, too.  Over the summer we're planning to start running some services on that.

nntp.perl.org and pm.org lists temporarily down

The server running the nntp server and the pm.org lists has gone temporarily insane.  We'll fix it in the morning.

Update: I lucked out that my "maybe this will fix it" attempt I started while going to sleep fixed it, so it was fixed shortly after breaking.

svn.perl.org and rt.cpan.org a little slow tonight

svn.perl.org and rt.cpan.org are both running a little slow right now because we're moving data around on the underlying xen instance, and it's causing IO starvation to the box.  Things should be back to normal later tonight.

Downtime (long) over, move complete!

As you've may have already noticed, perl.org is back up.  The move is complete (all 0.7 miles of it.)  At this point, we believe that everything is working again (at least as well as it was before the move), but it's possible that we missed something.  If you find something, please let us know at webmaster at perl.org, and we'll fix it when we wake up.

Special thanks to IX2 (our awesome colo provider) and Phyber (our new transit provider)!

Downtime / Move Starting Now

We are now beginning the move of the rest of our boxes to our new datacenter.  See you in a few hours.  Watch this space for updates.

Move slightly delayed

The perl.org datacenter move (that was going to happen today) has been postponed while we stabilize our network connection at the new site.  The move may happen tomorrow, Sunday, March 22nd, or later this week.  We'll try and remember to post here before we take things down.

perl.org is moving! (downtime announcement)

perl.org is moving... down the street.

Our wonderful colocation provider, IX2 Networks, is closing down one of their Los Angeles facilities, and we're moving to another one.  We're going to be sharing a cage with YellowBot.  We're looking forward to this move, as the new facility is above ground, our cell phones work, and you can even kind of see daylight if you try.

Since we're going to be physically moving our boxes, this means there will be some downtime and service unavailability.  For a few hours between Thursday, March 19th and Saturday, March 21st, some perl.org (and related) services may not function for you.  (We are being vague because we haven't yet decided when to actually move the hardware.)  We are going to try and minimize downtime as much as we can.  No email will be lost, although it may be delayed.  DNS services will still work fine.

We will update http://log.perl.org as we go.

svn.perl.org moved to a new machine

Tonight, we moved svn.perl.org (and a few other svn vhosts) from real hardware to a virtual Xen box.  You shouldn't notice anything different, but if you do, please let us know.

rt.perl.org briefly down

The box hosting rt.cpan.org (yes, right now it's just on one box) was rebooted to get a bit more memory.  It took a bit longer than usual because the hardware it's running on was rebuilding its raid.

.

DNS Troubles

We had some DNS issues on Saturday morning, the 21st of February.  This may have resulted in some emails destined for perl.org addresses to have been bounced.  For details and resolution, please see here and here.

search.cpan.org mirrors

Today search.cpan.org got a tiny slightly bit faster.  We're serving the CSS and images via the PantherExpress CDN, so in particular if you are in Asia/Australia or thereabout it'll all load a fraction or two of a second faster.

For the main site and search currently users in North America are served from Montréal (hosted by Weblocal), European users are served from London (hosted by Digital Craftsmen) and everyone else (most notably South America and Asia) are served from the perl.org servers in Los Angeles.

If you are in India or Japan I'd be curious to hear which mirror is "closer" network-wise to you, eu1.develooper.com or x17.develooper.com.

By the way, the countries visiting the site the most over the last ~5 months are in order the United States (34%), Germany (6%), United Kingdom (6%), Japan (5%), Canada (4%), France (4%), Russia (3%), China (2%), Australia (2%), Italy (1.5%), Netherlands (1.5%) and so on down the list.  Google Analytics says we have visitors from "228 countries/territories".  We're delighted by every one of them, especially the three visits from Antarctica.