Thursday, July 17, 2014

7/18/2014: Scheduled Maintenance -- Moving Day!

More details afterwards, but most perl.org services will be unavailable for most of the day on Friday, July 18th 2014.  We're moving!  DNS will still work, but anything running on one of our machines (many webservers and mailservers, etc) will be unavailable.  We'll post updates here.

export TZ=America/Los_Angeles

Updates:

  • 8:48AM - the first batch of machines shut down and powered off cleanly, which is a good omen. We're in transit to the datacenters at high rates of speed and the unracking will begin when we get there. 
  • 10:02AM 7/19/2014 - apologies for not posting any updates during the day yesterday, we were busy with actually doing the move.  Current state is that we're having trouble getting one of the key vm-hosting machines to boot, and the VMs on it include key servers.  We'll keep working on this this afternoon.
  • 2:00PM 7/19/2014 - the last machine is up and happy again.  everything should be working fine now.

Thursday, April 10, 2014

Heartbleed updates for rt.perl.org and rt.cpan.org

Both rt.cpan.org and rt.perl.org were running vulnerable versions of openssl.  They were updated on Tuesday April 8th.  At that time, the cert for rt.cpan.org was regenerated.  I'm still waiting for an updated cert for rt.perl.org.

Update 2014-04-16
A new cert for rt.perl.org has been deployed.

Tuesday, January 14, 2014

CPAN master mirror down for maintenance

The CPAN master rsync mirror is temporarily down. We expect it back around 06-08:00UTC. Another hard drive failed and then it went downhill from there while replacing it. Whee.

Update: Back since 8am UTC. Drives replaced, partition tables fixed, raid rebuilt, configuration files updated, initramfs updated, KVM firmware updated, cursing subsided.

Wednesday, January 8, 2014

Multiple Hard Drive Faliure


We've lost several of the drives from the machine that hosts PAUSE.  It's offline while we work to replace the drives and convince it to come back up.

Update 1/8/2013 5:49pm EST:  No good news so far.  We may need to restore from a backup.   <distraction>Did you hear that github was down?</distraction>

Update 1/10/2013 1:26am EST:  We're working to get PAUSE back up as soon as we can and apologize for any inconvenience this outage may be causing you.  It is taking a little longer than normal due to circumstances that are limiting our availability this week.  (i.e. bad timing for a failure.)

Update 1/11/2013 12:04pm EST: PAUSE is back up!  Thanks for your patience!

Friday, December 27, 2013

2013 Year End Maintenance

Happy Holidays!

We're doing some invasive maintenance and year-end cleanup down at the datacenter today (12/27/2013).  You may notice some service interruptions and glitches on perl.org, pm.org, cpan.org, and other hosted sites.

  One of the things we're doing:
    dd if=/dev/zero of=/dev/sda bs=1M &

Update 1:35pm:
  The maintenance completed without incident (and went very smoothly.  Almost too smoothly.)  We've been monitoring the systems for two hours, and everything seems stable and happy.

Thursday, October 24, 2013

rt.perl.org down for maintenance (Complete)

We're now hard at work on rt.perl.org.  We'll update this post when it's back up and ready to use.

2000 GMT - https://rt.perl.org is now back and upgraded to a new version of RT.

If you encounter problems, please report them to the perlbug-admin email address on the footer of every rt.perl.org page.

Tuesday, October 22, 2013

rt.perl.org maintenance 2013-10-24

We'll be upgrading https://rt.perl.org to the latest version of Request Tracker 4.0 starting at 1500 GMT on Thursday the 24th of October.  During this time, rt.perl.org will be inaccessible via the web and mail to RT will be queued for later delivery.  We'll blog when the upgrade starts and update the blog post as needed during the upgrade.