Tuesday, April 30, 2013

April Showers bring... Network Outages


It hasn't rained in weeks near the perl.org datacenter, but April has ended with a network outage anyway.  We're currently experiencing a lack of IP connectivity to the main perl.org server farm.  Many of our websites are unreachable and email will not be delivered until it comes back.

Update 10:22pm:  We're back up!  2h 9m of downtime.  Cause appears to be something in the facility's interconnect.  Details still being figured out.
Update 11:12pm:  We're still bouncing up and down a bit (with too much down in there.)  Suspecting a bad cable or port in the interconnect.
Update the next day: Total downtime 3 hours, 36 minutes.  The cause was isolated to a failed cable in the facility cross-connect.  Thanks to the awesome techs at IX2 and Phyber for working with the facility to isolate and solve the problem.

1 comment:

  1. Seems to be a layer 2 long distance interconnection or redundancy design fail.

    ReplyDelete