Tuesday, April 30, 2013
April Showers bring... Network Outages
It hasn't rained in weeks near the perl.org datacenter, but April has ended with a network outage anyway. We're currently experiencing a lack of IP connectivity to the main perl.org server farm. Many of our websites are unreachable and email will not be delivered until it comes back.
Update 10:22pm: We're back up! 2h 9m of downtime. Cause appears to be something in the facility's interconnect. Details still being figured out.
Update 11:12pm: We're still bouncing up and down a bit (with too much down in there.) Suspecting a bad cable or port in the interconnect.
Update the next day: Total downtime 3 hours, 36 minutes. The cause was isolated to a failed cable in the facility cross-connect. Thanks to the awesome techs at IX2 and Phyber for working with the facility to isolate and solve the problem.