All is well again

We got most things up pretty quickly and I've beaten our console server into submission again so it's ready next time we need it (going to the data center sucks).  If there are any of our services still missing, please let us know.

Fuel Pump Fail

The building where the perl.org datacenter is hosted was performing safety tests today that involved running on generator power for a few hours.  No problems were expected, as our UPS would have (and did) covered the transition.  And then a fuel pump failed in one of the generators, requiring it to be shut down, resulting in us losing power.

Several machines didn't come back up.  So if your favorite perl.org service isn't available or isn't working right today, that's why. 

Ask is on his way to the datacenter to get things sorted.  We'll update this blog as we have more information.

Blackholes!

This morning, one of our upstream sources of "is this ip a possible source of spam or not" that we use for scoring spam stopped serving real data and started returning "YES!!!!!!" to everything.  This may have caused email that already contained a weak spam signal to be marked as spam and rejected.  We've removed the bad data source, and things should be back to normal now.