Everything is green again

Just got back from the datacenter. Everything was running again about 60-90 minutes ago.

Since the last reboot the server in question had more disks installed. The disks are split across two 3ware raid controllers (one with the new disks and one with the old). The bios decided to try booting off the controller with the new disks – of course there's no boot record on those so it just stalled.

The serial console was working, but for some reason it'd not show me anything until it had skipped past the "press button to enter bios setup" window. Robert could get into the bios remotely, but he was too lazy to check the settings so downtown I went. Grrh.

file server reboot

We have one particular server that's hosting many of our databases, NFS for our internal software distributions etc etc. We reboot it a couple of times a year to get kernel updates and such. One of those times is in a minute (so a short outage of some services will occur). Crossed fingers it'll come up alright, we don't want to go to the datacenter right now. :-)

update: eek, it's not coming back. crap.

Overload!

You may have noticed some trouble accessing some perl.org services this morning. Our network link is being saturated, because one of the DNS services we host is getting hammered. We've shut down the server on our side, but the other side is still sending. Mad props to InterNAP for being helpful in diagnosing and putting workarounds in place to get us stable again.