Whoops. We had one outage earlier that dragged down a server (OOM killer went nuts). I rebooted it and of course it won't come back up. Either the (raid'ed) disk got corrupted, the bios ran out of battery and lost some critical settings or some variation of that. I can't get into the BIOS to make it netboot remotely.
In any case, I'm in Europe and Robert might have the day off (imagine!), so all the services (including www.pool.ntp.org and www.perl.org) running on that box might be down for the day.
Enjoy the holidays. :-)
Update - Robert got it fixed within a couple of hours. Yay. (ps. SELinux is really annoying).
Murphy's law doesn't take any vacations! :-)
ReplyDeleteHappy holidays guys,
GFK's
Hi GFK,
ReplyDeleteYeah, indeed.
I'm working on improving our monitoring a bit -- an early warning could easily have prevented the first failure that started the snowball.
Happy holidays to you too - now get off the computer! :-)
- ask