MySQL on our main database server (we were running an old-ish 4.1.x version) crashed and corrupted some (unrelated to the crash, I think) InnoDB tables. We did run it with some vaguely aggressive settings, but it shouldn't lead to corruption (only losing a few updates). Grrh!
Speaking of the database server, AMD are giving us at least one of their servers; we don't really know too much about it except that a package is supposed to arrive at my house. One of our planned uses is a new database server which should speed up RT and several other services.
From: Robert Spier
Subject: Perlbug Email Down, Web Interface Unstable
Date: December 6, 2005 9:52:20 PM PST
Dear p5p, p6i,
We're having some database issues.
In order to make my life easier when I try and recover from them, I've disabled incoming email to the RT instance. (It will be queued up and
delivered later.) You may also find the web interface to be unstable
as the database decides to disappear mid-stream.
I will be working on this tomorrow, and hope to have everything back
to normal within 24 hours.
(Details: it appears that somehow, we've made innodb unhappy.
Tomorrow, I will be doing a lot of dumps and restores to try and
straighten it out.)
update 1am PST: All seems well again. Unless it goes bonkers again during the night Robert is going to turn processing of the RT emails back on in the morning. - ask