May 20, 2004

mozillazine today ran into an interesting mysql problem, one I haven't really seen before and wonder what caused it.

For some reason, all the tables became corrupted. There were no obvious signs of corruption: the filesystem appeared to be only 60% full, there was not significant load on the box, nor any obvious defunct processes or signs that could lead to such an occurrence.

I tried the conventional SQL statement driven repair techniques: ANALYZE TABLE and REPAIR TABLE, but I wasn't having any joy. I decided to stop mysqld and enter into the more familiar myisamchk territory. Ran a safe-recover on the table, and repeatedly was hit with perror 28, (out of disk). A friend inside MySQL AB reminded me to check all possibilities: "the error code never lies".

Well, I was stumped. I had 27GB free, and the paths/permissions seemed just fine. So I copied the original corrupt table to another server, and ran the myisamchk - only to find that it worked perfectly. It turns out that the mozillazine server had 27GB of binary logs hanging around; but why would that be problematic? The server had a whole lot of space left to fill!

In the end, removing the binary logs (there really isn't a need for them just yet) enabled myisamchk to finish repairing the rest of the tables. I still don't know how the server felt it was out of disk space.

NB: I am aware of chroots etc, but none were employed. It seems like a perfectly unrestricted server.

mzine-mysql