kernel panic, mysqld crashed, some data lost.

[anyweb]

today at 2am approx, the linux-noob.com server had a kernel panic. I discovered this when I tried to browse the site this morning, I couldn't access it at all via http or ssh. Ping worked however. So I went to the server and hooked up a CRT and I could see the kernel panic details (some big page fault...)

 

after a restart dmesg was full of errors (missing inodes and such)

 

after the restart mysqld was hanging at 100% and linux-noob.com (this site) was non-functional (timed out)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2600 mysql 18 0 125m 25m 3764 S 100 5.0 27:25.50 mysqld

 

I attempted to backup the current database before attempting to restore it from the last known good backup (from last sunday, as I have a cron job to backup everything once a week)

 

and the backup attempt failed

 

root@www bin]# ./mysqldump -u root -p forums >> forums_export.sqlEnter password:

mysqldump: Got error: 1016: Can't open file: 'ibf_sessions.MYI' (errno: 145) when using LOCK TABLES

 

because of this, I decided to forgo 3 days data (mon/tue/wed/) and attempted a restore from the last known good backup, that worked just fine and the site is back up and running again.

 

because of this failure I will now put in place two things,

 

daily mysqld backup and I will replace this server with a newer one (shortly).

 

cheers

anyweb

[znx]

You could have tried using the inbuilt repair commands of MySQL:

 

Code:

optimize table ibf_sessions
repair table ibf_sessions

 

Unfortunately now that we have gone back the way and reopened it, then we will end up in a different sort of loss .. hehe :P So maybe now what we can do is compare your old dump against a new dump and then manually reinsert the differences (question though is what sort of issues would that cause!)

 

Oh well .. lets just get on with it, I'm certain the loss is minor.

[anyweb]

well as the new dump is replaced with the last backup (from sunday) their is effectively no new dump, i did attempt to backup (today) before restoring from sunday but that failed as i documented above.

 

so, it was three days data of posts from linux-noob.com only, the other databases were not affected.

 

cheers

 

anyweb