linux-noob.com DOWNTIME !

[anyweb]

hi guys

 

last night (Friday April 14th 2006) the site experienced a meltdown (the filesystem had a major hiccup)

 

 

because of the file system problems, and because i used a backup of the site from April 9th 2006, some members posts/details are lost from the last week or so, apologies

i first noticed it when i checked on the site this morning, and instead of the normal website, there was mysql errors, which pointed to 'read_only' .... 

mySQL query error: SELECT id, member_id, running_time, location FROM ibf_sessions WHERE id='331641a6b33c5cec3bf0d4bc6f9bbaa6' AND browser='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7' AND ip_address='81.229.174.200' 

SQL error: Incorrect key file for table 'ibf_sessions'; try to repair it

SQL error code:

Date: Saturday 15th of April 2006 09:20:30 AM

 

 

 

i ssh'd in and sure enough there were strange things happening, if i tried to create a file (touch test.txt) it would generate an error, read_only filesystem.

 

Code:

touch: cannot touch `test.txt': Read-only file system

 

 

This was bad, so i immediately ftp'd over the latest backup to the backup server, and rebooted the linux-noob.com box, after hundreds of 'inode errrors' the system finally cam back up, but after some quick testing i could see it was borked, mysql wasnt working and files were missing left right and center

 

luckily i had a backup web server ready (well nearly ready), installed with fcr4. So I compiled apache/mysql and php and then restored the last backup from April 9th

 

so, now the site is running on the backup server, please test it and tell me what is/is not working

 

it'll take a few days to iron out the leaks, and i'll get the other webserver reinstalled.

 

and yes, it's probably the last time i'll use LVM as a file system !! when something goes wrong, the whole system takes a nose dive

 

:(

 

sorry for the downtime !

 

cheers

anyweb

[hybrid]

Hats off to anyweb for getting LN back up so quickly :)

[anyweb]

thanks hybrid

 

just doing my job,

 

i really want linux-noob.com to be availble to all 24/7

 

so when there's a problem, i'll fix it

 

cheers

 

anyweb