Linux-Noob Forums
linux-noob.com DOWNTIME ! - Printable Version

+- Linux-Noob Forums (https://www.linux-noob.com/forums)
+-- Forum: General Stuff (https://www.linux-noob.com/forums/forum-4.html)
+--- Forum: Site News (https://www.linux-noob.com/forums/forum-66.html)
+--- Thread: linux-noob.com DOWNTIME ! (/thread-2036.html)

linux-noob.com DOWNTIME ! - anyweb - 2006-04-15


hi guys

 

last night (Friday April 14th 2006) the site experienced a meltdown (the filesystem had a major hiccup)

 

 

Quote:because of the file system problems, and because i used a backup of the site from April 9th 2006, some members posts/details are lost from the last week or so, apologies
i first noticed it when i checked on the site this morning, and instead of the normal website, there was mysql errors, which pointed to 'read_only' .... 

Quote:mySQL query error: SELECT id, member_id, running_time, location FROM ibf_sessions WHERE id='331641a6b33c5cec3bf0d4bc6f9bbaa6' AND browser='Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 Firefox/1.0.7' AND ip_address='81.229.174.200' 

SQL error: Incorrect key file for table 'ibf_sessions'; try to repair it

SQL error code:

Date: Saturday 15th of April 2006 09:20:30 AM
 

 

 

i ssh'd in and sure enough there were strange things happening, if i tried to create a file (touch test.txt) it would generate an error, read_only filesystem.

 



Code:
touch: cannot touch `test.txt': Read-only file system




 

 

This was bad, so i immediately ftp'd over the latest backup to the backup server, and rebooted the linux-noob.com box, after hundreds of 'inode errrors' the system finally cam back up, but after some quick testing i could see it was borked, mysql wasnt working and files were missing left right and center

 

luckily i had a backup web server ready (well nearly ready), installed with fcr4. So I compiled apache/mysql and php and then restored the last backup from April 9th

 

so, now the site is running on the backup server, please test it and tell me what is/is not working

 

it'll take a few days to iron out the leaks, and i'll get the other webserver reinstalled.

 

and yes, it's probably the last time i'll use LVM as a file system !! when something goes wrong, the whole system takes a nose dive

 

:(

 

sorry for the downtime !

 

cheers

anyweb



linux-noob.com DOWNTIME ! - hybrid - 2006-04-16

Hats off to anyweb for getting LN back up so quickly :)


linux-noob.com DOWNTIME ! - anyweb - 2006-04-16


thanks hybrid

 

just doing my job,

 

i really want linux-noob.com to be availble to all 24/7

 

so when there's a problem, i'll fix it

 

cheers

 

anyweb