Server crashed, now mysql won't start

5 posts / 0 new
Last post
#1 Tue, 04/12/2011 - 12:36
lex

Server crashed, now mysql won't start

My server just crashed, and after a reboot it won't start mysql.

I checked the log, and checked pages like http://www.felipecruz.com/repair-mysql-database.php which helped a lot.

But not enough ;)

myisamchk --silent --force */*.MYI
myisamchk: MyISAM file gci/jos_session.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file gci/jos_session.MYI
myisamchk: error: Record at pos: 335348 is not remove-marked
myisamchk: error: record delete-link-chain corrupted
myisamchk: MyISAM file hn/wp_options.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly

so I did on both those tables

myisamchk -r wp_options

and if i enter now myisamchk --silent --force /.MYI then I see nothing.

If I type start mysql (ubuntu server) it says "mysql start/running"

However, it isn't running, the sites all report no connection to the db, and in virtualmin mysql can't be started and shows as down.

I'll dive further into this, if that's possible, but this is already a cry for help at a place where I know people are willing to help, so, thanks already. All ideas are welcome!

--

edit: I'm not allowed to uplaod the error log, as I've got 1 mb of room on here, so I'll add a link now to it: http://www.penghosting.nl/temp/error.log.txt

Tue, 04/12/2011 - 12:48
andreychek

You have some pretty nasty looking errors in your error log, it looks like MySQL is segfaulting as it starts up. However, a close look shows these lines:

InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.

Something with your ibdata files and your my.cnf may have gotten out of sync (which could suggest either a recent change to the my.cnf file, or ibdata corruption).

What you may want to try is having it rebuild your various ibdata files.

Here's something you might try --

  • 'cd' into /var/lib/mysql

  • Create a tmp dir: mkdir old

  • Move the ibdata related files into it: mv ib* old

  • Now try starting up MySQL: /etc/init.d/mysql start

Does that work for you?

-Eric

Tue, 04/12/2011 - 13:20 (Reply to #2)
lex

I'm going to test that now, and it's true, I recently put a few 'top servers' as 'sub servers' and then noticed the db was still there, but now I needed the login things of the top server to get into it. So maybe it's to do with that. Thanks!!

Tue, 04/12/2011 - 13:30 (Reply to #3)
lex

Thanks Eric, it sure does!

Virtualmin still showed mysql as down (after refresh as well) but after a click on the green arrow that as well got sorted.

Problem now is the load on the server, just keeps going up and up (32 now with 'top) and I can't see anything weird (I think) in netstat.

Lot's of apaches. I'll check netstat again.

But: THANKS!

Tue, 04/12/2011 - 13:35 (Reply to #4)
lex

It's now going down to normal levels. Showing '3' now and server responding nicely. Wouldn't mind finding out what causes those peaks and what made the server crash 2 hours ago. There should be more hours in a day and more knowledge in my head.

Topic locked