MySQL table crashed again on Syslog server!
A couple of days ago we were having a downtime so I had to shutdown all my servers including Syslog server which is a php-syslog-ng server, as mentioned in my previous posts. But after it came back online no logs were collected anymore and by closer examination I found out that the "logs" table in MySQL has been marked as crashed and automatic repair didn't help either(Through mysqld log files and "check table [tablename]" through mysql client).
I used "myisamchk" tool which is used to check and repair MyISAM tables as follow:
[root@logcollector]# myisamchk -rf logs.MYI (tried the quick mode first but not helpful)
It took eight hours to check and fix the indexes!
This was the second time that my syslog-ng server went down for about 2 days because of a failure in MySQL table. The first time a sudden and unexpected shutdown brought about the table crash but this time was different. I am doubting that mysqld couldn't shutdown its databases safely and that might be because there are more than 150 devices sending their messages to this server. Checking my past rotated syslog databases I see that each month I have a database of 15 to 20 GB in size. I am thinking of some way to split my database into different parts to make it easier to handle.
Labels: Linux, Monitoring



0 Comments:
Post a Comment
<< Home