Centos buffer tiemout

There are no error messages relevant to the timeouts that I can find in any log. I checked the /var/log folder and looked at all logs that have timestamps indicating that they could have entrees correlated to the problem. Also checked http logs in each hosted domain to make sure none were causing a problem.

It just stops accepting new sockets for awhile, then starts again. Webmin hangs for a few seconds, ssh console hangs, pop3,smpt, and imap also hang.

The system is straight from the CentOS ISO with the VMPro install script a few months ago. Since the problem started, I have of course gone through many CentOS performance tuning steps.. all have failed.

Most of the sockets eventually get through since the delay clears before 60 seconds. Nagios has a default 10 second timeout. I tweaked that to 15 seconds and it didn't change the freq of failures much. I would propose that since most -eventually- get through, there are few/no error log traces.

There are only 275 users and 15 domains on this machine. The FreeBSD machine next to it has >1000 users (1 domain) with all the same basic services running - web, smtp, ftp, pop3, imap, dns but no delays.

Unfortunately, the domain on the freebsd machine cannot co-exist with the domains on the Centos machine due to the differences in default naming of users.

I've scripted your suggested commends and the next time I catch the system lagging I'll run it and attach the output.

During freeze:

running free total used free shared buffers cached Mem: 6098128 3507456 2590672 0 322436 2078856 -/+ buffers/cache: 1106164 4991964 Swap: 6094840 0 6094840

running uptime 18:26:58 up 4 days, 7:03, 1 user, load average: 4.97, 2.68, 2.09

number of connections - netstat -an|grep tcp|wc -l 65

During normal operations:

running free total used free shared buffers cached Mem: 6098128 3498608 2599520 0 322444 2079104 -/+ buffers/cache: 1097060 5001068 Swap: 6094840 0 6094840

running uptime 18:30:39 up 4 days, 7:06, 1 user, load average: 0.24, 1.51, 1.76

number of connections - netstat -an|grep tcp|wc -l 90

dmesg dump txt file attached

In the procmail log I'm getting a bunch of clamav errors - they happen with every email. I've tried to turn off clamav long ago but appeared to have failed.

    ERROR: Can't connect to clamd: No such file or directory\

I've turned off FTP and http and the delays still happen. I'm betting on this being SPAM related.... a burst of addresses come in and swap the system somehow. I've tried seriously increasing the connection capability without a cure. I'll try limiting inbound mail to 1 delivery address per connection to see if that smooths out the surges.

I have stats collection running every minute, and graphed a good correlation to the problem - jpeg attached.

It appears the load spikes every 12 to 15 minutes and is followed by a spike in received emails.

I've turned off ClamAV completely, and turned off bayes in Spamassassin to reduce load to little effect.

Reviewing detailed mail and proc log to see if there is a particular source.

(deleted excerpt from procmail log... it was a jumbled mess)

jpeg with cpu load and email per minute attached

This jpg I believe shows an iowait CPU load that is delaying the delivery of emails. When the iowait clears, the backlogged emails come through in a burst. Here is a summary from dmesg:

[root@home ~]# dmesg -c | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head

942 kjournald

736 repquota

356 pdflush

129 collectinfo

 62 lookup

 52 httpd

 49 syslogd

 48 spamd

 39 pop

 27 procmail

Haven't quite figured out what to do next to track this down.

I've tried turning off journaling by changing fstab to ext2 and rebooting. That did not work yet, as the source forgot to mention clearing the journal first and deleting the journal file before the fstab change. I should research two sources before making arbtrary changes :>) System still works same as before, but I'm hesitant to try changing ext2/3 settings again.

I've ordered a replacement hard drive and will clone the system. With a clone disk I'll try making the complete transition from ext3 to ext2. Also, the new disk may fix the problem.

I've adjusted vm.dirty_background_ratio=5 to see if that impacts anything - making pdflush work more often with smaller amount of cached writes. - period and magnitude of cycle stays the same. I'll try increasing to 20.

sidenote - I've found that in general you (i.e. me) should not just turn off something like clamav in the system settings if they are already turned on. The virtual servers (at least on my systems) do not clean themselves of the config settings. I had to re-enable clamAV in vritual min, go to each virtual directory and turn off clamAV, then disable clamAV again in VM.

To answer.... for the past 48 hours clamav has been totally turned off and there are no more log messages complaining about "could not find ClamAV". The iowait problem is still happening.

I changed the fstab settings to "noatime" and the high cpu load cycle appears to have changed. Without the atime updates there are fewer inodes to flush from the cache, and the time the system is stalled on each cycle appears to be reduced.

The best diagnostics right now are the System Statistics - CPU 1 min in VM showing the periodic nature of the CPU spikes, and top with "1" and "i" selected to show individual core iowaits and no idle processes.

Joe - I'll email you login info to the machine.

Joe, this is still a problem.

I've been troubleshooting for a long time now, and tuned the heck out of the server so that the problem is still there, but the impact is less.

There is a CPU surge every 12 min 45 seconds +/- 15 seconds.... up to 800 % use (8 cores). Top reports that the use is entirely iowait. It appears that the program using the CPU is the journal or a flush command.

If I block all incoming mail port 25 via a remote firewall/router the CPU cycling stops. If I allow incoming mail again it starts.

Today I'm going to try turning off SPAMAssassin to see if that changes the results.

Joe... problem fixed.

After adjusting every setting I could, the problem would not go away. Plugged the hard drive into a duplicate server, and the problem did not go away. So...

I installed CentOS on a new hard drive, installed VMPro via script, and restored the individual domains via backup/restore process. Once that was done, I tweaked the various settings that did not carry over. The new drive and OS worked fine...... no drop-outs at all.

I took the old disk and ran extensive diagnostics on it, and found no hardware problem. I assume that the OS/webmin/VM/Applications had been corrupted somehow. Unless you are very curious and would like a disk image to play with, I'll be wiping the old disk soon and reusing it elsewhere.

As part of the process through I did buy the 2nd copy of VM pro... so now I have a 50 domain version running on CentOS and a 10 domain version on FreeBSD.

thanks for the help... Steve