Fast Quadcore turns into a snail overnight

A fast QuadCore running CentOS 7 and Virtualmin Pro with only about 30 small websites went from very fast to a slug overnight. I have been trying to track down the error mod_fcgid: ap_pass_brigade failed in handle_request_ipc function I have searched and played around till my eyes are square and nothing is changing. It can take at least 30 to 60 seconds to load a site that was just 1 second yesterday. Rebooted Apache several times after tweaking various things but nothing is making a difference. Does anyone have any suggestions on how to figure out what the problem might be. The Apache error logs for a particular site that has submitted a complaint about speed only has these few error messages. I would expect dozens of lines of errors. So that is strange too and I am curious why the log files don't seem to be updating. To try to edit any of the virtual sites via Virtualmin takes a few minutes to get a screen to refresh. Terribly slow. The command "top" shows the CPU's are just around 0 to 1% so it is not like the server is working hard - just not responding. I just don't have a clue.

Status: 
Active

Comments

Howdy -- whenever you're seeing that issue, what is the output of the following commands:

uptime
free -m
mailq | tail -1

[root@wichita ~]# uptime 18:57:13 up 2 days, 4:43, 1 user, load average: 0.00, 0.03, 0.05 [root@wichita ~]# free -m
total used free shared buff/cache available
Mem: 7500 2830 1279 370 3389 3921
Swap: 7679 108 7571
[root@wichita ~]# mailq | tail -1
-- 2 Kbytes in 2 Requests.

Thanks for the info!

And just to clarify -- when you ran those commands, your site was experiencing that slowness issue you were describing?

Yes, the last test on the site with the original speed complaint was 31 seconds to display a webpage but sometimes it has taken even longer. The most basic WordPress site on the machine for testing took about 9 seconds to load. It is so simple of a webpage I would expect a small fraction of one second. I can't understand where the bottleneck was introduced. We have several Virtualmin Pro servers running CentOS 7.x and just one 6.x left. The 6.x is pretty much carefree. It is besides the point but 7.x has not been good to us. There is no going back, I wish there was. FTP has been the least stable of all the services. We can't keep it running on any of the machines. If we can fix it it does not last long, but that is another story. If I can't get this server to work better I will be forced to transfer the accounts to another machine and rebuild the server and then what do I do the next time? It is a solution but not a good one.

I'm not seeing any of the obvious problems there. Your resource usage all looks good.

You may want to try creating just a simple index.html page, and see how quickly that loads.

Then, create a test.php script, and give it the following contents:

<?php
 phpinfo();
?>

Does that script load quickly?

I did as you suggested and both the php and html page load instantly. I wish that explained something to me. What am I do believe now about how slow the other sites load, of which almost all of them are WordPress sites. There is a reason but it makes no sense to me. The control panel itself is just painfully slow. I thought it used a web server other than Apache which means that Apache is not necessarily the issue. Should I be expecting to load a website and then find any errors in the logfiles right afterwards? Because I don't see any logs and that makes it hard to debug anything. It is like working in the dark. And if they show up from yesterday they seem really skimpy. What would stop the log files from being parsed on a timely basis? I know, these are hard questions. Just hoping for some inspiration because I am stumped and depressed and tired of fighting these damn 7.x machines that work for months and then don't work. I might be getting too old for this game.

I was hoping your issue might just be a simple resource issue, but that doesn't appear to be the case.

The fact that the index.html and test.php files load quickly also seem to indicate that your server as a whole isn't performing slow.

Just to verify -- that WordPress site that you referred to, is that a brand new installation, with no plugins?

We've seen website plugins act up in the past, in a way that caused it to run slowly.

You should also be able to see Apache logs located in $HOME/logs/error_log.

You may also want to review any MySQL logs in /var/log.

Had a similar issue some time back. And on my server it turned out be disk problem. One of the disk in RAID was not working properly and causing problems. Though I wasnt able to find this using any tool, had to use test and try method.

In my case it would not be RAID because I have not used RAID in many years. If I did, I think I would opt for RAID 1 (mirroring) as RAID 5 is just making things a bit more complicated than I want them to be. WordPress plugins certainly could be a culprit but that would not explain why even the Virtualmin Control Panel is slow. I have moved a few of the accounts to a different machine this morning that by contrast load pages in just a second or two. It is probably on the order of 10 times the speed and yet they are both almost identical Xeon hardware. For the moment, I will just have to move more accounts if they complain about speed. And perhaps just start over with a new machine. That is admitting defeat but I don't seem to have a choice.

Yeah that is pretty strange! At the moment I'm not quite sure what the issue might be, the resource usage of your server isn't indicating a problem.

That is, your server is showing that there's still plenty of RAM and CPU to go around.

What output do you receive if you run these commands:

vmstat 1 10
dmesg | taill -50

Those will offer some additional debugging info that could possibly help us sort out what you're experiencing there.

Hopefully, the format will survive, not sure. But here is the response:

[root@wichita ~]# vmstat 1 10 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 162216 222668 0 3778828 0 0 11 7 8 9 1 0 99 0 0
0 0 162216 222528 0 3778828 0 0 0 96 335 229 0 0 100 0 0
0 0 162216 222528 0 3778828 0 0 0 0 355 222 0 0 100 0 0
1 0 162216 221784 0 3778828 0 0 0 24 374 244 0 0 100 0 0
0 0 162216 218560 0 3778828 0 0 0 0 519 412 1 0 99 0 0
0 0 162216 218560 0 3778828 0 0 0 0 363 296 0 0 100 0 0
0 0 162216 218560 0 3778828 0 0 0 0 349 226 0 0 100 0 0
0 0 162216 209276 0 3778828 0 0 0 208 1346 1854 7 0 92 1 0
0 0 162216 209168 0 3778764 0 0 0 0 319 195 0 0 100 0 0
0 0 162216 209168 0 3778764 0 0 0 0 324 207 0 0 100 0 0

[root@wichita ~]# dmesg | tail -50
[430480.886266] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187340 reply_cookie=0 error=n/a
[430480.886285] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187341 reply_cookie=0 error=n/a
[430480.886377] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187340 reply_cookie=0 error=n/a
[430480.886391] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187341 reply_cookie=0 error=n/a
[430488.893668] systemd[1]: Got notification message for unit httpd.service [430488.893672] systemd[1]: httpd.service: Got notification message from PID 23470 (READY=1, STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec) [430488.893674] systemd[1]: httpd.service: got READY=1 [430488.893676] systemd[1]: httpd.service: got STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec [430488.893718] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187342 reply_cookie=0 error=n/a
[430488.893737] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187343 reply_cookie=0 error=n/a
[430488.893832] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187342 reply_cookie=0 error=n/a
[430488.893846] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187343 reply_cookie=0 error=n/a
[430498.903265] systemd[1]: Got notification message for unit httpd.service [430498.903282] systemd[1]: httpd.service: Got notification message from PID 23470 (READY=1, STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec) [430498.903291] systemd[1]: httpd.service: got READY=1 [430498.903300] systemd[1]: httpd.service: got STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec [430498.903433] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187344 reply_cookie=0 error=n/a
[430498.903507] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187345 reply_cookie=0 error=n/a
[430498.903817] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187344 reply_cookie=0 error=n/a
[430498.903861] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187345 reply_cookie=0 error=n/a
[430504.130241] systemd[1]: Got notification message for unit systemd-journald.service [430504.130253] systemd[1]: systemd-journald.service: Got notification message from PID 554 (WATCHDOG=1) [430504.130260] systemd[1]: systemd-journald.service: got WATCHDOG=1 [430508.912732] systemd[1]: Got notification message for unit httpd.service [430508.912737] systemd[1]: httpd.service: Got notification message from PID 23470 (READY=1, STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec) [430508.912739] systemd[1]: httpd.service: got READY=1 [430508.912741] systemd[1]: httpd.service: got STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec [430508.912781] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187346 reply_cookie=0 error=n/a
[430508.912801] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187347 reply_cookie=0 error=n/a
[430508.912906] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187346 reply_cookie=0 error=n/a
[430508.912919] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187347 reply_cookie=0 error=n/a
[430508.912920] systemd[1]: Got notification message for unit systemd-logind.service [430508.912922] systemd[1]: systemd-logind.service: Got notification message from PID 759 (WATCHDOG=1) [430508.912924] systemd[1]: systemd-logind.service: got WATCHDOG=1 [430518.922711] systemd[1]: Got notification message for unit httpd.service [430518.922716] systemd[1]: httpd.service: Got notification message from PID 23470 (READY=1, STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec) [430518.922718] systemd[1]: httpd.service: got READY=1 [430518.922720] systemd[1]: httpd.service: got STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec [430518.922761] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187348 reply_cookie=0 error=n/a
[430518.922780] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187349 reply_cookie=0 error=n/a
[430518.922935] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187348 reply_cookie=0 error=n/a
[430518.922949] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187349 reply_cookie=0 error=n/a
[430528.932589] systemd[1]: Got notification message for unit httpd.service [430528.932594] systemd[1]: httpd.service: Got notification message from PID 23470 (READY=1, STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec) [430528.932596] systemd[1]: httpd.service: got READY=1 [430528.932598] systemd[1]: httpd.service: got STATUS=Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec [430528.932639] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187350 reply_cookie=0 error=n/a
[430528.932658] systemd[1]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187351 reply_cookie=0 error=n/a
[430528.932753] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187350 reply_cookie=0 error=n/a
[430528.932767] systemd-logind[759]: Got message type=signal sender=:1.0 destination=n/a object=/org/freedesktop/systemd1/unit/httpd_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=187351 reply_cookie=0 error=n/a

Yeah I'm unfortunately not seeing any obvious signs of problems there.

Your server isn't showing any resource problems. And the vmstat you just ran isn't showing any significant disk activity.

It's difficult to pinpoint what the trouble there might be...

I don't imagine it's possible the computer is on a slower connection?

It would be worth switching the patch cable and the switch that it is plugged into. I had not thought of that though it is a very long shot since it was fast with the same rack install and then just became slow. Well, I might have to give up and leave it a mystery. I can move the thirty odd accounts to another new machine in the middle of the night. The only thing I don't like about my method at the moment is each site has a dedicated IP, and the website has to be deleted on the first server before it is restored on the second. There is always a small chance the backup is not good. Otherwise, I have to give each site a new IP and change the DNS on the site and run both sites for a day or two. How do the rest of you pros prefer to migrate sites from one machine to the other?

Well, there is one difference to what I am doing now. Which is virtualmin restore-domain --source /root/backups/ --all-domains --all-features. That might save a little bit of time. As always, your the best and cPanel sucks. OK, maybe not sucks but not as good. Thanks for the tip!