hello all -
this AM i woke up to discover my nice stable server kept crashing. my VPS company is blaming the use of fcgi (please see below)
so i just changed everything from FCGld back to CGI-wrapper.
things had been working fine for weeks now, but this AM we had a nasty surprise. and i notice after rebooting, it would happen again within minutes.
i am wondering if i was being "hacked" somehow, or had some denial-of-service attack that would spawn off processes.
any thoughts would be very appreciated !
/var/log/messages:
Aug 11 12:51:34 edwardsmarkf kernel: [<ffffffff81148dfb>] ? __vm_enough_memory+0x3b/0x190
Aug 11 12:52:33 edwardsmarkf kernel: Out of memory: Kill process 8111 (php-cgi) score 8 or sacrifice child
Aug 11 12:55:44 edwardsmarkf kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Aug 11 12:55:44 edwardsmarkf kernel: Out of memory: Kill process 8112 (php-cgi) score 8 or sacrifice child
Aug 11 13:08:43 edwardsmarkf kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Aug 11 13:08:43 edwardsmarkf kernel: Out of memory: Kill process 8675 (mysqld) score 9 or sacrifice child
Aug 11 13:18:45 edwardsmarkf kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Aug 11 13:19:24 edwardsmarkf kernel: Out of memory: Kill process 8109 (php-cgi) score 8 or sacrifice child
Aug 11 13:30:56 edwardsmarkf kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
etc etc
Howdy,
Well, the issue isn't FCGI vs CGI... the issue is moreso that Apache is spawning more processes than your server has RAM to handle.
That could be the result of a DoS. Lately I've seen some bots that have hammered servers pretty hard.
It could also just be a large amount of web traffic, if you were expecting a spike.
You may want to review your logs and see if a particular site on your server was under duress during that time. And if you can attribute the load increase to a particular IP, you may want to block that IP.
However, what you may want to do is to edit your Apache config, and to lower the MaxClients setting.
By default, that's usually set to 150, which is pretty high for most VPS's. You may want to try lowering it a lot... how much depends on a lot of factors, but you'd simply want to make sure that your server has enough RAM to handle whatever MaxClients is set to -- for each client, you'd have an Apache process, and a PHP process.
Other things you can do are to reduce how much RAM Apache and PHP are using. For each module that they're using, it increases the RAM that they require.
-Eric
hi eric - my MaxClients is actually set at 256 and i believe it was set like that at default.
it seems to me trying to look out for DoS IP numbers is like playing whack-a-mole.
one posting suggested i remove +ExecCGI from httpd.conf - is there a way to do this in virtualmin ?
i just manually edited the httpd.conf file and removed all the +ExecCGI references. i notice now i dont get all the php-cgi processes running anymore.
what do you think about that approach??
To find out what website is getting hammered, I usually do
And if it's actually a specific website that's getting flooded (usual suspects are attempts to brute-force Worpress or Joomla or other CMSes), Apache
mod_qoscan help mitigate the number of connections specific websites/URLs may accept per second and concurrent total.thank you locutus -
we were indeed getting hit hard by one particular website. so i just lowered my MaxClients from 256 down to 50.
what are your thoughts on eliminating the "+ExecCGI" from httpd.conf ? i notice that ever since i did that by editing the httpd.conf file, i don't see all the php-cgi's that were in memory all the time.
I might be wrong, but wouldn't removing ExecCGI from the config prevent any external code, be it through CGI or FCGId, from being called?
MaxClients is a global setting, you surely know that. :) If you need/want such limits for specific URLs/sites, the mentioned mod_qos should help. I'm using that successfully to limit the impact of dictionary attacks on admin pages of sites with CMSes. I also think that when the MaxClient limit is reached, users get 500 errors for requests. mod_qos has the advantage of (among other things) slowing down further requests more and more, instead of rejecting them.
Howdy,
Yeah, disabling ExecCGI shouldn't actually help... in general you need those php-cgi's :-)
Now, using CGI rather than FCGID would prevent them from remaining in memory after they are finished executing.
However, it sounds like in your case, what you'd need to do is gather information about what's occurring when there's a spike such as what you described.
Reviewing log files can help with that, but that's a lot more difficult to do than when a spike is actually occurring.
You could always setup some sort of monitoring to let you know when the server is under duress, and when that's the case, you could review what's going on and where the traffic is coming from and going to.
-Eric
thank you so much for the advice.
i now have this script running every 5 minutes. it basically "counts' the number of php-cgi processes, and if it exceeds a limit, it restarts apache:
! /usr/bin/bash# #
process to monitor how many apache php-cgi processes are running, and reboot if we exceed the limit#
MAX_CGI=25; DATE=$(date);
phpCgiCnt=$(ps -ef | grep /usr/bin/php-cgi | grep -v grep | wc -l) ; if [[ $phpCgiCnt -gt $MAX_CGI ]] ; then free | mail -s 'restarting apache ! ! ' mark@mark.com ; echo $DATE 'apache restart! found' $phpCgiCnt 'cgi processes' $MAX_CGI 'allowed' ; free ; /etc/rc.d/init.d/httpd restart ; free ;
else echo $DATE 'apache OK with' $phpCgiCnt 'cgi processes' $MAX_CGI 'allowed' ; fi
hmm that didnt post - will try again later. but i notice the number of php-cgi processes seems to steadily increase.
when i first reboot i have about 8 of those php-cgi processes hanging around in memory, but now i have 14.
any idea why the number would keep increasing? i would have thought that any website running cgi-wrapper would not keep php-cgi processes alive in memory.
i am tempted to just reboot apache every midnight.