So, our server is maxing out memory pretty fast. And since it is on a shared host (VPS) i think services are terminated when they reach the hard memory limit.
Anyways, looking at the proccesses it looks like this from a snapshot: Should they really take 70m ? Can i reduce this somehow?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5268 mysql 15 0 343m 79m 8272 S 0 7.8 15:29.21 mysqld
12115 page1 15 0 205m 70m 7520 S 0 6.9 0:04.12 php5-cgi
12113 page1 16 0 191m 56m 7024 S 0 5.5 0:03.14 php5-cgi
3094 page2 15 0 182m 48m 7132 S 0 4.7 0:40.71 php5-cgi
3706 page4 15 0 180m 45m 7776 S 0 4.5 3:12.19 php5-cgi
17860 page5 15 0 180m 45m 7080 S 0 4.5 1:52.96 php5-cgi
3454 page5 15 0 180m 45m 7012 S 0 4.4 0:25.02 php5-cgi
28364 page6 15 0 177m 41m 7112 S 0 4.1 0:31.82 php5-cgi
28356 page6 15 0 175m 40m 7128 S 0 4.0 0:32.77 php5-cgi
29979 page2 15 0 174m 40m 8192 S 0 3.9 5:46.05 php5-cgi
20299 page6 15 0 173m 40m 6768 S 0 3.9 0:00.64 php5-cgi
3184 page2 15 0 173m 39m 6960 S 0 3.9 0:03.89 php5-cgi
3185 page2 15 0 172m 39m 6784 S 0 3.9 0:01.78 php5-cgi
7574 page2 15 0 174m 39m 7208 S 0 3.9 0:13.02 php5-cgi
3160 page2 16 0 172m 39m 6920 S 0 3.9 0:03.40 php5-cgi
13749 page2 15 0 170m 35m 7200 S 0 3.5 0:07.47 php5-cgi
26101 page2 15 0 166m 31m 7600 S 0 3.1 3:12.07 php5-cgi
1433 root 18 0 260m 27m 19m S 0 2.7 3:32.30 apache2
30566 page 15 0 155m 19m 6880 S 0 2.0 0:00.70 php5-cgi
30574 page5 15 0 152m 19m 6808 S 0 1.9 0:00.50 php5-cgi
24040 root 18 0 80356 17m 1872 S 0 1.7 0:09.90 perl
14227 page3 15 0 148m 12m 6860 S 0 1.2 0:02.16 php5-cgi
18398 page3 15 0 148m 12m 6776 S 0 1.2 0:00.13 php5-cgi
18232 page3 15 0 148m 12m 6804 S 0 1.2 0:00.36 php5-cgi
28286 list 18 0 55784 12m 3204 S 0 1.2 0:07.41 python
19679 www-data 18 0 261m 12m 2732 S 0 1.2 0:00.08 apache2
28289 list 18 0 55456 12m 3248 S 0 1.2 0:06.99 python
28283 list 18 0 55224 11m 3160 S 0 1.1 0:07.16 python
21668 www-data 15 0 261m 11m 2160 S 0 1.1 0:00.00 apache2
28280 list 18 0 55156 11m 3172 S 0 1.1 0:06.92 python
28281 list 18 0 54956 11m 3168 S 0 1.1 0:07.30 python
21666 www-data 15 0 261m 11m 2140 S 0 1.1 0:00.00 apache2
21665 www-data 18 0 261m 11m 2108 S 0 1.1 0:00.00 apache2
21650 www-data 15 0 261m 11m 2104 S 0 1.1 0:00.01 apache2
21670 www-data 18 0 261m 11m 2172 S 0 1.1 0:00.00 apache2
21671 www-data 15 0 261m 11m 2124 S 0 1.1 0:00.00 apache2
21667 www-data 15 0 261m 11m 2156 S 0 1.1 0:00.00 apache2
I had various problems with fcgid over the years related to memory and process management... I eventually gave up and now run everything in cgi mode. It's slightly slower but whatever, still fast enough.
@nisse: Do you actually have memory issues? Have processes been killed due to out-of-memory (you'll find that in the kernel log if it happens)?
What's the output of
free -m(put it in[code][/code] tags please)?Having a few php-cgi processes with around 70 MB of RAM is rather normal, situation is the same on my server. Not sure how you can reduce its memory usage, except for turning off PHP caches if you are using one.
Well services randomly stop working... i suspected it was due to memory issues. Where exactly should i look for that in kernel log? And which log file is that? On ubuntu btw.
Now i just restarted apache2 so the memory from free -m might is not what i usually is at.
Usually there is like 100 free and rest used...
the daemon pool is just that: it will stay resident in memory for on demand usage which scales inherently over time;
after taking the advice posted above, take this in to account: depending on the manner in which you read the process output, you may be noticing one of a few different things: for example, and first off, you may be reading the output wrong.
it may be describing the entire pool, or each of its threads- not processes. or maybe even any memory it may have cached from the file system for later use. cached memory is usually temporal and not dedicated to the application, and may be reused on the fly-
also, considering it is a dynamic worker pool, it will take some configuration to get it working properly with your setup. unfortunately the default config. out of the box is designed for a somewhat powerful server, and dedicated towards one or few high traffic sites;
what does your /etc/apache2/mod-available/fcgid.conf look like? here is an example, which should scale well on a Shared Server (using mpm_event in my case..):
<IfModule mod_fcgid.c>AddHandler fcgid-script .fcgi
FcgidMaxRequestsPerProcess 512
FcgidMaxProcesses 10
FcgidProcessLifeTime 150
FcgidIOTimeout 45
FcgidIdleTimeout 45
FcgidIdleScanInterval 45
FcgidBusyTimeout 60
FcgidBusyScanInterval 30
FcgidErrorScanInterval 30
FcgidZombieScanInterval 25
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 3
FcgidMaxRequestLen 5242880
FcgidConnectTimeout 10
</IfModule>
p.s. Fail2Ban "says" its using over 1GB of memory on my box, however, i usually experience OOM conditions from things rather like ClamScan, etc.
Good Luck!
The fcgid.conf looks like this:
i would have to look at the man page for mod_fcgid, but i am fairly certain that the default values for some things are set fairly high. furthermore, these values (for example):
FcgidMinProcessesPerClass 0FcgidMaxProcessesPerClass 5
They specify the maximum number of processes spawned for each user, or "site" in your case. This, combined with the proper configuration of timeouts, and idle scans, should help terminate and rid the system of un-needed fcgi related processes and threads.
Each user can have either a Pool, or single processes, launched to handle each new request by apache2. worker pools are just a process which loads several threads, to process new requests without having to spin up a new process. each one will report the same amount of memory, but im also fairly certainly that they "share" this,
however, on ubuntu, you can do a few things to restrict the amount of memory that each process uses.
first off, php.ini specifies the maximum amount of memory each process can use. usually set to atleast 64MB; remember, that each thread does not use 64MB, each process will.
second, i am fairly certain you can restrict the amount of virtual, paged, and real memory allowed for use by each process, or thread, by using "ulimit" in /etc/default/apache2, or by adding the www-data, and site users to /etc/security/limits.conf
moreover, i think you could add "ulimit" to your init.d script for apache2 to restrict memory consumption as well.
note: if i got anything wrong here, someone can feel free to right my mistakes and interject at any time
^_^
Good luck!
the idea i presume, for a shared server, would be to limit each User, or "Site", to at most just a few Processes, which terminate rather quickly. each process can use several Threads to increase scalability in high traffic scenarios.
If each user is entitled to 2 consecutive worker pools, which also have 3 threads each:
max processes = Users * 2 (Process Count)
max classes = max processes * 3 (Threads per Process)
note: i see you have no swap file. if you are running a container on openvz or proxmox, you should install and configure a monitor server like "Monit". sometimes, especially on a shared server, fcgid time out settings Need to be low, because if offending applications are using too much burst memory, applications will be killed seemingly at random.
monit, can detect if a PID dies, and launch the application if it ever goes down. it can also restart apache2 (for example) if it starts to use too much memory, or if it begins hogging CPU. this is a MUST on a virtualized container to ensure your websites stay online
This sounds like what we are running into.
Any good URL to some guide on how to install this "monit" ?
Thanks!
Is this by any chance an OpenVZ VPS?
Do you get anything from this command?
cat /proc/user_beancountersalso
uname -aApache memory usage in OpenVZ is crazy, there is a method to get it down quite a bit, but only if you are using OpenVZ.
This is what we get:
cat /proc/user_beancounters Version: 2.5 uid resource held maxheld barrier limit failcnt 7304: kmemsize 30249144 30617957 9223372036854775807 9223372036854775807 0 lockedpages 0 0 9223372036854775807 9223372036854775807 0 privvmpages 326600 329256 9223372036854775807 9223372036854775807 0 shmpages 11115 11131 9223372036854775807 9223372036854775807 0 dummy 0 0 9223372036854775807 9223372036854775807 0 numproc 102 105 9223372036854775807 9223372036854775807 0 physpages 210109 212479 9223372036854775807 9223372036854775807 0 vmguarpages 0 0 9223372036854775807 9223372036854775807 0 oomguarpages 210895 213265 9223372036854775807 9223372036854775807 0 numtcpsock 23 23 9223372036854775807 9223372036854775807 0 numflock 206 207 9223372036854775807 9223372036854775807 0 numpty 1 1 9223372036854775807 9223372036854775807 0 numsiginfo 0 1 9223372036854775807 9223372036854775807 0 tcpsndbuf 919168 919168 9223372036854775807 9223372036854775807 0 tcprcvbuf 376832 376832 9223372036854775807 9223372036854775807 0 othersockbuf 269904 320224 9223372036854775807 9223372036854775807 0 dgramrcvbuf 0 8472 9223372036854775807 9223372036854775807 0 numothersock 191 197 9223372036854775807 9223372036854775807 0 dcachesize 1836684 1858281 9223372036854775807 9223372036854775807 0 numfile 7385 7461 9223372036854775807 9223372036854775807 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 dummy 0 0 0 0 0 numiptent 14 14 9223372036854775807 9223372036854775807 0Here is the information, I found this at IntoVPS a few years ago, it has helped tremendously.
There are cases when you may notice that the memory usage is too high inside your VPS without a specific reason.
This may happen because of how OpenVZ calculates the memory used by your applications, most visible with Java applications or other servers using threads (Apache with mpm_worker or mpm_event).
You can read more here: http://wiki.openvz.org/Stack_size
A solution as explained in the article above is to decrease the stack size since the default value is too high in almost all cases.
For example, we installed Apache with mpm_worker in a VPS with MaxClients set to 25 and the threads per child set to 25 also. While 25 concurrent requests is not much the memory usage for that Apache spiked to 300MB. That's a little too high.
We then edited /etc/init.d/httpd (on CentOS, for other distributions it may be called apache2) and added: ulimit -s 256 on a separate line in the file, at the beginning like this:
# Source function library.
. /etc/rc.d/init.d/functions
ulimit -s 128
if [ -f /etc/sysconfig/httpd ]; then
. /etc/sysconfig/httpd
fi
After restarting Apache, the total memory used went down to around 30MB, which is acceptable.
This also works for other applications like MySQL for example.
monit is a really nice alternative to some other bloated monitor software available. i've heard a few minor complaints, but in all honesty, it has One file to configure, and once running, i have never experienced any issues. used in conjunction with the webmin service monitor for email and SMS: this makes a great pair!
on ubuntu, you can install monit using:
sudo apt-get install monitthe config is located under /etc/monit/monitrc, and everything you need is already located in this file (some lines are commented)
personally, i ignore any "alert", or email functionality (substituting webmin with its SMS functionality). and the only issue i had was with its "internal server". anyways, here is some working config:
## Monit has an embedded web server which can be used to view status of## services monitored and manage services from a web interface. See the
## Monit Wiki if you want to enable SSL for the web server.
#
set httpd port 2812 and
use address localhost # only accept connection from localhost
allow localhost # allow localhost to connect to the server and
allow admin:monit # require user 'admin' with password 'monit'
allow @monit # allow users of group 'monit' to connect (rw)
ignore racking you brain over this block if you can: its primarily used by the service it self, and does not consume very much memory or constitute any additional overhead that i can tell..
(however, you may want to change "@monit" to your users primary group)
Finally, you will want to configure each Mission Critical application on your server. you do this by simply pointing Monit to the applications PID file under /var/run. if this file is deleted or returns 0, the application is considered "down", and the appropriate action is taken
check process apache with pidfile /var/run/apache2.pidstart program = "/etc/init.d/apache2 start" with timeout 45 seconds
stop program = "/etc/init.d/apache2 stop"
if cpu > 60% for 5 cycles then alert
if cpu > 80% for 10 cycles then restart
if totalmem > 512.0 MB for 15 cycles then restart
if children > 99 then restart
if loadavg(5min) greater than 25 for 15 cycles then restart
This is an example on how to monitor apache2. this will also, in turn, monitor any php-cgi processes, due to being launched by Apache2. atleast this is my understanding on how the fastcgid server is handled (and that it is killed when apache2 shuts down)
edit: ill have to look into whether or not the fastcgid server maintains its own PID, and if "totalmem" evaluates to "apache2 + all children processes". ether way, this is a good place to start, and should be done for most applications which can consume large amounts of memory and cause DOS or OOM conditions inside a virtualized container,
anyways, openVZ can be perfectly acceptable to use in a production environment, and i haven't had any problems: as long as you take the correct precautions!
Regards, - Steve