Submitted by twm on Wed, 03/05/2014 - 10:50
One of our Virtualmin based development servers is believed to have a memory leak or another software related issue.
Current Configuration:
CPU: 2 x Dual Core Xeon
RAM: 8GB DDR2 ECC
HD: Drobo B800i SAN
VIRTUALIZAION: XenServer 6.2
VirtualMin: 4.06 GPL
After a fresh reboot the server is very snappy and I can work all day however when I get to the office the following morning I have only 200MB of free RAM while CPU utilization and IO are optimal.
Any ideas?
#top
top - 11:31:03 up 19:31, 3 users, load average: 0.08, 0.10, 0.09
Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.2%us, 1.0%sy, 0.0%ni, 95.8%id, 1.9%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8028512k total, 7698416k used, 330096k free, 374736k buffers
Swap: 4095992k total, 0k used, 4095992k free, 5111276k cached
# free -m
total used free shared buffers cached
Mem: 7840 7518 322 0 365 4991
-/+ buffers/cache: 2160 5679
Swap: 3999 0 3999
# ps aux | awk '{print $2, $4, $11}' | sort -k2r | head -n 20
PID %MEM COMMAND
1318 2.6 /usr/libexec/mysqld
4900 1.4 /usr/sbin/httpd
4901 1.4 /usr/sbin/httpd
4897 1.3 /usr/sbin/httpd
4899 1.3 /usr/sbin/httpd
4903 1.3 /usr/sbin/httpd
4902 1.2 /usr/sbin/httpd
10355 1.1 /usr/sbin/httpd
1618 1.1 /usr/sbin/httpd
1620 1.1 /usr/sbin/httpd
1622 1.1 /usr/sbin/httpd
1625 1.1 /usr/sbin/httpd
1619 1.0 /usr/sbin/httpd
1623 1.0 /usr/sbin/httpd
1621 0.9 /usr/sbin/httpd
4898 0.8 /usr/sbin/httpd
10354 0.7 /usr/sbin/httpd
1624 0.6 /usr/sbin/httpd
925 0.3 /usr/sbin/namedStatus:
Active
Comments
Submitted by andreychek on Wed, 03/05/2014 - 11:19 Comment #1
Howdy -- the memory usage you're seeing there is actually very normal.
The Linux kernel uses memory as a cache, when it's not needed elsewhere... so typically, a server actually becomes much faster over time due to it's caching.
In you're "free" output, if you look at the "cached" column, there's nearly 5GB of RAM there that is cached. That RAM is essentially free, and would be used if an application requested it. But so long as an application isn't asking for it, the Linux kernel uses it to speed other processes up.
Your CPU usage looks good as well.
If your application isn't performing well, the cause isn't apparent with the information you have there, you may be seeing some sort of bottleneck in the application. You could always try benchmarking your application to search for bottlenecks.
You could also try and review what's going on within MySQL, to see if any queries are taking a long time to run.
For example, you could connect to MySQL and run "show full processlist" just to see if there are a number of existing connections, and learn what they're doing.
Submitted by twm on Wed, 03/05/2014 - 12:05 Comment #2
It's strange as it seems the system gets slower and slower throughout the day as it is used. After a reboot everything is very very snappy with the only difference being low RAM utilization .. at least from what I can see.
The server is used as our local web development box. Most projects are built on Drupal.
Do you have any recommendations for benchmarking tools?
I am currently looking at webgrind: https://code.google.com/p/webgrind/
Submitted by andreychek on Wed, 03/05/2014 - 12:17 Comment #3
I believe you that your system is slowing down.
However, it's not likely related to memory usage.
Unless your system is using a lot of swap space, RAM usage wouldn't be slowing down your server.
Also, "cached" RAM can be considered free RAM -- on your system, there's plenty of free RAM.
Your server resources actually look very good.
I suspect the issue may be related to your application itself, or maybe one or modules being used by your application.
I unfortunately don't have a suggestion for how best to profile a PHP application though, that's not something we're very familiar with.
If you're having problems with PHP application performance, you could always try asking the Virtualmin community for some thoughts on how they handle issues like that... you could post a message in the Forums, describe your application setup, and ask about how other folks track down those sorts of issues.
One other thought that comes to mind -- if the issue is at all related to database optimization, Drupal does support memcache, which can greatly speed up database queries. You could always look into adding memcache into the mix.