Submitted by adamjedgar on Fri, 10/11/2019 - 16:14 Pro Licensee
Guys, this is driving me up the wall!
Once again, after having my system running stable for more than a month, it appears that something in a recent update has suddenly caused my system memory to spiral out of control again.
Memory usage went from a steady 21-23% to now almost 40% and fluctuating all over the place!
I do not have this problem with any other servers other than this one running Virtualmin (i also have a Centos7 system running Centos Web Panel).
How can i troubleshoot this problem?
Status:
Active
Comments
Submitted by adamjedgar on Fri, 10/11/2019 - 16:21 Pro Licensee Comment #1
Submitted by adamjedgar on Fri, 10/11/2019 - 16:50 Pro Licensee Comment #2
Submitted by andreychek on Fri, 10/11/2019 - 16:51 Comment #3
Howdy -- thanks for contacting us!
It looks like you have a good bit of RAM free, and fluctuations aren't abnormal at all. There's a lot of potential causes, though something like a bot hitting a website could definitely cause that.
Also note that we haven't issued any updates recently, any updates would have been from Debian itself.
We'd be happy to review your running processes though, and see if anything appears out of the ordinary.
What is the output of this command:
ps auxwfSubmitted by adamjedgar on Fri, 10/11/2019 - 16:53 Pro Licensee Comment #4
Submitted by adamjedgar on Fri, 10/11/2019 - 16:54 Pro Licensee Comment #5
# ps auxwfUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Sep26 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Sep26 0:11 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S Sep26 1:09 \_ [rcu_sched]
root 8 0.0 0.0 0 0 ? S Sep26 0:00 \_ [rcu_bh]
root 9 0.0 0.0 0 0 ? S Sep26 0:02 \_ [migration/0]
root 10 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [lru-add-drain]
root 11 0.0 0.0 0 0 ? S Sep26 0:02 \_ [watchdog/0]
root 12 0.0 0.0 0 0 ? S Sep26 0:00 \_ [cpuhp/0]
root 13 0.0 0.0 0 0 ? S Sep26 0:00 \_ [cpuhp/1]
root 14 0.0 0.0 0 0 ? S Sep26 0:02 \_ [watchdog/1]
root 15 0.0 0.0 0 0 ? S Sep26 0:02 \_ [migration/1]
root 16 0.0 0.0 0 0 ? S Sep26 0:12 \_ [ksoftirqd/1]
root 18 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kworker/1:0H]
root 19 0.0 0.0 0 0 ? S Sep26 0:00 \_ [kdevtmpfs]
root 20 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [netns]
root 21 0.0 0.0 0 0 ? S Sep26 0:01 \_ [khungtaskd]
root 22 0.0 0.0 0 0 ? S Sep26 0:00 \_ [oom_reaper]
root 23 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [writeback]
root 24 0.0 0.0 0 0 ? S Sep26 0:00 \_ [kcompactd0]
root 26 0.0 0.0 0 0 ? SN Sep26 0:00 \_ [ksmd]
root 27 0.0 0.0 0 0 ? SN Sep26 0:10 \_ [khugepaged]
root 28 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [crypto]
root 29 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kintegrityd]
root 30 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 31 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kblockd]
root 32 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [devfreq_wq]
root 33 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [watchdogd]
root 35 0.0 0.0 0 0 ? S Sep26 0:00 \_ [kswapd0]
root 36 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [vmstat]
root 48 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kthrotld]
root 49 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [ipv6_addrconf]
root 84 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 85 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [ata_sff]
root 86 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 87 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 88 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 90 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 92 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 117 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 118 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 119 0.0 0.0 0 0 ? S Sep26 0:00 \_ [scsi_eh_0]
root 120 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [scsi_tmf_0]
root 121 0.0 0.0 0 0 ? S Sep26 0:00 \_ [scsi_eh_1]
root 122 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [scsi_tmf_1]
root 124 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 125 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [bioset]
root 130 0.0 0.0 0 0 ? S< Sep26 0:13 \_ [kworker/1:1H]
root 148 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [kworker/u5:0]
root 158 0.0 0.0 0 0 ? S Sep26 0:50 \_ [jbd2/vda1-8]
root 159 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [ext4-rsv-conver]
root 197 0.0 0.0 0 0 ? S Sep26 0:00 \_ [kauditd]
root 207 0.0 0.0 0 0 ? S< Sep26 0:04 \_ [kworker/0:1H]
root 283 0.0 0.0 0 0 ? S Sep26 0:00 \_ [hwrng]
root 299 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [ttm_swap]
root 364 0.0 0.0 0 0 ? S< Sep26 0:00 \_ [edac-poller]
root 6665 0.0 0.0 0 0 ? S 06:44 0:00 \_ [kworker/0:1]
root 9045 0.0 0.0 0 0 ? S 07:29 0:00 \_ [kworker/u4:1]
root 13862 0.0 0.0 0 0 ? S 08:09 0:00 \_ [kworker/1:2]
root 14518 0.0 0.0 0 0 ? S 08:14 0:00 \_ [kworker/0:5]
root 17795 0.0 0.0 0 0 ? S 08:29 0:00 \_ [kworker/1:0]
root 17880 0.0 0.0 0 0 ? S 08:29 0:00 \_ [kworker/0:0]
root 17899 0.0 0.0 0 0 ? S 08:29 0:00 \_ [kworker/0:2]
root 21597 0.0 0.0 0 0 ? S 08:39 0:00 \_ [kworker/1:4]
root 21599 0.0 0.0 0 0 ? S 08:39 0:00 \_ [kworker/1:5]
root 23805 0.0 0.0 0 0 ? S 08:44 0:00 \_ [kworker/u4:2]
root 25069 0.0 0.0 0 0 ? S 08:49 0:00 \_ [kworker/0:3]
root 25083 0.0 0.0 0 0 ? S 08:49 0:00 \_ [kworker/0:4]
root 25109 0.0 0.0 0 0 ? S 08:49 0:00 \_ [kworker/1:1]
root 25638 0.0 0.0 0 0 ? S 08:54 0:00 \_ [kworker/u4:0]
root 1 0.0 0.1 204672 6752 ? Ss Sep26 5:15 /sbin/init
root 186 0.0 0.1 59352 7128 ? Ss Sep26 16:30 /lib/systemd/systemd-journald
root 231 0.0 0.0 45484 3116 ? Ss Sep26 0:01 /lib/systemd/systemd-udevd
root 319 0.0 0.1 46496 4760 ? Ss Sep26 3:45 /lib/systemd/systemd-logind
message+ 323 0.0 0.0 45116 3676 ? Ss Sep26 3:45 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 341 0.0 0.8 200720 34816 ? Ssl Sep26 0:20 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid
root 343 0.0 0.0 35924 3248 ? Ss Sep26 0:29 /usr/sbin/irqbalance --foreground
root 344 0.0 0.0 27988 2468 ? Ss Sep26 0:05 /usr/sbin/cron -f
root 345 0.0 0.0 250112 4040 ? Ssl Sep26 2:23 /usr/sbin/rsyslogd -n
root 352 0.0 0.0 4172 76 ? S< Sep26 0:31 /usr/sbin/atopacctd
root 371 0.0 0.0 8428 104 ? Ss Sep26 0:00 /sbin/rdnssd -u rdnssd -H /etc/rdnssd/merge-hook
rdnssd 373 0.0 0.0 16868 1512 ? S Sep26 0:00 \_ /sbin/rdnssd -u rdnssd -H /etc/rdnssd/merge-hook
root 420 0.0 0.1 79696 5452 ? Ss Sep26 1:07 /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 421 0.0 0.1 79696 5452 ? S Sep26 1:07 \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 422 0.0 0.1 79696 5452 ? S Sep26 1:07 \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 423 0.0 0.1 79696 5452 ? S Sep26 1:07 \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
root 424 0.0 0.1 79696 5452 ? S Sep26 1:07 \_ /usr/sbin/saslauthd -a pam -c -m /var/spool/postfix/var/run/saslauthd -r -n 5
postgrey 425 0.0 0.5 74520 20988 ? Ss Sep26 0:00 postgrey --pidfile=/var/run/postgrey.pid --daemonize --inet=10023
bind 546 0.0 0.9 289508 36828 ? Ssl Sep26 2:24 /usr/sbin/named -f -u bind
root 569 0.0 0.0 14524 1716 tty1 Ss+ Sep26 0:00 /sbin/agetty --noclear tty1 linux
root 585 0.0 0.1 69948 5672 ? Ss Sep26 0:00 /usr/sbin/sshd -D
root 612 0.0 0.0 18124 2540 ? Ss Sep26 0:01 /usr/sbin/dovecot
dovecot 628 0.0 0.0 9492 996 ? S Sep26 0:00 \_ dovecot/anvil
root 629 0.0 0.0 9624 2324 ? S Sep26 0:00 \_ dovecot/log
ntp 631 0.0 0.1 102416 4052 ? Ssl Sep26 1:01 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:111
root 647 0.0 0.0 20472 3052 ? Ss Sep26 0:00 /sbin/dhclient -4 -v -pf /run/dhclient.ens3.pid -lf /var/lib/dhcp/dhclient.ens3.leases -I -df /var/lib/dhcp/dhclient6.ens3.leases ens3
root 862 0.2 0.9 1213880 40000 ? Sl Sep26 58:47 /usr/bin/python3 /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
mysql 923 0.1 5.4 662684 219920 ? Ssl Sep26 26:39 /usr/sbin/mysqld
root 945 0.0 2.1 166432 87460 ? Ss Sep26 2:39 /usr/bin/perl -T -w /usr/sbin/spamd -d --pidfile=/var/run/spamd.pid --create-prefs --max-children 5 --helper-home-dir
root 1152 0.0 2.0 166432 82180 ? S Sep26 0:03 \_ spamd child
root 1153 0.0 2.0 166432 82180 ? S Sep26 0:02 \_ spamd child
root 1080 0.0 1.8 98800 73640 ? Ss Sep26 0:00 /usr/share/webmin/virtual-server/lookup-domain-daemon.pl
root 1090 0.0 0.1 287168 7328 ? Ssl Sep26 0:49 /usr/lib/policykit-1/polkitd --no-debug
root 1138 0.0 0.1 81636 4512 ? Ss Sep26 0:31 /usr/lib/postfix/sbin/master -w
postfix 1142 0.0 0.1 83872 6956 ? S Sep26 0:02 \_ qmgr -l -t unix -u
postfix 1196 0.0 0.2 88176 8924 ? S Sep26 0:04 \_ tlsmgr -l -t unix -u -c
postfix 1658 0.0 0.1 83704 6848 ? S Sep26 0:22 \_ anvil -l -t unix -u -c
postfix 8658 0.0 0.1 83704 6920 ? S 07:19 0:00 \_ pickup -l -t unix -u -c
postfix 18885 0.0 0.2 107292 10856 ? S 08:33 0:00 \_ smtpd -n smtp -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix 19029 0.0 0.2 107292 10480 ? S 08:33 0:00 \_ smtpd -n smtp -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix 24728 0.0 0.2 107292 11000 ? S 08:48 0:00 \_ smtpd -n submission -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix 25253 0.0 0.2 107292 10628 ? S 08:49 0:00 \_ smtpd -n submission -t inet -u -c -o stress=yes -s 2 -o smtpd_sasl_auth_enable=yes
postfix 25392 0.0 0.2 107428 10584 ? S 08:52 0:00 \_ smtpd -n smtps -t inet -u -c -o stress= -s 2 -o smtpd_sasl_auth_enable=yes -o smtpd_tls_wrappermode=yes
postfix 25393 0.0 0.1 83704 6900 ? S 08:52 0:00 \_ proxymap -t unix -u
root 1154 0.0 0.5 79828 22768 ? Ss Sep26 0:13 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 1157 0.0 0.5 85540 22476 ? Ss Sep26 0:45 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 25766 37.0 2.3 159172 95848 ? S 08:54 0:00 \_ /usr/share/webmin/shell/index.cgi
root 25767 0.0 0.0 4276 796 ? S 08:54 0:00 \_ sh -c (ps auxwf) 2>&1
root 25768 0.0 0.0 36768 2952 ? R 08:54 0:00 \_ ps auxwf
root 17265 0.0 0.2 315288 11548 ? Ssl Sep26 0:47 /usr/lib/packagekit/packagekitd
root 25396 0.0 0.0 89952 464 ? Ss Sep26 0:00 gpg-agent --homedir /root/.gnupg --use-standard-socket --daemon
root 9522 0.0 0.9 393676 40160 ? Ss Oct09 0:21 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
ajecrea+ 9524 0.0 6.5 647488 267096 ? S Oct09 2:49 \_ php-fpm: pool 152349738012302
ajecrea+ 9775 0.0 6.4 645656 261756 ? S Oct09 2:50 \_ php-fpm: pool 152349738012302
ajecrea+ 10188 0.0 6.7 657492 273828 ? S Oct09 2:46 \_ php-fpm: pool 152349738012302
ajecrea+ 17514 0.0 6.6 654660 271100 ? S Oct09 2:43 \_ php-fpm: pool 152349738012302
root 15336 0.0 0.2 25344 8416 ? S<Ls 00:00 0:06 /usr/bin/atop -a -R -w /var/log/atop/atop_20191012 600
root 5023 0.0 0.7 268868 31772 ? Ss 06:25 0:00 /usr/sbin/apache2 -k start
www-data 5067 0.0 0.6 268592 24532 ? S 06:25 0:00 \_ /usr/sbin/apache2 -k start
www-data 5068 0.0 0.9 1476876 36684 ? Sl 06:25 0:07 \_ /usr/sbin/apache2 -k start
Submitted by adamjedgar on Fri, 10/11/2019 - 16:58 Pro Licensee Comment #6
Yes i think its a Debian update that has caused this...why is the question?
Is there a way i can download system usage stats before 3 days ago, and then now? (so i can compare then to see what has caused this big increase in RAM usage?)
Submitted by adamjedgar on Fri, 10/11/2019 - 17:02 Pro Licensee Comment #7
~]# apache2ctl -MLoaded Modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
dav_module (shared)
dav_fs_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
fcgid_module (shared)
filter_module (shared)
lbmethod_byrequests_module (shared)
mime_module (shared)
mpm_event_module (shared)
negotiation_module (shared)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_fcgi_module (shared)
proxy_http_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
security2_module (shared)
setenvif_module (shared)
slotmem_shm_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
suexec_module (shared)
unique_id_module (shared)
Submitted by andreychek on Fri, 10/11/2019 - 17:15 Comment #8
Well, I'm really not seeing anything out of the ordinary there, that looks perfectly normal.
Using more memory also doesn't necessarily indicate a problem, it could just mean a service sees that more RAM is available and is doing more caching. Or it could mean something like a PHP module is using more. Those sorts of things would be tricky to track down without knowing how much RAM was being used by each service previously.
While you can see some general usage stats in the Virtualmin "System Statistics" area, it doesn't store anything that specific.
Unfortunately, unless you explicitly stored that information, there really wouldn't be anything to compare to.
However, I'm really not seeing signs of a problem there though. Sorry, but I'm not sure there's much we could do to assist.
Submitted by adamjedgar on Fri, 10/11/2019 - 18:53 Pro Licensee Comment #9
hmmm, this is interesting...i went to ajecreative virtual server, then changed php-fpm to fcgi, then back to php-fpm again (this restarted php-fpm).
Now look at my stats...
https://drive.google.com/file/d/1xj1t-dlpApHnd0byFuiqVepvXv_H3WSA/view?u...
It will be interesting to see if this stabilizes, or if it starts to gradually climb. I will update you guys again once i have some firm idea.
Submitted by paulocoghi on Tue, 10/15/2019 - 14:31 Comment #10
You can use an open source monitor tool like Netdata, that is extremely light. With it you can decide for how much time of statistics you want to preserve, for example, for the last 15 days.
As Andrey said, with this information is a lot easier to track scenarios like yours.
https://www.netdata.cloud/