collectinfo.pl makes %100 CPU

Hi, Today when i saw websites are too slow. I checked virtualmin panel and i saw %100 CPU. Problem is /usr/libexec/webmin/virtual-server/collectinfo.pl. When i checked it in ssh:

[root@sunucu ~]# /usr/libexec/webmin/virtual-server/collectinfo.pl
Can't locate ./virtual-server-lib.pl in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/libexec/webmin/virtual-server/collectinfo.pl line 7.

Well.. I reboot server but problem still exists. I didnt change anything in last months.. only updated with yum update.

Operating system CentOS Linux 7.4.1708
Webmin version 1.860
Usermin version 1.720
Virtualmin version 6.01
Theme version Authentic Theme 18.49-9
Status: 
Closed (duplicate)

Comments

Body: View changes
Kernel and CPU
Linux 3.10.0-693.5.2.el7.x86_64 on x86_64

Yesterday i did an update and probably something broke in that. Here is yum.log:

Dec 02 09:53:39 Updated: systemd-libs-219-42.el7_4.4.x86_64
Dec 02 09:53:39 Updated: 1:grub2-common-2.02-0.65.el7.centos.2.noarch
Dec 02 09:53:40 Installed: 1:grub2-tools-minimal-2.02-0.65.el7.centos.2.x86_64
Dec 02 09:53:41 Installed: 1:grub2-tools-2.02-0.65.el7.centos.2.x86_64
Dec 02 09:54:00 Updated: webmin-1.860-1.noarch
Dec 02 09:54:01 Updated: kmod-20-15.el7_4.4.x86_64
Dec 02 09:54:02 Updated: openssh-7.4p1-13.el7_4.x86_64
Dec 02 09:54:02 Updated: 1:mariadb101u-config-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:03 Updated: 1:mariadb101u-common-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:03 Updated: 1:mariadb101u-libs-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:04 Updated: 1:mariadb101u-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:05 Updated: 1:mariadb101u-errmsg-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:07 Installed: 1:grub2-tools-extra-2.02-0.65.el7.centos.2.x86_64
Dec 02 09:54:07 Updated: 1:grub2-pc-modules-2.02-0.65.el7.centos.2.noarch
Dec 02 09:54:08 Updated: 1:grub2-pc-2.02-0.65.el7.centos.2.x86_64
Dec 02 09:54:08 Updated: nss-3.28.4-15.el7_4.x86_64
Dec 02 09:54:09 Updated: nss-sysinit-3.28.4-15.el7_4.x86_64
Dec 02 09:54:09 Updated: libcurl-7.29.0-42.el7_4.1.x86_64
Dec 02 09:54:10 Updated: php-common-5.4.16-43.el7_4.x86_64
Dec 02 09:54:10 Updated: php-pdo-5.4.16-43.el7_4.x86_64
Dec 02 09:54:11 Updated: php-cli-5.4.16-43.el7_4.x86_64
Dec 02 09:54:12 Updated: kernel-tools-libs-3.10.0-693.5.2.el7.x86_64
Dec 02 09:54:12 Updated: python2-psutil-2.2.1-3.el7.x86_64
Dec 02 09:54:13 Updated: iptables-1.4.21-18.2.el7_4.x86_64
Dec 02 09:54:14 Updated: kmod-libs-20-15.el7_4.4.x86_64
Dec 02 09:54:16 Updated: systemd-219-42.el7_4.4.x86_64
Dec 02 09:54:17 Updated: systemd-sysv-219-42.el7_4.4.x86_64
Dec 02 09:54:17 Installed: libspf2-1.2.10-5.20150405gitd57d79fd.el7.x86_64
Dec 02 09:54:18 Updated: selinux-policy-3.13.1-166.el7_4.5.noarch
Dec 02 09:54:19 Updated: python2-cryptography-1.7.2-1.el7_4.1.x86_64
Dec 02 09:54:19 Updated: python2-acme-0.19.0-1.el7.noarch
Dec 02 09:54:20 Updated: python2-certbot-0.19.0-1.el7.noarch
Dec 02 09:54:22 Updated: certbot-0.19.0-1.el7.noarch
Dec 02 09:54:29 Updated: selinux-policy-targeted-3.13.1-166.el7_4.5.noarch
Dec 02 09:54:30 Installed: milter-greylist-4.5.16-3.el7.x86_64
Dec 02 09:54:31 Updated: php-fpm-5.4.16-43.el7_4.x86_64
Dec 02 09:54:31 Updated: systemd-python-219-42.el7_4.4.x86_64
Dec 02 09:54:32 Updated: openssh-server-7.4p1-13.el7_4.x86_64
Dec 02 09:54:33 Updated: iptables-services-1.4.21-18.2.el7_4.x86_64
Dec 02 09:54:33 Updated: kexec-tools-2.0.14-17.2.el7.x86_64
Dec 02 09:54:36 Updated: 1:mariadb101u-server-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:38 Updated: proftpd-1.3.5e-3.el7.x86_64
Dec 02 09:54:39 Updated: kernel-tools-3.10.0-693.5.2.el7.x86_64
Dec 02 09:54:40 Updated: php-5.4.16-43.el7_4.x86_64
Dec 02 09:54:40 Updated: php-mysql-5.4.16-43.el7_4.x86_64
Dec 02 09:54:40 Updated: php-soap-5.4.16-43.el7_4.x86_64
Dec 02 09:54:41 Updated: php-xmlrpc-5.4.16-43.el7_4.x86_64
Dec 02 09:54:41 Updated: php-process-5.4.16-43.el7_4.x86_64
Dec 02 09:54:42 Updated: php-xml-5.4.16-43.el7_4.x86_64
Dec 02 09:54:42 Updated: php-gd-5.4.16-43.el7_4.x86_64
Dec 02 09:54:43 Updated: php-mbstring-5.4.16-43.el7_4.x86_64
Dec 02 09:54:43 Updated: curl-7.29.0-42.el7_4.1.x86_64
Dec 02 09:54:44 Updated: nss-tools-3.28.4-15.el7_4.x86_64
Dec 02 09:54:44 Installed: 1:grub2-2.02-0.65.el7.centos.2.x86_64
Dec 02 09:54:45 Updated: 1:mariadb101u-devel-10.1.28-1.ius.centos7.x86_64
Dec 02 09:54:45 Updated: openssh-clients-7.4p1-13.el7_4.x86_64
Dec 02 09:54:53 Installed: kernel-3.10.0-693.5.2.el7.x86_64
Dec 02 09:54:53 Updated: virtualmin-config-6.0.21-1.noarch
Dec 02 09:55:08 Updated: 2:wbm-virtual-server-6.01.gpl-3.noarch
Dec 02 09:55:09 Updated: wget-1.14-15.el7_4.1.x86_64
Dec 02 09:55:10 Installed: p7zip-16.02-2.el7.x86_64
Dec 02 07:55:11 Updated: tzdata-2017c-1.el7.noarch
Dec 02 07:55:11 Installed: perl-IO-Tty-1.10-11.el7.x86_64
Dec 02 07:55:12 Updated: procmail-3.22-36.el7_4.1.x86_64
Dec 02 07:55:13 Updated: python-gobject-base-3.22.0-1.el7_4.1.x86_64
Dec 02 07:55:13 Installed: bzip2-1.0.6-13.el7.x86_64
Dec 02 07:55:14 Updated: python-perf-3.10.0-693.5.2.el7.x86_64
Dec 02 07:55:14 Updated: apr-1.4.8-3.el7_4.1.x86_64
Dec 02 07:55:14 Updated: epel-release-7-11.noarch
Dec 02 07:55:16 Updated: kernel-headers-3.10.0-693.5.2.el7.x86_64
Dec 02 07:55:39 Erased: 1:grub2-tools-efi-2.02-0.64.el7.centos.x86_64

Howdy -- you could try configuring that to run less frequently by going into System Settings -> Virtualmin Config -> Status Collection, and there, try setting "Interval between status collection job runs" to a different value... perhaps "60", or you could even try disabling it altogether.

Isnt this need for letsencrypt cert renewals and other things? Is it really ok to disable it? Well.. I am seeing new files with problem.. like this:

[root@sunucu ~]# /usr/libexec/webmin/proc/index_cpu.cgi
Can't locate ./proc-lib.pl in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/libexec/webmin/proc/index_cpu.cgi line 4.

You may just want to start by lowering how frequently it runs. If it continues to cause problems, we can look into disabling it. It's largely used for stats generation and whatnot, though we'll verify that's still the case with Jamie before doing so.

Where are you seeing that error above though?

That particular script would normally be called from within Webmin, which should in turn be setting up the proper paths for the proc-lib.pl file it's referring to.

Well, i didnt see any error before the updates. But when i see 100% CPU with websites lagging and checked the panel, i saw 9-10 /usr/libexec/webmin/virtual-server/collectinfo.pl execution is basically ruining server.

The collectinfo.pl CPU problem will be solved when the 1.870 (or later) release of Webmin comes out.

jbatun21564065's picture
Submitted by jbatun21564065 on Fri, 12/22/2017 - 13:36

I regret to inform, that in following this thread I made the update and the system to the latest available version, and it did not work.

Process ID Owner UCP Command 24503 root 37.0% /usr/libexec/webmin/proc/index_cpu.cgi

Operating system CentOS Linux 7.4.1708 Perl version 5.016003 Path to Perl /usr/bin/perl BIND version 9.9 Postfix version 2.10.1 Mail injection command /usr/lib/sendmail -t Apache version 2.4.6 PHP versions 5.6.25, 7.0.10 Webalizer version 2.23-08 Logrotate version 3.8.6 MySQL version 5.5.56-MariaDB ProFTPD version 1.35 SpamAssassin version 3.4.0 ClamAV version 0.99.2

Are you seeing collectinfo.pl using a lot of CPU, or index_cpu.cgi ?

I'm having trouble with collectinfo.pl after upgrading webmin from 1.955 to 1.960 and virtualmin from 6.12 to 6.13

I have defined "Interval between status collection job runs" as 15 min. collectinfo.pl constantly running and never finished. Wen I look at processes, there are 4 or more collectinfo.pl which eats most of the cpu resource

Ilia's picture
Submitted by Ilia on Mon, 11/09/2020 - 06:03

I'm having trouble with collectinfo.pl after upgrading webmin from 1.955 to 1.960 and virtualmin from 6.12 to 6.13

Sorry about that. It will be fixed in Webmin 1.962, meanwhile, please update the following files and restart Webmin.

For RHEL:

curl https://raw.githubusercontent.com/webmin/webmin/master/webmin/webmin-lib.pl -o /usr/libexec/webmin/webmin/webmin-lib.pl
curl https://raw.githubusercontent.com/webmin/webmin/master/system-status/system-status-lib.pl -o /usr/libexec/webmin/system-status/system-status-lib.pl

For Debain/Ubuntu:

curl https://raw.githubusercontent.com/webmin/webmin/master/webmin/webmin-lib.pl -o /usr/share/webmin/webmin/webmin-lib.pl
curl https://raw.githubusercontent.com/webmin/webmin/master/system-status/system-status-lib.pl -o /usr/share/webmin/system-status/system-status-lib.pl

Restart Webmin:

/etc/webmin/restart

I have the same issue on Ubunti 18.04, Webmin 1.960, Virtualmin 6.13

with /usr/share/webmin/virtual-server/collectinfo.pl

I tried the suggestion from @Ilia , but it did not work completely. I had to reboot the whole system and then one single collectinfo.pl is running with 30% CPU instead of two collectinfo.pl with 60-70% each.

I have the same issue since last update to webmin 1.960 and virtualmin 6.13. My system load increases quite rapidly after reboot, with lots of i/o disk writes. I updated webmin-lib.pl and system-status-lib.pl as @Ilia suggested, but unfortunately without any effect on this issue. Below are the top 15 processes running quite soon after reboot. On other Virtualmin installs I don't seem to have this issue.

For now I have disabled the Status collection in Virtualmin altogether to prevent the system from running out of resources.

CPU load averages:  10.62 (1 mins) , 9.06 (5 mins) , 6.59 (15 mins)
CPU type:   Intel Xeon E312xx (Sandy Bridge) (2399 MHz) , 2 cores
ID  Owner   CPU Command
9098    root    5.5 %   /usr/share/webmin/authentic-theme/index.cgi
1   root    0.8 %   /sbin/init
8120    root    0.5 %   /usr/share/webmin/virtual-server/collectinfo.pl
9007    root    0.4 %   /usr/bin/python3 /usr/bin/cloud-init modules --mode=final
4599    root    0.3 %   /usr/share/webmin/virtual-server/collectinfo.pl
7218    root    0.3 %   /usr/share/webmin/virtual-server/collectinfo.pl
234 root    0.2 %   /lib/systemd/systemd-journald
715 bind    0.2 %   /usr/sbin/named -u bind
823 root    0.2 %   /usr/bin/python3 /usr/bin/fail2ban-server --async -b -s /var/run/fail2ban/fail2b ...
1517    root    0.2 %   /usr/share/webmin/virtual-server/collectinfo.pl
3559    root    0.2 %   /usr/share/webmin/virtual-server/collectinfo.pl
6112    root    0.2 %   /usr/share/webmin/virtual-server/collectinfo.pl
7217    root    0.1 %   /usr/share/webmin/webmincron/webmincron.pl
8119    root    0.1 %   /usr/share/webmin/webmincron/webmincron.pl
9071    root    0.1 %   /usr/bin/php-cgi -v

Hey, @Ilia thank you for suggestion. I waited for Webmin update 1.962 and just updated it but no luck. I updated and restarted the server itself but nothing has changed,

I waited more than 30 min. for collectinfo.pl running and it never terminates itself (and probably new instances of collectinfo.pl will be multiply if I wait more 15 mins.)

I'm helplessly deactivating log collecting for now but need to have a workaround for this issue.

Thank you for any helps.

I had the same issue and updated to Webmin version 1.962. Before that I had set the interval to 30 minutes. Now, I think the issue is solved. Can't see collectinfo.pl in the process list.

This is still an issue. OS: Ubuntu 16.x

After current virtualmin/webmin update was applied, these issues started up. Previously version didn't have these issues. Status collection is disabled yet it still runs.

Collectinfo.pl & webmincron.pl both stack up consuming all CPU/RAM. This is a code issue as the issue wasn't present prior.

root 28917 0.0 1.0 155232 88728 ? S 09:49 0:00 /usr/share/webmin/webmincron/webmincron.pl

root 28918 53.5 2.7 310880 221792 ? S 09:49 1:09 /usr/share/webmin/virtual-server/collectinfo.pl

Even though status collection is disabled, it appears to kick off everytime the webmincron.pl runs.

What else was changed in the recent release that would be contributing to these issues?

strace shows a loop --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29157, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- close(6) = 0 wait4(29157, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 29157 lstat("/usr/share/webmin/proc", {st_mode=S_IFDIR|0755, st_size=24576, ...}) = 0 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 6 lseek(6, 0, SEEK_CUR) = 0 fstat(6, {st_mode=S_IFREG|0644, st_size=3344, ...}) = 0 mmap(NULL, 3344, PROT_READ, MAP_SHARED, 6, 0) = 0x7fc41d6fb000 lseek(6, 3344, SEEK_SET) = 3344 munmap(0x7fc41d6fb000, 3344) = 0 close(6) = 0 open("/etc/shadow", O_RDONLY|O_CLOEXEC) = 6 lseek(6, 0, SEEK_CUR) = 0 fstat(6, {st_mode=S_IFREG|0640, st_size=3205, ...}) = 0 mmap(NULL, 3205, PROT_READ, MAP_SHARED, 6, 0) = 0x7fc41d6fb000 lseek(6, 3205, SEEK_SET) = 3205 munmap(0x7fc41d6fb000, 3205) = 0 close(6) = 0 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 6 lseek(6, 0, SEEK_CUR) = 0 fstat(6, {st_mode=S_IFREG|0644, st_size=3344, ...}) = 0 mmap(NULL, 3344, PROT_READ, MAP_SHARED, 6, 0) = 0x7fc41d6fb000 lseek(6, 3344, SEEK_SET) = 3344 munmap(0x7fc41d6fb000, 3344) = 0 close(6) = 0 open("/etc/shadow", O_RDONLY|O_CLOEXEC) = 6 lseek(6, 0, SEEK_CUR) = 0 fstat(6, {st_mode=S_IFREG|0640, st_size=3205, ...}) = 0 mmap(NULL, 3205, PROT_READ, MAP_SHARED, 6, 0) = 0x7fc41d6fb000 lseek(6, 3205, SEEK_SET) = 3205 munmap(0x7fc41d6fb000, 3205) = 0 close(6) = 0 pipe([6, 8]) = 0 pipe([9, 10]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc41d6ef9d0) = 29174 close(10) = 0 close(8) = 0 read(9, "", 4) = 0 close(9) = 0 ioctl(6, TCGETS, 0x7ffe83698b50) = -1 ENOTTY (Inappropriate ioctl for device) lseek(6, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 read(6, "", 8192) = 0 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29174, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---

Ilia's picture
Submitted by Ilia on Sat, 11/21/2020 - 12:33

What Webmin version you're seeing the problem on?

Hello @Ilia, we are having this issue too on a Debian system with webmin 1.962, I've applied the fix you supply but there is no effect.

I had to disable background collection for now to fix this for now.

All the packages are updated to the latest available on your debian stretch repositories, also the system packages are up to date.

Ilia's picture
Submitted by Ilia on Wed, 11/25/2020 - 04:28

Hello @Ilia, we are having this issue too on a Debian system with webmin 1.962

Installing Webmin 1.962 would be enough.

I've applied the fix you supply but there is no effect.

It really must not be happening with latest Webmin 1.962. I assume simple Webmin restart will solve this problem, considering it wasn't an issue before installing Webmin 1.960?

The problem appeared when we updated webmin to the latest version (1.962) and it wasn't happening before.

We had to reboot the system several times due to this problem, so restarting webmin is not fixing anything.

Ilia's picture
Submitted by Ilia on Wed, 11/25/2020 - 12:53

What is the output of top command? Is there a way to login to the system and have a look?

Hi @Ilia this issue still persist for us.

Here is the screenshot of top snipboard.io/8AuXdB.jpg We defined collection for 15 mins, so after 15 mins collectinfo.pl duplicating itself and so every 15 mins goes on +1 instance: snipboard.io/6PKT7X.jpg

We are using debian buster, all webmin, virtualmin and other system packages are updated to the latest versions. If it helps, using awstats and system has ~150 domains/subdomains. php 5.6 and all 7.x versions installed and all of the domains using FPM with apache (no nginx installed)

Ilia's picture
Submitted by Ilia on Thu, 01/07/2021 - 18:04

Replied to your message on the forum.

Thanks.

Ilia's picture
Submitted by Ilia on Fri, 01/08/2021 - 05:04

I must have made a mistake in one of the patches earlier. I just don't understand what it is yet. Looking into it..

Anyone having the same issue, please jump in here.