My Virtualmin GPL server recently completely freaked out while attempting to access some of its Web pages I received 500 Internal Server Errors. After ruling out network issues, it appears the problem was that the server's CPU and Memory usage were so crazy that it would no longer respond to any network connections even ssh log in attempts would TCP timeout.
I created a ticket with my hosting company, and the sysadmin on their end was able to ssh into the server using the hosting provider's private network. There he determined that the problem was tons of clamscan process all trying to run at the same time. Check out the attached graph from their monitoring software. Max system load was 94.8! How could there possibly be 94.8 runnable processes at the same time!!!
I had no idea that clamscan was even running on the system. I didn't think the server checked SPAMs for virusus, but apparently it tries to.
The hosting sysadmin disabled clamscan by setting its permissions of 000. Now there are Permission denied errors in procmail's log file.
I have no clue how to debug this mess. Please help me any way you can. Also, could someone knowledgeable please explain Virtualmin GPL's mail system, and how and where clamscan fits into it.
I can't debug this problem, because I don't know enough information about it yet.
Thanks,
Dave.
Below are some snippets from some log files. I can reply attaching bigger chunks if needed, but I'd rather not due to leaking domain names and IP address and email address, and all of that crap.
procmail.log:
From oGKVfbY575@ahahe.com Wed Jul 3 03:28:52 2013
Subject: [SPAM] The 50 Best Foods for Weight Loss
Folder: /home/someuser/Maildir/.spam/new/1372840164.5970_0.somedom 14946
Time:1372840164 From:oGKVfbY575@ahahe.com To:d2e711c7@anotherdomain.com User:someuser Size:14998 Dest:/home/mybadmin/Maildir/.spam/new/1372840164.5970_0.mydomain.com Mode:Spam
sh: /usr/bin/clamscan: Permission denied
messages (Around the time of the insane loadavg, it's full of OOM Killer logs.)
Jul 3 02:18:29 web4 kernel: controller invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jul 3 02:18:30 web4 kernel: controller cpuset=/ mems_allowed=0
Jul 3 02:18:44 web4 kernel: Pid: 8758, comm: controller Not tainted 2.6.32-358.11.1.el6.x86_64 #1
Jul 3 02:18:44 web4 kernel: Call Trace:
Jul 3 02:18:44 web4 kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jul 3 02:18:44 web4 kernel: [] ? dump_header+0x90/0x1b0
Jul 3 02:18:44 web4 kernel: [] ? __delayacct_freepages_end+0x2e/0x30
Jul 3 02:18:44 web4 kernel: [] ? security_real_capable_noaudit+0x3c/0x70
Jul 3 02:18:44 web4 kernel: [] ? oom_kill_process+0x82/0x2a0
Jul 3 02:18:44 web4 kernel: [] ? select_bad_process+0xe1/0x120
Jul 3 02:18:44 web4 kernel: [] ? out_of_memory+0x220/0x3c0
Jul 3 02:18:44 web4 kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jul 3 02:18:44 web4 kernel: [] ? alloc_pages_current+0xaa/0x110
Jul 3 02:18:44 web4 kernel: [] ? __page_cache_alloc+0x87/0x90
Jul 3 02:18:44 web4 kernel: [] ? find_get_page+0x1e/0xa0
Jul 3 02:18:44 web4 kernel: [] ? filemap_fault+0x1a7/0x500
Jul 3 02:18:44 web4 kernel: [] ? __do_fault+0x54/0x530
Jul 3 02:18:44 web4 kernel: [] ? handle_pte_fault+0xf7/0xb50
Jul 3 02:18:44 web4 kernel: [] ? __ip_local_out+0x9f/0xb0
Jul 3 02:18:44 web4 kernel: [] ? ip_local_out+0x25/0x30
Jul 3 02:18:44 web4 kernel: [] ? ip_queue_xmit+0x190/0x420
Jul 3 02:18:44 web4 kernel: [] ? copy_user_generic+0xe/0x20
Jul 3 02:18:44 web4 kernel: [] ? handle_mm_fault+0x23a/0x310
Jul 3 02:18:44 web4 kernel: [] ? __do_page_fault+0x139/0x480
Jul 3 02:18:44 web4 kernel: [] ? wait_consider_task+0x9d/0xb20
Jul 3 02:18:44 web4 kernel: [] ? read_tsc+0x9/0x20
Jul 3 02:18:44 web4 kernel: [] ? ktime_get_ts+0xb1/0xf0
Jul 3 02:18:44 web4 kernel: [] ? poll_select_copy_remaining+0xf8/0x150
Jul 3 02:18:44 web4 kernel: [] ? do_page_fault+0x3e/0xa0
Jul 3 02:18:44 web4 kernel: [] ? page_fault+0x25/0x30
Jul 3 02:18:44 web4 kernel: Mem-Info:
Jul 3 02:18:44 web4 kernel: Node 0 DMA per-cpu:
Jul 3 02:18:44 web4 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 4: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 5: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 6: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 7: hi: 0, btch: 1 usd: 0
Jul 3 02:18:44 web4 kernel: Node 0 DMA32 per-cpu:
Jul 3 02:18:44 web4 kernel: CPU 0: hi: 186, btch: 31 usd: 34
Jul 3 02:18:44 web4 kernel: CPU 1: hi: 186, btch: 31 usd: 30
Jul 3 02:18:44 web4 kernel: CPU 2: hi: 186, btch: 31 usd: 5
Jul 3 02:18:44 web4 kernel: CPU 3: hi: 186, btch: 31 usd: 3
Jul 3 02:18:44 web4 kernel: CPU 4: hi: 186, btch: 31 usd: 9
Jul 3 02:18:44 web4 kernel: CPU 5: hi: 186, btch: 31 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 6: hi: 186, btch: 31 usd: 0
Jul 3 02:18:44 web4 kernel: CPU 7: hi: 186, btch: 31 usd: 2
Jul 3 02:18:44 web4 kernel: active_anon:310818 inactive_anon:104950 isolated_anon:1120
Jul 3 02:18:44 web4 kernel: active_file:363 inactive_file:568 isolated_file:0
Jul 3 02:18:44 web4 kernel: unevictable:1 dirty:3 writeback:176 unstable:0
Jul 3 02:18:44 web4 kernel: free:13204 slab_reclaimable:3226 slab_unreclaimable:13473
Jul 3 02:18:44 web4 kernel: mapped:327 shmem:26 pagetables:18014 bounce:0
Jul 3 02:18:44 web4 kernel: Node 0 DMA free:8264kB min:336kB low:420kB high:504kB active_anon:1372kB inactive_anon:5676kB active_file:12kB inactive_file:60kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15268kB mlocked:0kB dirty:0kB writeback:4kB mapped:12kB shmem:4kB slab_reclaimable:28kB slab_unreclaimable:124kB kernel_stack:0kB pagetables:92kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:218 all_unreclaimable? no
Jul 3 02:18:44 web4 kernel: lowmem_reserve[]: 0 1982 1982 1982
Jul 3 02:18:44 web4 kernel: Node 0 DMA32 free:44856kB min:44716kB low:55892kB high:67072kB active_anon:1241900kB inactive_anon:413868kB active_file:1440kB inactive_file:2212kB unevictable:4kB isolated(anon):4480kB isolated(file):0kB present:2030100kB mlocked:4kB dirty:12kB writeback:700kB mapped:1296kB shmem:100kB slab_reclaimable:12876kB slab_unreclaimable:53768kB kernel_stack:3696kB pagetables:71964kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul 3 02:18:44 web4 kernel: lowmem_reserve[]: 0 0 0 0
Jul 3 02:18:44 web4 kernel: Node 0 DMA: 6*4kB 10*8kB 16*16kB 5*32kB 3*64kB 3*128kB 2*256kB 3*512kB 1*1024kB 2*2048kB 0*4096kB = 8264kB
Jul 3 02:18:44 web4 kernel: Node 0 DMA32: 1528*4kB 840*8kB 628*16kB 324*32kB 67*64kB 14*128kB 8*256kB 5*512kB 1*1024kB 0*2048kB 0*4096kB = 44960kB
Jul 3 02:18:44 web4 kernel: 25022 total pagecache pages
Jul 3 02:18:44 web4 kernel: 24007 pages in swap cache
Jul 3 02:18:44 web4 kernel: Swap cache stats: add 4307521, delete 4283514, find 142406403/142858653
Jul 3 02:18:44 web4 kernel: Free swap = 4kB
Jul 3 02:18:44 web4 kernel: Total swap = 1048568kB
Jul 3 02:18:44 web4 kernel: 522224 pages RAM
Jul 3 02:18:44 web4 kernel: 47365 pages reserved
Jul 3 02:18:44 web4 kernel: 31665 pages shared
Jul 3 02:18:44 web4 kernel: 453364 pages non-shared
Jul 3 02:18:44 web4 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Jul 3 02:18:44 web4 kernel: [ 556] 0 556 2716 1 0 -17 -1000 udevd
Jul 3 02:18:44 web4 kernel: [ 2401] 0 2401 2660 1 0 -17 -1000 udevd
Jul 3 02:18:44 web4 kernel: [ 2402] 0 2402 2715 1 1 -17 -1000 udevd
Jul 3 02:18:44 web4 kernel: [ 2583] 0 2583 1539 2 0 0 0 portreserve
Jul 3 02:18:44 web4 kernel: [ 2590] 0 2590 62367 182 3 0 0 rsyslogd
Jul 3 02:18:44 web4 kernel: [ 2644] 0 2644 2707 94 0 0 0 irqbalance
Jul 3 02:18:44 web4 kernel: [ 8228] 81 8228 5383 2 0 0 0 dbus-daemon
Jul 3 02:18:44 web4 kernel: [ 8257] 0 8257 1019 1 1 0 0 acpid
Jul 3 02:18:44 web4 kernel: [ 8266] 68 8266 6340 136 1 0 0 hald
Jul 3 02:18:44 web4 kernel: [ 8267] 0 8267 4526 2 0 0 0 hald-runner
Jul 3 02:18:44 web4 kernel: [ 8295] 0 8295 5055 2 1 0 0 hald-addon-inpu
Jul 3 02:18:44 web4 kernel: [ 8306] 68 8306 4451 2 0 0 0 hald-addon-acpi
Jul 3 02:18:44 web4 kernel: [ 8323] 0 8323 16029 1 0 -17 -1000 sshd
Jul 3 02:18:44 web4 kernel: [ 8331] 38 8331 7540 74 0 0 0 ntpd
Jul 3 02:18:44 web4 kernel: [ 8367] 0 8367 27050 2 2 0 0 mysqld_safe
Jul 3 02:18:44 web4 kernel: [ 8543] 0 8543 4814 2 5 0 0 dovecot
Jul 3 02:18:44 web4 kernel: [ 8544] 97 8544 3243 2 1 0 0 anvil
Jul 3 02:18:44 web4 kernel: [ 8546] 0 8546 3276 2 3 0 0 log
Jul 3 02:18:45 web4 kernel: [ 8555] 0 8555 16602 2 1 0 0 saslauthd
Jul 3 02:18:45 web4 kernel: [ 8556] 0 8556 16602 2 0 0 0 saslauthd
Jul 3 02:18:45 web4 kernel: [ 8557] 0 8557 16602 2 4 0 0 saslauthd
Jul 3 02:18:45 web4 kernel: [ 8558] 0 8558 16602 2 0 0 0 saslauthd
Jul 3 02:18:45 web4 kernel: [ 8559] 0 8559 16602 2 1 0 0 saslauthd
Jul 3 02:18:45 web4 kernel: [ 8635] 0 8635 19682 57 1 0 0 master
Jul 3 02:18:45 web4 kernel: [ 8642] 89 8642 19816 170 0 0 0 qmgr
Jul 3 02:18:45 web4 kernel: [ 8663] 0 8663 37546 41 1 0 0 proftpd
Jul 3 02:18:45 web4 kernel: [ 8685] 0 8685 27543 2 0 0 0 abrtd
Jul 3 02:18:45 web4 kernel: [ 8693] 0 8693 27015 47 1 0 0 abrt-dump-oops
Jul 3 02:18:45 web4 kernel: [ 8701] 0 8701 85462 280 6 0 0 httpd
Jul 3 02:18:45 web4 kernel: [ 8710] 0 8710 29308 57 4 0 0 crond
Jul 3 02:18:45 web4 kernel: [ 8721] 0 8721 5363 1 1 0 0 atd
Jul 3 02:18:45 web4 kernel: [ 8731] 0 8731 25230 7 1 0 0 rhnsd
Jul 3 02:18:45 web4 kernel: [ 8739] 0 8739 25971 2 0 0 0 rhsmcertd
Jul 3 02:18:45 web4 kernel: [ 8752] 0 8752 1604 12 1 0 0 nimbus
Jul 3 02:18:45 web4 kernel: [ 8758] 0 8758 2228 88 5 0 0 controller
Jul 3 02:18:45 web4 kernel: [ 8773] 89 8773 19701 72 1 0 0 tlsmgr
Jul 3 02:18:45 web4 kernel: [ 8776] 0 8776 21085 102 1 0 0 spooler
Jul 3 02:18:45 web4 kernel: [ 8780] 0 8780 2251 67 4 0 0 hdb
Jul 3 02:18:45 web4 kernel: [ 8791] 0 8791 3076 113 4 0 0 cdm
Jul 3 02:18:45 web4 kernel: [ 8812] 0 8812 2982 76 2 0 0 processes
Jul 3 02:18:45 web4 kernel: [ 8820] 0 8820 23375 134 1 0 0 miniserv.pl
Jul 3 02:18:45 web4 kernel: [ 8839] 0 8839 23500 296 0 0 0 miniserv.pl
Jul 3 02:18:45 web4 kernel: [ 8850] 0 8850 1015 2 0 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8852] 0 8852 1015 2 3 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8854] 0 8854 1015 2 7 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8856] 0 8856 1015 2 3 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8858] 0 8858 1015 2 6 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8860] 0 8860 1015 2 0 0 0 mingetty
Jul 3 02:18:45 web4 kernel: [ 8863] 0 8863 1019 2 0 0 0 agetty
Jul 3 02:18:45 web4 kernel: [ 9127] 0 9127 23299 58 0 -17 -1000 auditd
Jul 3 02:18:45 web4 kernel: [ 3094] 0 3094 48304 386 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [15547] 542 15547 55052 16 0 0 0 php-cgi
Jul 3 02:18:45 web4 kernel: [19869] 549 19869 54951 2 0 0 0 php-cgi
Jul 3 02:18:45 web4 kernel: [20597] 557 20597 54924 2 4 0 0 php-cgi
Jul 3 02:18:45 web4 kernel: [ 7082] 0 7082 9814 2 2 0 0 ssl-params
Jul 3 02:18:45 web4 kernel: [16296] 0 16296 35029 6 2 0 0 crond
Jul 3 02:18:45 web4 kernel: [16298] 0 16298 39793 2 0 0 0 backup.pl
Jul 3 02:18:45 web4 kernel: [16330] 0 16330 38845 756 0 0 0 lfd
Jul 3 02:18:45 web4 kernel: [19366] 0 19366 19777 15 0 0 0 local
Jul 3 02:18:45 web4 kernel: [19463] 0 19463 2307 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [19464] 0 19464 2307 1 4 0 0 sh
Jul 3 02:18:45 web4 kernel: [19465] 0 19465 7010 58 0 0 0 tar
Jul 3 02:18:45 web4 kernel: [19466] 0 19466 1074 75 4 0 0 gzip
Jul 3 02:18:45 web4 kernel: [19467] 0 19467 1024 10 0 0 0 cat
Jul 3 02:18:45 web4 kernel: [20085] 0 20085 19777 2 0 0 0 local
Jul 3 02:18:45 web4 kernel: [21402] 0 21402 19777 18 0 0 0 local
Jul 3 02:18:45 web4 kernel: [21591] 0 21591 28736 895 1 0 0 miniserv.pl
Jul 3 02:18:45 web4 kernel: [21775] 0 21775 36300 30 1 0 0 miniserv.pl
Jul 3 02:18:45 web4 kernel: [21816] 0 21816 2833 2 0 0 0 sh
Jul 3 02:18:45 web4 kernel: [21822] 0 21822 18905 250 0 0 0 rpm
Jul 3 02:18:45 web4 kernel: [21901] 0 21901 39557 229 0 0 0 miniserv.pl
Jul 3 02:18:45 web4 kernel: [21923] 0 21923 19777 115 0 0 0 local
Jul 3 02:18:45 web4 kernel: [22057] 0 22057 2833 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [22060] 0 22060 18862 748 4 0 0 rpm
Jul 3 02:18:45 web4 kernel: [22256] 557 22256 2307 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [22266] 557 22266 41732 11725 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [22383] 0 22383 19777 114 0 0 0 local
Jul 3 02:18:45 web4 kernel: [22414] 48 22414 85495 362 2 0 0 httpd
Jul 3 02:18:45 web4 kernel: [22660] 0 22660 19777 11 0 0 0 local
Jul 3 02:18:45 web4 kernel: [22725] 0 22725 19777 115 1 0 0 local
Jul 3 02:18:45 web4 kernel: [22980] 0 22980 19777 27 4 0 0 local
Jul 3 02:18:45 web4 kernel: [23014] 557 23014 2307 2 4 0 0 sh
Jul 3 02:18:45 web4 kernel: [23015] 557 23015 57619 13078 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [23237] 0 23237 19777 20 0 0 0 local
Jul 3 02:18:45 web4 kernel: [23386] 0 23386 19777 114 0 0 0 local
Jul 3 02:18:45 web4 kernel: [23429] 89 23429 19702 85 5 0 0 pickup
Jul 3 02:18:45 web4 kernel: [23605] 0 23605 19777 21 0 0 0 local
Jul 3 02:18:45 web4 kernel: [23947] 0 23947 38996 440 5 0 0 lfd
Jul 3 02:18:45 web4 kernel: [24014] 48 24014 85495 359 3 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24081] 48 24081 85495 387 2 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24082] 0 24082 35029 6 2 0 0 crond
Jul 3 02:18:45 web4 kernel: [24137] 0 24137 34458 5826 0 0 0 monitor.pl
Jul 3 02:18:45 web4 kernel: [24200] 0 24200 19777 17 0 0 0 local
Jul 3 02:18:45 web4 kernel: [24267] 48 24267 85499 358 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24304] 48 24304 85495 361 1 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24450] 27 24450 343008 697 0 0 0 mysqld
Jul 3 02:18:45 web4 kernel: [24509] 48 24509 85495 365 3 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24894] 0 24894 19777 73 1 0 0 local
Jul 3 02:18:45 web4 kernel: [24905] 0 24905 19777 118 0 0 0 local
Jul 3 02:18:45 web4 kernel: [24950] 48 24950 85528 374 1 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24952] 48 24952 85495 357 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24955] 48 24955 85495 357 4 0 0 httpd
Jul 3 02:18:45 web4 kernel: [24973] 48 24973 85495 350 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25108] 48 25108 85495 389 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25139] 0 25139 82029 2864 0 0 0 rhn_check
Jul 3 02:18:45 web4 kernel: [25159] 89 25159 19872 4 0 0 0 cleanup
Jul 3 02:18:45 web4 kernel: [25226] 89 25226 19755 108 4 0 0 trivial-rewrite
Jul 3 02:18:45 web4 kernel: [25243] 48 25243 85528 401 2 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25244] 48 25244 85495 355 1 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25271] 606 25271 2307 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [25294] 606 25294 56915 22255 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25308] 0 25308 19777 2 4 0 0 local
Jul 3 02:18:45 web4 kernel: [25342] 48 25342 85495 363 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25369] 0 25369 35029 9 0 0 0 crond
Jul 3 02:18:45 web4 kernel: [25392] 48 25392 85495 347 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25393] 0 25393 34474 4180 0 0 0 backup.pl
Jul 3 02:18:45 web4 kernel: [25410] 636 25410 2205 15 1 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25435] 89 25435 19871 141 1 0 0 cleanup
Jul 3 02:18:45 web4 kernel: [25441] 549 25441 2205 2 0 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25445] 557 25445 2205 47 1 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25453] 549 25453 2205 39 3 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25475] 549 25475 6392 2 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25476] 549 25476 2307 2 6 0 0 sh
Jul 3 02:18:45 web4 kernel: [25477] 549 25477 57618 21448 4 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25480] 513 25480 2205 2 3 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25511] 513 25511 6392 12 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25536] 0 25536 4324 6 2 0 0 anacron
Jul 3 02:18:45 web4 kernel: [25537] 513 25537 2307 6 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [25538] 0 25538 23948 105 0 0 0 sshd
Jul 3 02:18:45 web4 kernel: [25540] 513 25540 57651 38995 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25573] 557 25573 2205 48 1 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25585] 536 25585 2205 26 1 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25590] 568 25590 2205 28 0 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25602] 0 25602 27116 144 3 0 0 bash
Jul 3 02:18:45 web4 kernel: [25608] 536 25608 2205 28 0 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25633] 536 25633 6392 2 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25634] 536 25634 2307 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [25635] 536 25635 57652 48145 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25648] 536 25648 6392 6 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25660] 568 25660 6392 40 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25667] 536 25667 2307 2 7 0 0 sh
Jul 3 02:18:45 web4 kernel: [25670] 536 25670 42888 33779 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25677] 568 25677 2307 2 0 0 0 sh
Jul 3 02:18:45 web4 kernel: [25678] 568 25678 42761 34550 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25720] 48 25720 85495 381 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25732] 48 25732 85496 377 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25736] 48 25736 85495 373 4 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25753] 48 25753 85495 382 2 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25754] 48 25754 85495 376 2 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25755] 48 25755 85496 401 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25759] 48 25759 85495 398 5 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25762] 48 25762 85495 385 0 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25851] 513 25851 2205 32 0 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25886] 48 25886 85495 378 4 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25889] 48 25889 85495 368 1 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25891] 89 25891 24099 526 4 0 0 smtpd
Jul 3 02:18:45 web4 kernel: [25901] 0 25901 35029 78 1 0 0 crond
Jul 3 02:18:45 web4 kernel: [25902] 513 25902 6392 93 0 0 0 clam-wrapper.pl
Jul 3 02:18:45 web4 kernel: [25903] 89 25903 19700 248 2 0 0 anvil
Jul 3 02:18:45 web4 kernel: [25904] 513 25904 2307 37 0 0 0 sh
Jul 3 02:18:45 web4 kernel: [25905] 513 25905 25794 20638 0 0 0 clamscan
Jul 3 02:18:45 web4 kernel: [25906] 0 25906 32509 7803 4 0 0 monitor.pl
Jul 3 02:18:45 web4 kernel: [25917] 89 25917 19873 420 1 0 0 cleanup
Jul 3 02:18:45 web4 kernel: [25924] 48 25924 85495 377 3 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25959] 636 25959 2205 15 0 0 0 procmail
Jul 3 02:18:45 web4 kernel: [25960] 636 25960 22364 5941 0 0 0 spamassassin
Jul 3 02:18:45 web4 kernel: [25961] 48 25961 85495 393 1 0 0 httpd
Jul 3 02:18:45 web4 kernel: [25962] 0 25962 19777 384 3 0 0 local
Jul 3 02:18:45 web4 kernel: [26006] 48 26006 85495 389 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26007] 48 26007 85495 407 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26013] 89 26013 19873 447 0 0 0 cleanup
Jul 3 02:18:48 web4 kernel: [26015] 89 26015 24099 549 6 0 0 smtpd
Jul 3 02:18:48 web4 kernel: [26036] 557 26036 6392 171 0 0 0 clam-wrapper.pl
Jul 3 02:18:48 web4 kernel: [26053] 557 26053 2307 119 7 0 0 sh
Jul 3 02:18:48 web4 kernel: [26054] 557 26054 30180 24347 6 0 0 clamscan
Jul 3 02:18:48 web4 kernel: [26067] 568 26067 2205 119 0 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26068] 89 26068 19753 328 4 0 0 smtp
Jul 3 02:18:48 web4 kernel: [26079] 549 26079 2205 28 5 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26080] 549 26080 24476 8068 0 0 0 spamassassin
Jul 3 02:18:48 web4 kernel: [26082] 0 26082 2205 116 3 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26087] 0 26087 2205 37 4 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26088] 0 26088 28918 19054 0 0 0 lookup-domain.p
Jul 3 02:18:48 web4 kernel: [26092] 568 26092 6392 172 4 0 0 clam-wrapper.pl
Jul 3 02:18:48 web4 kernel: [26096] 557 26096 6392 171 0 0 0 clam-wrapper.pl
Jul 3 02:18:48 web4 kernel: [26101] 568 26101 2307 120 0 0 0 sh
Jul 3 02:18:48 web4 kernel: [26102] 557 26102 2307 119 5 0 0 sh
Jul 3 02:18:48 web4 kernel: [26106] 89 26106 19753 328 0 0 0 smtp
Jul 3 02:18:48 web4 kernel: [26107] 568 26107 27649 22480 0 0 0 clamscan
Jul 3 02:18:48 web4 kernel: [26108] 48 26108 85462 409 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26112] 89 26112 19711 316 4 0 0 bounce
Jul 3 02:18:48 web4 kernel: [26114] 557 26114 13813 9057 0 0 0 clamscan
Jul 3 02:18:48 web4 kernel: [26115] 0 26115 2307 120 4 0 0 sh
Jul 3 02:18:48 web4 kernel: [26116] 0 26116 35029 168 2 0 0 crond
Jul 3 02:18:48 web4 kernel: [26118] 48 26118 85462 410 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26120] 48 26120 85462 386 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26123] 48 26123 85462 386 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26125] 0 26125 3342 133 2 0 0 ps
Jul 3 02:18:48 web4 kernel: [26129] 48 26129 85462 386 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26130] 48 26130 85462 392 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26131] 48 26131 85462 386 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26133] 48 26133 85462 387 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26134] 48 26134 85462 386 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26135] 48 26135 85462 391 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26136] 0 26136 14288 5460 0 0 0 monitor.pl
Jul 3 02:18:48 web4 kernel: [26147] 0 26147 26399 5344 0 0 0 miniserv.pl
Jul 3 02:18:48 web4 kernel: [26160] 606 26160 2205 119 0 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26167] 48 26167 85495 391 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26168] 48 26168 85462 382 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26169] 48 26169 85462 349 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26170] 48 26170 85495 390 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26171] 48 26171 85462 372 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26172] 48 26172 85462 395 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26173] 48 26173 85495 416 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26174] 48 26174 85462 388 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26180] 606 26180 6392 172 0 0 0 clam-wrapper.pl
Jul 3 02:18:48 web4 kernel: [26183] 48 26183 85462 349 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26184] 606 26184 2307 120 1 0 0 sh
Jul 3 02:18:48 web4 kernel: [26185] 606 26185 10550 5990 0 0 0 clamscan
Jul 3 02:18:48 web4 kernel: [26188] 0 26188 28901 7882 0 0 0 miniserv.pl
Jul 3 02:18:48 web4 kernel: [26189] 48 26189 85462 402 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26190] 48 26190 85462 382 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26191] 48 26191 85462 411 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26219] 48 26219 85462 388 3 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26220] 48 26220 85462 387 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26225] 89 26225 19711 312 0 0 0 bounce
Jul 3 02:18:48 web4 kernel: [26226] 48 26226 85462 389 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26227] 48 26227 85462 400 1 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26229] 48 26229 85462 383 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26230] 48 26230 85462 380 2 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26231] 48 26231 85462 384 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26232] 48 26232 85462 372 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26233] 48 26233 85462 378 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26239] 89 26239 24066 531 0 0 0 smtpd
Jul 3 02:18:48 web4 kernel: [26241] 0 26241 19777 372 5 0 0 local
Jul 3 02:18:48 web4 kernel: [26242] 0 26242 2205 115 1 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26243] 606 26243 2205 120 1 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26249] 0 26249 2205 115 0 0 0 procmail
Jul 3 02:18:48 web4 kernel: [26258] 0 26258 19147 185 0 0 0 sendmail
Jul 3 02:18:48 web4 kernel: [26260] 48 26260 85462 380 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26261] 568 26261 4791 37 2 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26262] 48 26262 85462 348 4 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26263] 48 26263 85462 383 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26265] 626 26265 16000 96 1 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26266] 626 26266 4791 38 5 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26267] 626 26267 5435 38 0 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26268] 568 26268 5435 38 1 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26269] 626 26269 16000 97 7 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26270] 626 26270 16000 96 5 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26271] 568 26271 4791 37 2 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26272] 626 26272 4791 37 6 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26273] 568 26273 5435 39 5 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26274] 606 26274 4791 38 2 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26275] 626 26275 4791 37 7 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26276] 568 26276 16000 98 2 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26277] 626 26277 4791 38 3 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26278] 568 26278 16000 96 3 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26279] 626 26279 4791 37 2 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26280] 606 26280 4791 38 3 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26281] 568 26281 4791 38 6 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26282] 553 26282 4791 38 6 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26283] 568 26283 4791 38 7 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26284] 568 26284 4791 38 5 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26285] 568 26285 16000 99 0 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26286] 568 26286 5435 38 4 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26287] 626 26287 16000 96 5 0 0 php-cgi
Jul 3 02:18:48 web4 kernel: [26290] 0 26290 38845 2085 0 0 0 lfd
Jul 3 02:18:48 web4 kernel: [26292] 0 26292 2307 126 0 0 0 sh
Jul 3 02:18:48 web4 kernel: [26293] 0 26293 19682 71 0 0 0 master
Jul 3 02:18:48 web4 kernel: [26297] 0 26297 85463 328 0 0 0 httpd
Jul 3 02:18:48 web4 kernel: [26301] 0 26301 2091 86 0 0 0 diff
Jul 3 02:18:48 web4 kernel: [26302] 606 26302 4807 55 0 0 0 clam-wrapper.pl
Jul 3 02:18:48 web4 kernel: Out of memory: Kill process 23015 (clamscan) score 69 or sacrifice child
Jul 3 02:18:48 web4 kernel: Killed process 23015, UID 557, (clamscan) total-vm:230476kB, anon-rss:52024kB, file-rss:288kB
While copying and pasting the output I found that 2 out of 3 backups are still running. Which is really weird. Normally, they would be done by now according to previous backups logs. But how do backups cause clamscan to totally freak out?
Thanks again,
Dave.
Howdy,
The backups and clamscan issues aren't likely to be related, outside of the backups perhaps just taking longer if you have a high load.
Regarding why ClamAV is that heavily used... what kind of email volume are you seeing on your server?
And what does this command output:
mailq | tail -1That will show how many emails are currently in your mail queue.
Lastly -- in Virtualmin, if you go into Email Messages -> Spam and Virus Scanning, what is "Virus scanning program" set to?
-Eric
Yeah, I don't think the backups and the weird clamscan problem are related either.
I'm not sure, there don't seem to be any cool pretty graphs to look at. But there are only a few dozen domain names on the server with only 3 or 4 of them being remotely busy. I doubt the server receives more than a few hundred emails per day, and certainly not more than a thousand.
[root@web4 ~]# mailq | tail -1
-- 95 Kbytes in 7 Requests.
"Virus scanning program" is currently set to clamdscan instead of clamscan, but when this crazy problem happened it was set to clamscan. Our server only has 2gigs of ram, so during installation I chose this setting to save memory. I have since changed my mind, and we're now running clamdscan, and I also turned on the spamassassin server as well.
I think clamscan processes hung for some reason, and just built up, and built up until they were killed off, and disabled. But I don't know. This problem seems not to have a concrete cause.
The sysadmin at our hosting provider ran the following commands on the maillog to view how much email we were seeing. I pasted them below:
Neither the per hour email counts or per hour byte counts below show a significant spike at Jul 3 00:00-01:00. I am not sure what caused clamscan to act in this fashion, I assume it hasn't occurred before?
[root@web4 log]# export IFS=$'\n';for i in 'Jul '{2..3}' 0?'{0..23}:; do echo -ne "$i\t" ;egrep "$i" maillog|egrep -o size= -c;done|sed -r 's,0\?,,'|egrep -v ':[^0-9]+0$'
Jul 2 0: 148
Jul 2 1: 146
Jul 2 2: 156
Jul 2 3: 136
Jul 2 4: 183
Jul 2 5: 235
Jul 2 6: 235
Jul 2 7: 192
Jul 2 8: 212
Jul 2 9: 191
Jul 2 10: 278
Jul 2 11: 205
Jul 2 12: 554
Jul 2 13: 525
Jul 2 14: 318
Jul 2 15: 184
Jul 2 16: 164
Jul 2 17: 143
Jul 2 18: 108
Jul 2 19: 104
Jul 2 20: 158
Jul 2 21: 162
Jul 2 22: 180
Jul 2 23: 107
Jul 3 0: 392
Jul 3 1: 174
Jul 3 2: 314
Jul 3 3: 709
Jul 3 4: 254
Jul 3 5: 275
Jul 3 6: 219
Jul 3 7: 55
[root@web4 log]#
[root@web4 log]# export IFS=$'\n';for i in 'Jul '{2..3}' 0?'{0..23}:; do echo -ne "$i\t" ;egrep "$i" maillog|egrep -o size='[^,]+'|cut -d = -f2|awk 'BEGIN {t=0} {t+=$1} END { print t }';done|sed -r 's,0\?,,'|egrep -v ':[^0-9]+0$'
Jul 2 0: 1695740
Jul 2 1: 1482535
Jul 2 2: 1430766
Jul 2 3: 547280
Jul 2 4: 939471
Jul 2 5: 1003869
Jul 2 6: 1095034
Jul 2 7: 776350
Jul 2 8: 1272074
Jul 2 9: 650174482
Jul 2 10: 1075594485
Jul 2 11: 1040508476
Jul 2 12: 982291199
Jul 2 13: 972627727
Jul 2 14: 1463541380
Jul 2 15: 1047309141
Jul 2 16: 1305308054
Jul 2 17: 767457621
Jul 2 18: 1359172
Jul 2 19: 1056364
Jul 2 20: 8156372
Jul 2 21: 11563517
Jul 2 22: 1975934
Jul 2 23: 960144
Jul 3 0: 5342974
Jul 3 1: 7763894
Jul 3 2: 4279537
Jul 3 3: 8149454
Jul 3 4: 2707278
Jul 3 5: 5503118
Jul 3 6: 1679448
Jul 3 7: 265139
[root@web4 log]#
Thanks,
Dave.