miniserv.pl runaway

Submitted by scottm on Thu, 04/18/2013 - 22:07

Odd miniserv.pl issue.... logs show NOTHING but logins, normal activity.

Log onto a particular user account, two miniserv.pl processes start and build both CPU and memory utilization til it halts the machine. 4GB RAM and 4 Cores all with two processes!

Emptied the .spam folder, got rid of the ~/.usermin/mailbox/sort.* files, reduced the inbox to less than half the emails (now sitting at 5000) without alot of disk space being utilized.

With nothing strange in the logs, etc., I'm kinda lost here. What to look for?

Status: 
Active

Comments

Submitted by JamieCameron on Thu, 04/18/2013 - 23:12

I'd be interested to see the output from the command :

lsof -p XXX

where XXX is the PID of one of the runaway miniserv.pl processes. That should show what files it is reading, and give some idea as to the cause of the issue.

Submitted by scottm on Fri, 04/19/2013 - 06:18

Here you go Jamie:

COMMAND     PID           USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
miniserv. 17813 powell.domaincom  cwd    DIR    9,2      4096  68060434 /usr/libexec/usermin/mailbox
miniserv. 17813 powell.domaincom  rtd    DIR    9,2      4096         2 /
miniserv. 17813 powell.domaincom  txt    REG    9,2     19200  67347662 /usr/bin/perl
miniserv. 17813 powell.domaincom  mem    REG    9,2    144776  95158578 /lib64/ld-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2   1726344  95158628 /lib64/libc-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     23360  95158629 /lib64/libdl-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2    149968  95158645 /lib64/libpthread-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2    614992  95158649 /lib64/libm-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2   1262288  67438260 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     95464  95158633 /lib64/libselinux.so.1
miniserv. 17813 powell.domaincom  mem    REG    9,2    247496  95158632 /lib64/libsepol.so.1
miniserv. 17813 powell.domaincom  mem    REG    9,2     53448  95158456 /lib64/librt-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2    114352  95158644 /lib64/libnsl-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     48600  95158641 /lib64/libcrypt-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2      9472  95158630 /lib64/libkeyutils-1.2.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     92816  95158631 /lib64/libresolv-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     10096  95158634 /lib64/libcom_err.so.2.1
miniserv. 17813 powell.domaincom  mem    REG    9,2   1367456  95158636 /lib64/libcrypto.so.0.9.8e
miniserv. 17813 powell.domaincom  mem    REG    9,2     35984  67342825 /usr/lib64/libkrb5support.so.0.1
miniserv. 17813 powell.domaincom  mem    REG    9,2    153720  67344226 /usr/lib64/libk5crypto.so.3.1
miniserv. 17813 powell.domaincom  mem    REG    9,2    613928  67344228 /usr/lib64/libkrb5.so.3.3
miniserv. 17813 powell.domaincom  mem    REG    9,2    190976  67344254 /usr/lib64/libgssapi_krb5.so.2.2
miniserv. 17813 powell.domaincom  mem    REG    9,2    323400  95158638 /lib64/libssl.so.0.9.8e
miniserv. 17813 powell.domaincom  mem    REG    9,2     18152  95158491 /lib64/libutil-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     21424  67471297 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
miniserv. 17813 powell.domaincom  mem    REG    9,2    119224  67471293 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     13056  67469320 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     19464  67471270 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     24856  67471304 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
miniserv. 17813 powell.domaincom  mem    REG    9,2    323000  67829782 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     85544  95158635 /lib64/libz.so.1.2.3
miniserv. 17813 powell.domaincom  mem    REG    9,2     15480  67438209 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Digest/MD5/MD5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     53880  95158343 /lib64/libnss_files-2.5.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     25952  67829772 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Socket6/Socket6.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     33112  67469508 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/SDBM_File/SDBM_File.so
miniserv. 17813 powell.domaincom  mem    REG    9,2     27832  95158338 /lib64/libnss_dns-2.5.so
miniserv. 17813 powell.domaincom    0r   CHR    1,3       0t0      1286 /dev/null
miniserv. 17813 powell.domaincom    1w   CHR    1,3       0t0      1286 /dev/null
miniserv. 17813 powell.domaincom    2w   REG    9,2  45563280 170197232 /var/usermin/miniserv.error
miniserv. 17813 powell.domaincom    3w   CHR    1,3       0t0      1286 /dev/null
miniserv. 17813 powell.domaincom    4w   REG    9,2 101436333 170197236 /var/usermin/miniserv.log
miniserv. 17813 powell.domaincom    5u  sock    0,5       0t0    304628 can't identify protocol
miniserv. 17813 powell.domaincom    8u  IPv4 304620       0t0       TCP relay1.domaincom.com:dnp->mo-76-4-173-108.dhcp.embarqhsd.net:13641 (ESTABLISHED)
miniserv. 17813 powell.domaincom   11w  FIFO    0,6       0t0    304621 pipe
miniserv. 17813 powell.domaincom   13r  FIFO    0,6       0t0    304622 pipe

miniserv.log looks reasonable:

76.4.173.108 - root [18/Apr/2013:23:12:05 -0400] "GET /mailboxes/view_mail.cgi?user=powell%2Edomaincom&idx=5162&folder=0&dom=130818490217025 HTTP/1.1" 200 15184 76.4.173.108 - root [19/Apr/2013:06:57:31 -0400] "GET /virtual-server/edit_user.cgi?dom=130818490217025&user=powell%2Edomaincom&unix=1 HTTP/1.1" 200 27675 76.4.173.108 - root [19/Apr/2013:06:57:35 -0400] "GET /mailboxes/list_mail.cgi?user=powell.domaincom&dom=130818490217025 HTTP/1.1" 200 26333 76.4.173.108 - root [19/Apr/2013:06:58:00 -0400] "GET /mailboxes/list_mail.cgi?start=&folder=0&user=powell.domaincom&dom=130818490217025 HTTP/1.1" 200 26255 76.4.173.108 - root [19/Apr/2013:06:58:56 -0400] "GET /mailboxes/list_mail.cgi?start=&folder=0&user=powell.domaincom&dom=130818490217025 HTTP/1.1" 200 26313

And the miniserv.error isn't showing anything either:

[19/Apr/2013:07:03:21 -0400] [76.4.173.108] Bad Request : This web server is running in SSL mode. Try the URL https://relay1.openrsm.com:10000/ instead.

Submitted by JamieCameron on Fri, 04/19/2013 - 16:02

That's unusual, I don't see any open files from the process you ran lsof on. Was that the one that was using up 100% CPU?

Submitted by scottm on Sun, 04/21/2013 - 17:59

Yes, and it's with ALL mailboxes. I put squirrelmail on to test and it's doing just fine. The internal "root" account can read all email fine too.

Submitted by JamieCameron on Sun, 04/21/2013 - 21:49

Does it help if you delete all the files matching .usermin/mailbox/*index* under the home directory of a user who is having this problem? These are just cache files that will be re-generated.

Submitted by scottm on Mon, 04/22/2013 - 08:11

No change... TWO miniserv.pl processes start and combined quickly consume all memory and send the machine into swap. CPU fluxes between 70-99% utilization (20 is usual).

Submitted by JamieCameron on Mon, 04/22/2013 - 15:34

Any chance I could login to the system that is experiencing this problem in order to debug it?

Submitted by JamieCameron on Mon, 04/22/2013 - 19:30

Thanks for the login - I was able to connect to your system.

Which specific mailbox did you see this problem for though? I tried a couple, but didn't see processes get stuck at 100% CPU load.

Submitted by scottm on Mon, 04/22/2013 - 19:34

scottm.openrsm (via mail.openrsm.com:20000)

Most everyone else logs in and the machine isn't thrashing, but I'm receiving reports that the page never completes.. Just sits there.

englandj.jazzkitchencom is one that is doing that.

Submitted by JamieCameron on Mon, 04/22/2013 - 23:31

It looks like the scottm account problem was due to the file /home/openrsm/homes/scottm/.usermin/mailbox/1.virt being corrupt, which caused an infinite loop reading the contents of the search results folder. When I renamed that file, the initial problem went away.

However, there is also a problem that the password for the scottm account is out of sync with what usermin expects. You may need to change it to something different, then change it back again.