Submitted by tomiskou on Thu, 10/25/2012 - 11:11
Hi,
I have just done an upgrade of my CentOS and the SSL part of webmin has stopped working. SSL connections to usermin and httpd work without any problems.
Below, you can see an extract of my yum.log. When I run the update for the first time packages usermin, wbt-virtual-server-theme and virtualmin failed becuase there was not enough free RAM. At that moment, I have worked with webmin for a little moment before I rerun the upgrade of the packages that failed (everything seemed to be working at that moment). When the install of webmin-1.600-1 succeeded, I couldn't connect to it anymore. After playing with it for a while I have figured out, that when I turn off ssl, webmin works. I have tried downgrading and reinstalling webmin and doing the same with a couple of other packages but it still doesn't work. If you have any idea what got broken and how to fix it I would appreciate your input very much.
Thank you!
Greetings, tomiskou
yum.log:
Oct 25 12:46:38 Updated: 30:bind-libs-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:39 Updated: 30:bind-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:40 Updated: 30:caching-nameserver-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:46:44 Updated: 30:bind-utils-9.3.6-20.P1.el5_8.5.x86_64
Oct 25 12:47:00 Updated: cyrus-sasl-lib-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:00 Updated: cyrus-sasl-lib-2.1.22-7.el5_8.1.i386
Oct 25 12:47:01 Updated: cyrus-sasl-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:02 Updated: cyrus-sasl-md5-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:03 Updated: cyrus-sasl-plain-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:03 Updated: cyrus-sasl-gssapi-2.1.22-7.el5_8.1.x86_64
Oct 25 12:47:04 Updated: cyrus-sasl-2.1.22-7.el5_8.1.i386
Oct 25 12:47:04 Updated: cyrus-sasl-gssapi-2.1.22-7.el5_8.1.i386
Oct 25 12:47:10 Updated: 12:dhclient-3.0.5-31.el5_8.1.x86_64
Oct 25 12:47:27 Updated: ghostscript-8.70-14.el5_8.1.x86_64
Oct 25 12:47:28 Updated: ghostscript-devel-8.70-14.el5_8.1.x86_64
Oct 25 12:47:29 Updated: ghostscript-8.70-14.el5_8.1.i386
Oct 25 12:48:23 Updated: glibc-common-2.5-81.el5_8.7.x86_64
Oct 25 12:48:30 Updated: glibc-2.5-81.el5_8.7.x86_64
Oct 25 12:48:31 Updated: nscd-2.5-81.el5_8.7.x86_64
Oct 25 12:48:35 Updated: glibc-headers-2.5-81.el5_8.7.x86_64
Oct 25 12:48:37 Updated: glibc-2.5-81.el5_8.7.i686
Oct 25 12:48:39 Updated: glibc-devel-2.5-81.el5_8.7.x86_64wbt-virtual-server-theme
Oct 25 12:48:48 Updated: initscripts-8.45.42-1.el5.centos.1.x86_64
Oct 25 12:49:21 Installed: kernel-2.6.18-308.16.1.el5.x86_64
Oct 25 12:49:32 Erased: kmod-dahdi-linux-fwload-vpmadt032
Oct 25 12:49:39 Updated: kernel-headers-2.6.18-308.16.1.el5.x86_64
Oct 25 12:49:44 Updated: libxml2-2.6.26-2.1.15.el5_8.5.x86_64
Oct 25 12:49:45 Updated: libxml2-2.6.26-2.1.15.el5_8.5.i386
Oct 25 12:49:49 Updated: libxml2-python-2.6.26-2.1.15.el5_8.5.x86_64
Oct 25 12:49:53 Updated: libxslt-1.1.17-4.el5_8.3.x86_64
Oct 25 12:49:54 Updated: libxslt-1.1.17-4.el5_8.3.i386
Oct 25 12:50:14 Updated: mysql-libs-5.5.28-12.el5.art.x86_64
Oct 25 12:50:18 Updated: mysql-5.5.28-12.el5.art.x86_64
Oct 25 12:50:21 Updated: mysql-server-5.5.28-12.el5.art.x86_64
Oct 25 12:50:24 Updated: mysql-libs-5.5.28-12.el5.art.i386
Oct 25 12:50:24 Updated: mysql-devel-5.5.28-12.el5.art.i386
Oct 25 12:50:25 Updated: mysql-server-5.5.28-12.el5.art.i386
Oct 25 12:50:26 Updated: mysql-devel-5.5.28-12.el5.art.x86_64
Oct 25 12:50:26 Updated: mysql-5.5.28-12.el5.art.i386
Oct 25 12:50:44 Updated: 2:nmap-6.01-2.el5.art.x86_64
Oct 25 12:50:52 Updated: perl-DBD-Pg-1.49-4.el5_8.x86_64
Oct 25 12:51:14 Installed: perl-Digest-SHA-5.71-1.el5.rf.x86_64
Oct 25 12:51:15 Updated: perl-Net-DNS-0.66-1.el5.art.x86_64
Oct 25 12:51:35 Updated: php-common-5.3.18-11.el5.art.x86_64
Oct 25 12:51:35 Updated: php-pdo-5.3.18-11.el5.art.x86_64
Oct 25 12:51:36 Updated: php-cli-5.3.18-11.el5.art.x86_64
Oct 25 12:51:37 Updated: php-5.3.18-11.el5.art.x86_64
Oct 25 12:51:37 Updated: php-xml-5.3.18-11.el5.art.i386
Oct 25 12:51:37 Updated: php-5.3.18-11.el5.art.i386
Oct 25 12:51:38 Updated: php-snmp-5.3.18-11.el5.art.x86_64
Oct 25 12:51:38 Updated: php-xmlrpc-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-devel-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-xml-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-mbstring-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-odbc-5.3.18-11.el5.art.x86_64
Oct 25 12:51:40 Updated: php-imap-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-gd-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-pgsql-5.3.18-11.el5.art.x86_64
Oct 25 12:51:41 Updated: php-mysql-5.3.18-11.el5.art.x86_64
Oct 25 12:51:51 Updated: postgresql-8.1.23-6.el5_8.x86_64
Oct 25 12:51:56 Updated: postgresql-server-8.1.23-6.el5_8.x86_64
Oct 25 12:52:04 Updated: postgresql-libs-8.1.23-6.el5_8.x86_64
Oct 25 12:52:05 Updated: postgresql-libs-8.1.23-6.el5_8.i386
Oct 25 12:52:10 Updated: sudo-1.7.2p1-14.el5_8.4.x86_64
Oct 25 12:52:19 Updated: tzdata-2012f-1.el5.x86_64
Oct 25 12:52:50 usermin-1.520-1.noarch: 100
Oct 25 12:53:20 Updated: wbm-virtual-server-3.94.gpl-2.noarch
Oct 25 12:53:25 Updated: 2:wbm-virtualmin-htpasswd-2.6-1.noarch
Oct 25 12:53:32 Updated: 2:wbm-virtualmin-registrar-2.2-1.noarch
Oct 25 12:53:38 Updated: 2:wbt-virtual-server-mobile-2.5-1.noarch
Oct 25 12:53:46 2:wbt-virtual-server-theme-8.5-2.noarch: 100
Oct 25 13:35:10 Updated: usermin-1.520-1.noarch
Oct 25 13:35:21 Updated: 2:wbt-virtual-server-theme-8.5-2.noarch
Oct 25 13:39:37 Updated: webmin-1.600-1.noarchStatus:
Active
Comments
Submitted by JamieCameron on Thu, 10/25/2012 - 11:18 Comment #1
It may have failed to start due to lack of RAM. Try SSHing in and running /etc/webmin/start as root.
How much RAM do you have?
Submitted by tomiskou on Thu, 10/25/2012 - 12:24 Comment #2
Hi,
Thanks for a prompt reply.
I have 512MB/1024MB (burst) which has been enough for the moment (about 2 years of running on the same machine).
webmin starts even with ssl=1 option and nmap can see it running on port 10000. When I connect with Firefox I get 'connection was interrupted'. As I said already ssl=0 works without problems as well as usermin with ssl=1.
Thanks.
tomiskou
Submitted by andreychek on Thu, 10/25/2012 - 12:34 Comment #3
What do you see in the Webmin logs whenever you're having that problem?
The Webmin log is in /var/webmin/miniserv.log.
Also, it sounds like you're using OpenVZ, which unfortunately often introduces resource problems -- can you paste in your /proc/user_beancounters file?
Submitted by tomiskou on Thu, 10/25/2012 - 13:28 Comment #4
I do not see anything in webmin logs nor in any other logs.
Yep, I am using OpenVZ. The output of /proc/user_beancounters is below, but I have compared the fail counts before and after trying to connect to webmin as well as before and after restarting webmin and no new fail counts have appeared. Could you tell me which variable could be related to my problem?
I do not see why this would be a resource problem since this feature worked for me for last two years (on the same machine). If it really is a resource problem, I should be able to resolve it by downgrading to the version of webmin which worked, no? This doesn't help however and so I assume that the problem is in the other packages that I have upgraded (I just do not see which and why). I do not want to downgrade all of them since downgrading cyrus-sasl, for example cost me a lot of time to reinstall and properly reconfigure all the programs that depended on it.
I have a related question, as mentioned above I had to reinstall a couple of other packages and I screwed up my postfix. I have managed to restore its basic functionality but the user defined mail filters do not work. Would you point to the configuration file in which this could be fixed? (The filters are completely ignored and all the mail gets delivered to my mailbox instead of being delivered to appropriate subfolders defined in these filters).
Thanks.
tomiskou
/proc/user_beancounters:
Version: 2.5uid resource held maxheld barrier limit failcnt
19043: kmemsize 21720013 80268896 134217728 134217728 0
lockedpages 0 1024 1024 1024 156
privvmpages 146797 282755 262144 262144 817402
shmpages 5077 27743 102400 102400 0
dummy 0 0 0 0 0
numproc 94 359 640 640 0
physpages 63248 171942 0 9223372036854775807 0
vmguarpages 0 0 131072 9223372036854775807 0
oomguarpages 63260 171943 104857 9223372036854775807 0
numtcpsock 30 266 1024 1024 0
numflock 7 81 2048 2048 0
numpty 3 3 64 64 0
numsiginfo 0 220 1024 1024 0
tcpsndbuf 651376 5429880 5368709 10737418 13539424
tcprcvbuf 491520 5389296 5368709 10737418 1181
othersockbuf 292536 4583448 5368709 10737418 0
dgramrcvbuf 0 198208 1342177 2684354 0
numothersock 163 1024 1024 1024 788
dcachesize 927732 2290221 8053063 12582912 0
numfile 3213 14600 32768 32768 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 50 50 1536 1536 0
Submitted by andreychek on Thu, 10/25/2012 - 14:03 Comment #5
That's a good step to review the user_beancounters both before and after an issue, that does rule out that the SSL issue is due to a current restriction.
However, even though those failures may not be occurring now, it sounds like there may have been resource problems during the package updates, which could lead to corrupt packages and config files.
As you see in the user_beancounters output, there's a bunch of resource failures that your server is seeing. Even one failure in the failcnt field can mean serious problems. There's quite a few there, unfortunately.
One thing I'd suggest, is that I'd encourage you to work with your provider to increase your resources -- as so long as you have any failures, you could end up with really strange issues :-/ Show them where you're running into failcnt's, and they should be able to help with that.
We're not trying to give you a hard time, it's just an issue we've been seeing a lot of -- folks who have failcnt's not at 0 have some of the most bizarre issues! It's becoming so common that we're actually considering adding a warning into Virtualmin to notify folks when the failcnt field is non-zero.
As far as how to resolve this -- since OpenVZ limits generate hard failures, it's difficult to say what exactly may have gone awry during the package updates... but one place to start may be to go into Server Configuration -> Manage SSL Certificates for a domain that has SSL setup, and to click the "Copy to Webmin" button. That will re-setup SSL in Webmin.
As far as Postfix goes -- you'd want to verify your /etc/procmailrc file. It's possible that became corrupt. Also, in /etc/postfix/main.cf, make sure that "mailbox_command" is set to the following:
/usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAMEAlthough we can try and get you going in the right direction with this, remember that the Support area here is for Virtualmin Pro customers. For future reference, if you're using Virtualmin GPL, you'd want to use the Forums for support. We monitor those, along with lots of wonderful folks in the community... thanks!
Submitted by tomiskou on Sun, 10/28/2012 - 18:55 Comment #6
I have solved the issue with user filters and postfix. It was actually as easy as setting "Allow mailbox users to create mail filters?" to Yes in "Email Messages -> Spam and Virus Scanning" which must have gotten reset to No when I removed postfix by mistake when downgrading cyrus-sasl.
Meanwhile I have noticed that from time to time I do get "Failed to initialize SSL connection" error message in miniserv.log. When running webmin in SSL=1 mode. It never works but the error message appears only half of the time, which is why I didn't notice it at first. "Copy to Webmin" didn't work.
I know that you are not trying yo give me hard time and I appreaciate any help. I can see how resource limits may lead to most confusing errors and that it is impossible to resolve them all. Unfortunatelly I have already had a discussion about bean_counters and resources with my provider when I had another issue, it looks like increasing some of the limits is out of question. I will just change my provider when I find some time.
Meanwhile I will keep trying to figure out what got broken during the failed upgrade.
In any case thanks for your assistance.
Submitted by andreychek on Sun, 10/28/2012 - 20:01 Comment #7
Meanwhile I have noticed that from time to time I do get "Failed to initialize SSL connection" error message in miniserv.log. When running webmin in SSL=1 mode. It never works but the error message appears only half of the time, which is why I didn't notice it at first. "Copy to Webmin" didn't work.
Are you or any of your users having problems? Or are you just seeing that in the logs?
It's not uncommon to see that problem in the logs -- I have that message 146 times on my own personal server over the last month.
The issue is that bots doing probing can cause that error to appear, as they don't always connect using SSL mode. That's safe to ignore though, unless you or your users are seeing problems when connecting with your browser.
Submitted by tomiskou on Mon, 10/29/2012 - 13:07 Comment #8
Unfortunatelly, it is not something that only appears in logs. I cannot login to webmin with my browser when run in SSL=1 mode at all and so I temporarily run it in SSL=0 mode (and I have tried couple of different browsers and machines, just to be sure that it is not a client side problem).
I was digging a little bit deeper and have noticed that when run in SSL=1 mode Net::SSLeay::accept($ssl_con) in miniserv.pl either returns False or simply dies (lines after the call are ignored). I didn't yet have time to figure out how to go deeper since if I understand it correctly Net::SSLeay is a perl openssl wrapper. Otherwise I would have already traced the code. I have tried reinstalling openssl and and Perl-Net-SSLeay with yum reinstall, which didn't help.
What is funny is that usermin (apparently also driven by miniserv.pl) runs in SSL=1 mode without any problems. I have tried to use the certificates I am using in usermin also in webmin, just to be sure that the problem doesn't come from the certificates, but this doesn't help.
I will try to debug usermin and webmin side by side to see what the difference in SSL related configuration is. Perl is not in the list of my native languages however, so it might take a little while before I figure out the best of doing that.