Virtualmin inaccessible after reboot. Quota issue (Promox -> OpenVz -> Debian 6.0)

Greetings.

Here's my config:

Proxmox as the bare metal hypervisor OpenVZ as the virtualization Debian 6.0 Squeeze 64 bit (also have a separate 32 bit install)

Two problems:

1) After Virtualmin installed. It's accessible and I followed through the configuration process. After the reboot, it's not accessible. I can see that Apache is running fine and it can also be validated via the browser to the same IP. I could not get virtualmin interface to come up at all http://serverip:10000. It works fine right before the reboot. What went wrong?

2) During the server validation process, Virtualmin complained about quota not enabled:

"Quotas are not enabled on the filesystem / which contains home directories under /home and email files under /home. Quota editing has been disabled."

I went to the Webmin->System-.Disk Quota but could not figure out how to properly configure it.

Quota is enabled at the Promox control panel for OpenVZ. I also look at another virtualmin support thread and added the below comments to /etc/init.d/vzquota:

BEGIN INIT INFO

Provides: vzquota Required-Start: $local_fs $remote_fs $network $syslog Required-Stop: $local_fs $remote_fs $network $syslog Default-Start: 2 3 4 5 Default-Stop: 0 1 6 Short-Description: Create /etc/fstab file

END INIT INFO

Very strange, every time I reboot, the above comment dissappeared from /etc/init.d/vzquota.

Virtualmin seems to work fine with Centos 5.2.x as an OpenVZ container. So is this an issue between Virtualmin and Debian?

Thank you in advance for your help.

Status: 
Closed (fixed)

Comments

After Virtualmin installed. It's accessible and I followed through the configuration process. After the reboot, it's not accessible.

Is it possible Webmin isn't running? You can try restarting it by running this command as root:

/etc/init.d/webmin restart

During the server validation process, Virtualmin complained about quota not enabled: Very strange, every time I reboot, the above comment dissappeared from /etc/init.d/vzquota.

Well, many OpenVZ setups don't work well with quotas. Also, some OpenVZ providers overwrite certain config files each reboot, which can make changes like you tried impossible.

If you don't require quotas, you can always tell Virtualmin not to try to use them in System Settings -> Virtualmin Config, and set "Set quotas for domain and mail users" to "No".

Just a reminder, this is Debian 6.0 running under OpenVZ.

I just discovered another ordeal. Virtualmin Pro will not install. Virtualmin GPL installed just fine. I wonder why?

I went ahead and testing it with the GPL version, then convert it to Virtualmin Pro. I do notice the "Server Monitoring" feature doesn't appear on the GPL version converting to Pro. This feature does show up on Pro with other successful install.

You are absolutely right. Webmin wasn't running. I have to start it up manually each time.

In the control panel, Webmin is configured to auto start at boot. I wonder why it does this only to Debian Lenny (5.0) and Squeeze (6.0). Webmin start up just fine with Ubuntu and Centos.

Can you help point me as to where I could look to trouble-shoot this issue? It kind of make me wonder what else is not starting. Is there somewhere I could look to make sure everything is supposed to start up the way they are intended?

I just discovered another ordeal. Virtualmin Pro will not install. Virtualmin GPL installed just fine. I wonder why?

Well, it's difficult to say without any error messages, but OpenVZ can be a bit fickle due to the way it handles memory. It's possible it was failing during one installation attempt, and worked during the other, purely due to memory problems.

I do notice the "Server Monitoring" feature doesn't appear on the GPL version converting to Pro

If you look in System Settings -> Features and Plugins, is the "Status Monitoring" option there checked? Or is that the area where you aren't seeing Status Monitoring at all?

You are absolutely right. Webmin wasn't running. I have to start it up manually each time.

What is the output of this command:

ps auxw | grep init

Thanks very much for the quick reply.

  • Where do I go to get the error log from the failed installation?

  • Here's the result:

root@a:~# ps auxw | grep init root 1 0.0 0.0 8352 800 ? Ss 08:04 0:00 init [2] root 21 0.0 0.0 100 16 ? S 08:04 0:00 [init-logger] root 1094 0.0 0.0 6028 708 pts/0 S+ 08:04 0:00 grep init

Where do I go to get the error log from the failed installation?

Either from the output you see on the screen during the installation, or if that doesn't have the information you need, from the install log in /root/virtualmin-install.log. However, that's most likely been overwritten with your more recent install.

init [2]

Okay, this means that you're running in run level 2.

What do you see if you run this command -- it should show if there's a startup script created for Webmin:

ls /etc/rc2.d/ | grep webmin

Thanks. I will reattempt the install again with the Pro and get the error log.

Here's the result from the last command:

root@a:~# ls /etc/rc2.d/ | grep webmin S02webmin

What should I do next?

Okay, try running this command as root:

mv /etc/rc2.d/S02webmin /etc/rc2.d/S99webmin

After your next reboot, does Webmin launch as expected?

Hi,

I ran the command as root:

mv /etc/rc2.d/S02webmin /etc/rc2.d/S99webmin

Reboot, but webmin will not start. I had to manually started it. Any other suggestions? Thanks for your help.

What output do you receive when running these two commands:

free -m

cat /proc/user_beancounters

root@a:~# free -m

total used free shared buffers cached
Mem: 8192 625 7566 0 0 0
-/+ buffers/cache: 625 7566
Swap: 0 0 0

root@a:~# cat /proc/user_beancounters


Version: 2.5
uid resource held maxheld barrier limit failcnt
101: kmemsize 15488383 16942145 9223372036854775807 9223372036854775807 0
lockedpages 0 0 9223372036854775807 9223372036854775807 0
privvmpages 160171 197401 2097152 2109652 0
shmpages 8870 8886 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
numproc 81 89 1024 1024 0
physpages 69114 106489 0 9223372036854775807 0
vmguarpages 0 0 2097152 9223372036854775807 0
oomguarpages 69114 106489 2097152 9223372036854775807 0
numtcpsock 24 24 9223372036854775807 9223372036854775807 0
numflock 10 15 9223372036854775807 9223372036854775807 0
numpty 1 1 255 255 0
numsiginfo 0 2 1024 1024 0
tcpsndbuf 418560 0 9223372036854775807 9223372036854775807 0
tcprcvbuf 393216 0 9223372036854775807 9223372036854775807 0
othersockbuf 242760 268424 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 4360 9223372036854775807 9223372036854775807 0
numothersock 166 180 9223372036854775807 9223372036854775807 0
dcachesize 563504 623634 9223372036854775807 9223372036854775807 0
numfile 2189 2420 9223372036854775807 9223372036854775807 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
dummy 0 0 0 0 0
numiptent 10 10 9223372036854775807 9223372036854775807

Jamie, Webmin isn't launching at bootup for him.

He's booting into run level 2, and there's an S99webmin in rc2.d.

Although he's running OpenVZ, there's no apparent memory problems, no memory failures have ever been logged, and it looks like he has 8GB of dedicated RAM.

Can you think of any other issues that may be preventing Webmin from launching during bootup?

I also want to add.

This only happens on Debian 6.0 (both 32 and 64 bit). The problem does not occur with Centos, Ubuntu, and Debian Lenny 5.0.

Please let me know if you need me to reinstall under Ubuntu and compare to see why this is happening on Debian 6.0?

Perhaps the system isn't booting to runlevel 2?

What does the runlevel command output after your system is booted?

Hi Jamie,

root@a:~# runlevel
N 2

If you check the log file /var/webmin/miniserv.error just after a reboot , are there entries in it indicating that Webmin tried to start at boot time?

Hi Jamie,

1) This is what was there:

GNU nano 2.2.4 File: /var/webmin/miniserv.error

[08/May/2011:07:19:38 -0700] miniserv.pl started
[08/May/2011:07:19:38 -0700] PAM authentication enabled
[08/May/2011:07:20:20 -0700] Reloading configuration
[08/May/2011:07:20:22 -0700] Reloading configuration
restarting miniserv
[08/May/2011:07:20:24 -0700] Restarting
[08/May/2011:07:20:26 -0700] miniserv.pl started
[08/May/2011:07:20:26 -0700] PAM authentication enabled
Failed to initialize SSL connection
Failed to initialize SSL connection
Failed to initialize SSL connection
[08/May/2011:09:53:29 -0700] miniserv.pl started
[08/May/2011:09:53:29 -0700] PAM authentication enabled

2) JUST NOW, I DELETED ALL THE LINES TO AVOID CONFUSION. REBOOTED THE SERVER. THEN MANUALLY STARTED /etc/init.d/webmin start. Here's the log:

GNU nano 2.2.4 File: /var/webmin/miniserv.error

[08/May/2011:21:38:38 -0700] miniserv.pl started
[08/May/2011:21:38:38 -0700] PAM authentication enabled

Could it be an SSL issue? Thanks very much for your help.

Hi Jamie,

Just checking to see what your thoughts are. Thanks very much.

So it sounds like Webmin's init script isn't being started at boot time ..

What does /etc/init.d/webmin contain on your system?

Also, what is the output from :

ls -l /etc/rc*/S*webmin

and finally, do you have access to the system's console to see boot messages?

#!/bin/sh
# chkconfig: 235 99 10
# description: Start or stop the Webmin server
#
### BEGIN INIT INFO
# Provides: webmin
# Required-Start: $network $syslog
# Required-Stop: $network
# Default-Start: 2 3 5
# Default-Stop: 0 1 6
# Description: Start or stop the Webmin server
### END INIT INFO

start=/etc/webmin/start
stop=/etc/webmin/stop
lockfile=/var/lock/subsys/webmin
confFile=/etc/webmin/miniserv.conf
pidFile=/var/webmin/miniserv.pid
name='Webmin'

case "$1" in
'start')
$start >/dev/null 2>&1 /dev/null 2>&1
fi
;;
'stop')
$stop
RETVAL=$?
if [ "$RETVAL" = "0" ]; then
rm -f $lockfile
fi
pidfile=`grep "^pidfile=" $confFile | sed -e 's/pidfile=//g'`
if [ "$pidfile" = "" ]; then
pidfile=$pidFile
fi
rm -f $pidfile
;;
'status')
pidfile=`grep "^pidfile=" $confFile | sed -e 's/pidfile=//g'`
if [ "$pidfile" = "" ]; then
pidfile=$pidFile
fi
if [ -s $pidfile ]; then
pid=`cat $pidfile`
kill -0 $pid >/dev/null 2>&1
if [ "$?" = "0" ]; then
echo "$name (pid $pid) is running"
RETVAL=0
else
echo "$name is stopped"
RETVAL=1
fi
else
echo "$name is stopped"
RETVAL=1
fi
;;
'restart')
$stop ; $start
RETVAL=$?
;;
*)
echo "Usage: $0 { start | stop | restart }"
RETVAL=1
;;
esac
exit $RETVAL

root@a:~# ls -l /etc/rc*/S*webmin

lrwxrwxrwx 1 root root 16 May 9 13:38 /etc/rc2.d/S02webmin -> ../init.d/webmin
lrwxrwxrwx 1 root root 16 May 9 13:38 /etc/rc3.d/S02webmin -> ../init.d/webmin
lrwxrwxrwx 1 root root 16 May 9 13:38 /etc/rc5.d/S02webmin -> ../init.d/webmin
root@a:~#

That all looks fine to me ..

Any way you can get the console output from the boot process, to see what actions were run and if any failed?

I do have full access to the server. The server run Proxmox as the management hypervisor. I utilize OpenVZ as the virtualization choice. Debian 6.0 is the OS within OpenVZ and run as a container.

After Proxmox boot, I can click on the Open VNC to access the server. However, I do not get to see the normal console boot. I do see a log option on Promox for this particular container. This is what it presented:

starting init logger
INIT: version 2.88 booting
Activating swap...done.
mount: permission denied
Cleaning up ifupdown....
Fast boot enabled, so skipping file system check. ... (warning).
Setting kernel variables ...done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting up networking....
Configuring network interfaces...done.
Starting portmap daemon....
Cleaning up temporary files....
/etc/init.d/urandom: 77: cannot create /dev/urandom: Operation not permitted
INIT: Entering runlevel: 2
stty: standard input: Invalid argument
Starting enhanced syslogd: rsyslogd.
Starting system message bus: dbus.
Starting domain name service...: bind9.
Starting OpenBSD Secure Shell server: sshd.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting PostgreSQL 8.4 database server: main.
Starting SpamAssassin Mail Filter Daemon: spamd.
Starting Postfix Mail Transport Agent: postfix.
Starting Mailman master qrunner: mailmanctl.
Starting ClamAV virus database updater: freshclam.
Starting ClamAV daemon: clamd .
Starting IMAP/POP3 mail server: dovecotIf you have trouble with authentication failures,
enable auth_debug setting. See http://wiki.dovecot.org/WhyDoesItNotWork
This message goes away after the first successful login.
.
Starting SASL Authentication Daemon: saslauthd.
Starting portmap daemon...Already running..
Starting ftp server: proftpd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting web server: apache2.
INIT: no more processes left in this runlevel

Looks like it never actually tried to start Webmin at all ..

I wonder if perhaps the runlevel is not what we think it is. What is the output from :

ls /etc/rc*.d/*mysql*

root@a:~# ls /etc/rc*.d/*mysql*

/etc/rc0.d/K02mysql /etc/rc3.d/S02mysql /etc/rc6.d/K02mysql
/etc/rc1.d/K02mysql /etc/rc4.d/S02mysql
/etc/rc2.d/S02mysql /etc/rc5.d/S02mysql
root@a:~#

The only difference I see here between the webmin and mysql actions is in runlevel 4 .. although I wouldn't expect that to matter, since the system appears to be booting to level 2.

You can try fixing this by running :

ln -s /etc/init.d/webmin /etc/rc4.d/S02webmin

Hi Jamie,

Ran the command: ln -s /etc/init.d/webmin /etc/rc4.d/S02webmin

Still same problem. Webmin has to manually started.

Jamie,

I am thinking of calling in a priest from a nearby church. He may be able to help with two things. 1) Exorcising my servers to get rid of the lurking poltergeist surrounding this possible Debian bug or 2) Help me pray to God that it will work fine with the manual start-up method.

Okay I am just kidding. But seriously, this may not be a big deal with the manual start. However, my ISP have been encountering minor problems with power reset at the data center. If this happens in the middle of the night, it would be nice to know webmin would start-up by itself without my intervention to do a manual start up. I would love to find out what it is that cause it not to start up. This may help uncover other potential hidden areas that we may not be aware of.

I also forgot to mention, the websites would start up just fine. This prove that Apache is not affected. It's just Webmin/Virtual have no access. Please let me know what else I would need to do on this end. Thanks for your help in this matter. I sincerely appreciate it very much.

I am totally confused by this, as I've never seen this kind of startup failure before.

One hack work-around would be to edit the file /etc/rc.local , and at the end add the line :

/etc/webmin/start

Hi jamie,

The hack work-around seems to work. I guess we will leave it at that. I sincerely think it's an OpenZV issue as the same install seems to boot fine under KVM. Promox is not directly involved with webmin/virtualmin so at least we can rule that out.

Just an FYI. The reason why I am stuck with OpenVZ over KVM is performance related. I tested all linux distros with KVM. Nothing survive a load test of 10 Requests Per Second (RPS). That's about 25M RPS per month. For some strange reason, the CPU load goes extremely high and the servers come to a crawl. Under OpenVZ, it cruise right through it even at 20 RPS (50M per month). I know this has nothing to do with webmin/virtualmin. But I thought this tip may save webmaster lots of time from trial and errors. It cost me many days, countless installations, updates, etc... just to find that out. Also, under OpenVZ, the best distro tested under performance is Debian 6.0 amd64 distro (if you are running 64 bit hardware). You must also use the one provided by Proxmox. The one came from OpenVZ PreCreated version will not work as well. I hope that helps. Thanks.

Ok, I'm glad that worked. All I can guess is that OpenVZ or Proxmox is somehow changing the way boot scripts are run, which is breaking Webmin.

Jamie. Thanks very much for all of your time and kind effort. It is greatly appreciated.

Jamie,

I forgot to ask. Do I add this line before the "exit 0" or after? Thanks again.

Before the exit 0 , as the exit statement will terminate the script.

Hi Jamie,

I would like to trouble-shoot this matter with one last attempt. Any help is always appreciated.

Since Webmin has no problem starting under a Ubuntu 8.0.4 install, I went ahead and created another virtual machine. I compare the two boot logs. The only suspecting statements which I could see are these on the Debian machine:

/etc/init.d/urandom: 77: cannot create /dev/urandom: Operation not permitted
INIT: Entering runlevel: 2
stty: standard input: Invalid argument

Here are the complete logs from the two servers:

1) THIS IS THE BOOT LOG FROM UBUNTU 8.0.4 VIRTUAL MACHINE:
---------------------------------------------------------------------------------------------------
Boot/Init Command Syslog

INIT LOG Online
starting init logger
* Setting preliminary keymap... [ OK ]
* Starting basic networking... [ OK ]
* Loading kernel modules... * Loading manual drivers... [ OK ]
* Setting kernel variables... [ OK ]
* Activating swap... [ OK ]
mount: permission denied
* Mounting local filesystems... [ OK ]
* Activating swapfile swap... [ OK ]
* Checking minimum space in /tmp... [ OK ]
* Skipping firewall: ufw (not enabled)... [ OK ]
* Configuring network interfaces... [ OK ]
* Starting system log daemon... [ OK ]
* Starting domain name service... bind [ OK ]
* Starting OpenBSD Secure Shell server sshd [ OK ]
* Starting MySQL database server mysqld [ OK ]
* Checking for corrupt, not cleanly closed and upgrade needing tables.
* Starting PostgreSQL 8.3 database server [ OK ]
Starting SpamAssassin Mail Filter Daemon: spamd.
* Starting ClamAV daemon clamd LibClamAV Warning: ***********************************************************
LibClamAV Warning: *** This version of the ClamAV engine is outdated. ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning: ***********************************************************
LibClamAV Warning: ***********************************************************
LibClamAV Warning: *** This version of the ClamAV engine is outdated. ***
LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
LibClamAV Warning: ***********************************************************
[ OK ]
* Starting ClamAV virus database updater freshclam [ OK ]
* Starting Mailman master qrunner mailmanctl [ OK ]
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting SASL Authentication Daemon saslauthd [ OK ]
* Starting ftp server proftpd [ OK ]
* Starting deferred execution scheduler atd [ OK ]
* Starting periodic command scheduler crond [ OK ]
* Starting web server apache2 [ OK ]
* Running local boot scripts (/etc/rc.local) [ OK ]

2) THIS IS THE BOOT LOG FROM DEBIAN 6.0 VIRTUAL MACHINE:
--------------------------------------------------------
Boot/Init Command Syslog

INIT LOG Online
starting init logger
INIT: version 2.88 booting
Activating swap...done.
mount: permission denied
Cleaning up ifupdown....
Fast boot enabled, so skipping file system check. ... (warning).
Setting kernel variables ...done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting up networking....
Configuring network interfaces...done.
Starting portmap daemon....
Cleaning up temporary files....
/etc/init.d/urandom: 77: cannot create /dev/urandom: Operation not permitted
INIT: Entering runlevel: 2
stty: standard input: Invalid argument
Starting enhanced syslogd: rsyslogd.
Starting system message bus: dbus.
Starting domain name service...: bind9.
Starting OpenBSD Secure Shell server: sshd.
Starting MySQL database server: mysqld.
Checking for corrupt, not cleanly closed and upgrade needing tables..
Starting PostgreSQL 8.4 database server: main.
Starting SpamAssassin Mail Filter Daemon: spamd.
Starting Postfix Mail Transport Agent: postfix.
Starting Mailman master qrunner: mailmanctl.
Starting ClamAV virus database updater: freshclam.
Starting ClamAV daemon: clamd .
Starting IMAP/POP3 mail server: dovecotIf you have trouble with authentication failures,
enable auth_debug setting. See http://wiki.dovecot.org/WhyDoesItNotWork
This message goes away after the first successful login.
.
Starting SASL Authentication Daemon: saslauthd.
Starting portmap daemon...Already running..
Starting ftp server: proftpd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting web server: apache2.
INIT: no more processes left in this runlevel

I don't think that issue with /dev/urandom would cause the problem, as Webmin can live without it.

I thought of one more debugging thing you can try - on the problem system, edit /etc/init.d/webmin and change the line :

/etc/webmin/start

to :

strace -o /tmp/webmin-strace.txt -f /etc/webmin/start

Then reboot, and if the /tmp/webmin-strace.txt file exists email it to me at jcameron@webmin.com

I see:

start=/etc/webmin/start

I changed to this:

strace -o /tmp/webmin-strace.txt -f /etc/webmin/start

then this:

start=strace -o /tmp/webmin-strace.txt -f /etc/webmin/start

None produced the /tmp/webmin-strace.txt file. Did I do that correctly?

Ok, it looks like /etc/init.d/webmin isn't even being run ...

I have no idea how that could be the case though.

Hi Jamie,

Was I supposed to start up webmin manually first. As part of the last test, I removed the line etc/webmin/start from the /etc/rc.local file.

I am not sure if this is related. But I tried to upload using the webmin file manager. I keep getting the below error (tried on two different computers):

"Failed to list /home/test : java.security.AccessControlException : access denied (java.net.SocketPermission 192.168.1.15:10000 connect,resolve"

Any idea?

Removing /etc/webmin/start from /etc/rc.local wouldn't cause /etc/init.d/webmin to not be run..

I thunk the best solution is just to stick with starting from /etc/rc.local .

Thanks Jamie. I agree that the best solution is to stick with starting from /etc/rc.local.

What do you think of the below error? Could it be related to Webmin miniserver? I get this message every time trying to upload a file through the File Manager via Webmin.

"Failed to list /home/test : java.security.AccessControlException : access denied (java.net.SocketPermission 192.168.1.15:10000 connect,resolve"

That is unusual, as the IP appears to be on an internal network.

Are you connecting to your Virtualmin system from the same internal IP network?

Please don't pay attention to the IP. I am not listing the real IP in this post.

Here's what I found out. If I access Virtualmin/webmin from an internal network, the file uploaded fine via webmin file manager.

When I access it from external and go through the PIX firewall, this message came up. Does this mean we can rule out that it's not the same problem as we are encountering with Webmin not starting.

Also, if it's not related (as it doesn't seem to be), I would appreciate any pointer as to why we are seeing this error. Thanks again for your help. Happy Friday.

That message might be related to your firewall. Is it a proxy, reverse NAT device, or something else?

It's a Cisco PIX with standard NAT config. Previously, I was able to upload just fine. This is now happening about the same time as the Webmin issue under OpenVZ. But as I can see, it make no logical sense that Webmin has anything to do with this java error. Any help pointing me in the right direction is greatly appreciated. Thanks again.

OK, I remember the cause of this now .. another user reported it once before. It can happen due to an odd interaction between javascript and java. It is pretty harmless , as it only prevents the directory from being refreshed after uploading a file .. but it will be fixed in the next Webmin release.

Hi Jamie,

Thanks for pointing that out. I really appreciate it. This is surprising because I really thought it has nothing to do with Webmin and more of because of the firewall. But then again, my firewall is just a basic NAT device.

This does not happen when uploading from a local network. It only happen when I accessed Webmin from outside the firewall.

Yeah, there are rules that control what a java function is allowed to do when called back from javascript, in order to prevent un-trusted javascript mis-using functions in a trusted java applet. I don't fully understand the scope of these rules, but they come into play in an annoying way in this case ..

One last thing. I notice when accessing Webmin, the browser doesn't have an option to get the SSL certificate. Is there anyway to generate one for the Webmin server? Thanks.

You can generate or install a new cert at Webmin -> Webmin Configuration -> SSL Encryption.

Automatically closed -- issue fixed for 2 weeks with no activity.