Hi
Yesterday we have a catastrophic failure of hardware and have managed to recover but ended up using a previous kernel during boot. This resulted in SELinux relabelling the entire file structure and took some 4-5 hours to complete.
I cannot afford to repeat this procedure as it's a live production server, so I need to debug a problem preventing Webmin from starting.
As it stands, I have a Centos 6.4 server running a number of cloudmin VMs via KVM. We have both Cloudmin Pro and Virtualmin Pro licenses. The host system is running, and our Cloudmin master (on another machine) is able to connect and manage the VMs via SSH, which are all also running without issue.
Unfortunately though I cannot make Webmin start on this host server, and neither can I find any logs with relevant info as to why it's failing.
SELinux is currently in permissive mode.
ps aux | grep webmin and psaux | grep miniserv return no results.
Both service webmin restart and /etc/init.d/webmin restart just return nothing to the shell and have no effect on any log files in /etc/webmin/
Where else should I look for clues to dig into this?
Thanks in advance.
Comments
Submitted by andreychek on Thu, 02/12/2015 - 09:43 Comment #1
Howdy -- are you seeing any messages appear in /var/webmin/miniserv.error?
Submitted by VirtualNoob on Thu, 02/12/2015 - 10:06 Comment #2
Yeah, here's the output:
Error: Failed to open /etc/webmin/servers/1371650214345260.serv for writing : No such file or directoryError
-----
Failed to open /etc/webmin/servers/1371650214345260.serv for writing : No such file or directory
-----
Error: Failed to open /etc/webmin/server-manager/alert-status for writing : No such file or directory
Error
-----
Failed to open /etc/webmin/server-manager/alert-status for writing : No such file or directory
-----
However, the modified time on the file is yesterday afternoon despite trying to restart webmin today several times.
Submitted by andreychek on Thu, 02/12/2015 - 10:14 Comment #3
Hmm, what is the output of these commands:
ls -la /etc/webminls -la /etc/webmin/servers/
ls -la /etc/webmin/server-manager/alert-status/
At any point in your troubleshooting yesterday, were the Webmin related packages uninstalled or reinstalled?
Submitted by VirtualNoob on Thu, 02/12/2015 - 10:25 Comment #4
No webmin packages were installed or uninstalled during troubleshooting. Outputs below...
# ls -la /etc/webmintotal 772
drwxr-xr-x. 125 root root 4096 Jan 10 2014 .
drwxr-xr-x. 77 root root 4096 Feb 12 01:28 ..
drwx--x--x. 2 root bin 4096 Feb 22 2013 acl
drwx--x--x. 2 root bin 4096 Feb 22 2013 adsl-client
drwx--x--x. 2 root bin 4096 Feb 22 2013 ajaxterm
drwx--x--x. 2 root bin 4096 Feb 22 2013 apache
drwx--x--x. 2 root bin 4096 Feb 22 2013 at
drwx--x--x. 2 root bin 4096 Feb 22 2013 backup-config
drwx--x--x. 2 root bin 4096 Feb 22 2013 bacula-backup
drwx--x--x. 2 root bin 4096 Jun 19 2013 bandwidth
drwx--x--x. 2 root bin 4096 Jul 22 2013 bind8
drwx--x--x. 2 root bin 4096 Feb 22 2013 burner
drwx--x--x. 2 root bin 4096 Feb 22 2013 cfengine
drwx--x--x. 2 root bin 4096 Feb 22 2013 change-user
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-copy
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-cron
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-passwd
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-shell
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-software
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-useradmin
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-usermin
drwx--x--x. 2 root bin 4096 Feb 22 2013 cluster-webmin
-rw-r--r--. 1 root root 573 Nov 30 2013 config
drwx--x--x. 2 root bin 4096 Nov 30 2013 cpan
drwx--x--x. 2 root bin 4096 Feb 22 2013 cron
drwx--x--x. 2 root bin 4096 Feb 22 2013 custom
-rw-r--r--. 1 root root 109 Mar 13 2013 custom-lang
drwx--x--x. 2 root bin 4096 Feb 22 2013 dfsadmin
drwx--x--x. 2 root bin 4096 Jun 19 2013 dhcpd
drwx--x--x. 2 root bin 4096 Feb 22 2013 dnsadmin
drwx--x--x. 2 root bin 4096 Feb 22 2013 dovecot
drwx--x--x. 2 root bin 4096 Feb 22 2013 exim
drwx--x--x. 2 root bin 4096 Feb 22 2013 exports
drwx--x--x. 2 root bin 4096 Feb 22 2013 fdisk
drwx--x--x. 2 root bin 4096 Feb 22 2013 fetchmail
drwx--x--x. 2 root bin 4096 Feb 22 2013 file
drwx--x--x. 2 root bin 4096 Feb 22 2013 filter
drwx--x--x. 2 root bin 4096 Feb 22 2013 firewall
drwx--x--x. 2 root bin 4096 Feb 22 2013 frox
drwx--x--x. 2 root bin 4096 Feb 22 2013 fsdump
drwx--x--x. 2 root bin 4096 Feb 22 2013 grub
drwx--x--x. 2 root bin 4096 Feb 22 2013 heartbeat
drwx--x--x. 2 root bin 4096 Feb 22 2013 htaccess-htpasswd
drwx--x--x. 2 root bin 4096 Feb 22 2013 idmapd
drwx--x--x. 2 root bin 4096 Feb 22 2013 inetd
drwx--x--x. 2 root bin 4096 Feb 22 2013 init
drwx--x--x. 2 root bin 4096 Feb 22 2013 inittab
-rw-r--r--. 1 root root 1265 Nov 30 2013 installed.cache
drwx--x--x. 2 root bin 4096 Feb 22 2013 ipfilter
drwx--x--x. 2 root bin 4096 Feb 22 2013 ipfw
drwx--x--x. 2 root bin 4096 Feb 22 2013 ipsec
drwx--x--x. 2 root bin 4096 Feb 22 2013 iscsi-client
drwx--x--x. 2 root bin 4096 Feb 22 2013 iscsi-server
drwx--x--x. 2 root bin 4096 Feb 22 2013 iscsi-target
drwx--x--x. 2 root bin 4096 Feb 22 2013 jabber
drwx--x--x. 2 root bin 4096 Feb 22 2013 krb5
drwx--x--x. 2 root bin 4096 Feb 22 2013 ldap-client
drwx--x--x. 2 root bin 4096 Feb 22 2013 ldap-server
drwx--x--x. 2 root bin 4096 Feb 22 2013 ldap-useradmin
drwx--x--x. 2 root bin 4096 Feb 22 2013 lilo
drwx--x--x. 2 root bin 4096 Apr 7 2013 logrotate
drwx------. 2 root root 16384 Jan 10 2014 lost+found
drwx--x--x. 2 root bin 4096 Feb 22 2013 lpadmin
drwx--x--x. 2 root bin 4096 Jan 10 2014 lvm
drwx--x--x. 2 root bin 4096 Mar 27 2013 mailboxes
drwx--x--x. 2 root bin 4096 Feb 22 2013 mailcap
drwx--x--x. 2 root bin 4096 Feb 22 2013 majordomo
drwx--x--x. 2 root bin 4096 Feb 22 2013 man
-rw-r--r--. 1 root root 1869 Jul 11 2013 miniserv.cert
-rw-r--r--. 1 root root 1390 Jul 11 2013 miniserv.chain
-rw-------. 1 root bin 997 Feb 11 22:56 miniserv.conf
-rw-------. 1 root bin 1674 Jul 11 2013 miniserv.pem
-rw-------. 1 root bin 346 Jul 24 2013 miniserv.users
-rw-r--r--. 1 root root 169974 Nov 30 2013 module.infos.cache
drwx--x--x. 2 root bin 4096 Feb 22 2013 mon
drwx--x--x. 2 root bin 4096 Jan 10 2014 mount
drwx--x--x. 2 root bin 4096 Feb 22 2013 mysql
drwx--x--x. 2 root bin 4096 Feb 22 2013 net
drwx--x--x. 2 root bin 4096 Feb 22 2013 nis
drwx--x--x. 2 root bin 4096 Feb 22 2013 openslp
drwx--x--x. 2 root bin 4096 Feb 11 12:43 package-updates
drwx--x--x. 2 root bin 4096 Feb 22 2013 pam
drwx--x--x. 2 root bin 4096 Feb 22 2013 pap
drwx--x--x. 2 root bin 4096 Feb 22 2013 passwd
-rw-r--r--. 1 root root 14 Feb 22 2013 perl-path
drwx--x--x. 2 root bin 4096 Feb 22 2013 phpini
drwx--x--x. 2 root bin 4096 Feb 22 2013 postfix
drwx--x--x. 2 root bin 4096 Feb 22 2013 postgresql
drwx--x--x. 2 root bin 4096 Feb 22 2013 ppp-client
drwx--x--x. 2 root bin 4096 Feb 22 2013 pptp-client
drwx--x--x. 2 root bin 4096 Feb 22 2013 pptp-server
drwx--x--x. 2 root bin 4096 Mar 4 2013 proc
drwx--x--x. 2 root bin 4096 Feb 22 2013 procmail
drwx--x--x. 2 root bin 4096 Feb 22 2013 proftpd
drwx--x--x. 2 root bin 4096 Feb 22 2013 pserver
drwx--x--x. 2 root bin 4096 Feb 22 2013 qmailadmin
drwx--x--x. 2 root bin 4096 Feb 22 2013 quota
drwx--x--x. 2 root bin 4096 Jun 19 2013 raid
-rwxr-xr-x. 1 root root 166 Nov 30 2013 reload
-rwxr-xr-x. 1 root root 48 Nov 30 2013 restart
drwx--x--x. 2 root bin 4096 Feb 22 2013 samba
drwx--x--x. 2 root bin 4096 Feb 22 2013 sarg
drwxr-xr-x. 2 root root 12288 Feb 12 01:14 security-updates
drwx--x--x. 2 root bin 4096 Feb 22 2013 sendmail
drwx--x--x. 2 root bin 4096 Feb 22 2013 sentry
drwxr-xr-x. 14 root root 4096 Feb 12 01:14 server-manager
drwx--x--x. 2 root bin 4096 Feb 12 01:15 servers
drwx--x--x. 2 root bin 4096 Feb 22 2013 shell
drwx--x--x. 2 root bin 4096 Feb 22 2013 shorewall
drwx--x--x. 2 root bin 4096 Feb 22 2013 shorewall6
drwx--x--x. 2 root bin 4096 Feb 22 2013 smart-status
drwx--x--x. 2 root bin 4096 Feb 22 2013 smf
drwx--x--x. 2 root bin 4096 Feb 22 2013 software
drwx--x--x. 2 root bin 4096 Feb 22 2013 spam
drwx--x--x. 2 root bin 4096 Feb 22 2013 squid
drwx--x--x. 2 root bin 4096 Jul 22 2013 sshd
-rwxr-xr-x. 1 root root 237 Nov 30 2013 start
drwx--x--x. 2 root bin 4096 Feb 22 2013 status
-rwxr-xr-x. 1 root root 159 Nov 30 2013 stop
drwx--x--x. 2 root bin 4096 Feb 22 2013 stunnel
drwx--x--x. 2 root bin 4096 Feb 22 2013 syslog
drwx--x--x. 2 root bin 4096 Feb 22 2013 syslog-ng
drwx--x--x. 3 root bin 4096 Feb 11 12:43 system-status
drwx--x--x. 2 root bin 4096 Feb 22 2013 tcpwrappers
drwx--x--x. 2 root bin 4096 Feb 22 2013 telnet
drwx--x--x. 2 root bin 4096 Sep 18 2013 time
drwx--x--x. 2 root bin 4096 Feb 22 2013 tunnel
-rwxr-xr-x. 1 root root 445 Nov 30 2013 uninstall.sh
drwx--x--x. 2 root bin 4096 Feb 22 2013 updown
drwx--x--x. 2 root bin 4096 Feb 22 2013 useradmin
drwx--x--x. 2 root bin 4096 Feb 22 2013 usermin
-rw-r--r--. 1 root root 12 Feb 22 2013 var-path
-rw-r--r--. 1 root root 6 Nov 30 2013 version
drwx--x--x. 2 root bin 4096 Feb 22 2013 vgetty
drwxr-xr-x. 2 root root 4096 Mar 13 2013 virtualmin-password-recovery
drwx--x--x. 2 root bin 4096 Feb 22 2013 webalizer
drwx--x--x. 2 root bin 4096 Feb 3 17:38 webmin
-rw-------. 1 root root 5264 Jul 24 2013 webmin.acl
drwx--x--x. 3 root bin 4096 Feb 22 2013 webmincron
-rw-r--r--. 1 root root 0 Mar 12 2013 webmin.groups
drwx--x--x. 2 root bin 4096 Feb 22 2013 webminlog
drwxr-xr-x. 123 root root 4096 Nov 30 2013 webmin-native
drwx--x--x. 2 root bin 4096 Feb 22 2013 wuftpd
drwx--x--x. 2 root bin 4096 Feb 22 2013 xinetd
# ls -la /etc/webmin/serverstotal 56
drwx--x--x. 2 root bin 4096 Feb 12 01:15 .
drwxr-xr-x. 125 root root 4096 Jan 10 2014 ..
-rw-------. 1 root root 2091 May 4 2014 1362431184119870.serv
-rw-------. 1 root root 2581 Feb 11 22:02 1363122013371480.serv
-rw-------. 1 root root 2261 Feb 11 21:18 136317062278770.serv
-rw-------. 1 root root 2251 Feb 11 21:25 136438567319630.serv
-rw-r--r--. 1 root root 6 Feb 12 01:15 136438567319630.serv.lock
-rw-------. 1 root root 892 Feb 11 21:18 1365183175270530.serv
-rw-------. 1 root root 912 Feb 11 21:18 1365183319271700.serv
-rw-------. 1 root root 2212 Feb 11 21:18 1365756797171590.serv
-rw-------. 1 root root 2277 Feb 11 21:18 1367940912615490.serv
-rw-------. 1 root root 2287 Feb 11 21:18 1371650214345260.serv
-rw-------. 1 root root 1148 Feb 11 21:18 1374514084268770.serv
-rw-------. 1 root bin 124 Nov 30 2013 config
# ls -la /etc/webmin/server-manager/alert-status-rw-r--r--. 1 root root 0 Feb 11 12:38 /etc/webmin/server-manager/alert-status
Submitted by andreychek on Thu, 02/12/2015 - 10:32 Comment #5
Okay, here is what I would try --
mv /etc/webmin/server-manager/alert-status /etc/webmin/server-manager/alert-status.origrm /etc/webmin/servers/136438567319630.serv.lock
Once you do that, are you able to start Webmin?
If not, what is the output of this command:
find /etc/webmin -name "*.lock"Submitted by VirtualNoob on Thu, 02/12/2015 - 10:44 Comment #6
Thanks
I ran those commands and tried to start webmin, no output to the shell and webmin not started.
find /etc/webmin -name "*.lock"/etc/webmin/webmin-native/servers/1365183175270530.serv.lock
Submitted by andreychek on Thu, 02/12/2015 - 10:48 Comment #7
Okay, try deleting that lock file too.
Once you do, try restarting Webmin again.
If that doesn't help -- I think we took care of everything in the logs, is anything new showing up now in that miniserv.error file?
Submitted by VirtualNoob on Thu, 02/12/2015 - 10:54 Comment #8
OK, lock file removed and service started / restarted. Still no joy.
Nothing new in the miniserv.error file either, still the same entries from yesterday. Any other ideas?
Cheers
Submitted by andreychek on Thu, 02/12/2015 - 11:01 Comment #9
What output do you receive if you run this command:
strace /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.confIf the strace command isn't found, you may need to install it first with the command "yum install strace".
Submitted by VirtualNoob on Thu, 02/12/2015 - 11:14 Comment #10
output:
:::::truncated:::::write(2, "Failed to bind to port 10000 : A"..., 54Failed to bind to port 10000 : Address already in use
) = 54
write(3, "Reading crons from /etc/webmin/w"..., 545) = 545
write(2, "Could not listen on any ports", 29Could not listen on any ports) = 29
close(6) = 0
close(4) = 0
close(3) = 0
exit_group(1) = ?
[root@kvmhost webmin]#
Then I checked netstat and found it was webmin listening on 10000. It's currently running and I can connect through the web interface.
So thanks very much for helping me work through it all. Would you suggest any checks or housekeeping to ensure it's 100% happy again?
Submitted by andreychek on Thu, 02/12/2015 - 11:20 Comment #11
You may want to go into System Settings -> Re-Check Config to make sure it doesn't detect any problems there.
Also, I'd suggest just verifying that your websites work and that you can send/receive email.
If all that checks out -- those are usually the most important things, other things can be taken care of as they come up.
Submitted by VirtualNoob on Thu, 02/12/2015 - 11:25 Comment #12
Will do. Once again thanks very much for the prompt responses and assisting with this.
Top quality service as always.
Submitted by andreychek on Thu, 02/12/2015 - 11:34 Comment #13
Sure thing, we're glad to hear it's working for you now!
Submitted by Issues on Thu, 02/26/2015 - 11:40 Comment #14
Automatically closed -- issue fixed for 2 weeks with no activity.