VM not start properly at boot (Ubuntu)

Submitted by allitsk on Sun, 08/24/2014 - 14:17

OS Ubuntu 12.04 LTS (fully upgraded)

Cloudmin started VM by init script in /etc/init.d/cloudmin-kvm, but properly start only first VM in init script. Second VM start by qemu-kvm, but vnc password setup is not push by nc.pl, error in console log:

Failed to connect : Connection refused at /usr/bin/nc.pl line 19.

After system boot I can push vnc setup manually by command: /usr/bin/nc.pl --sleep 3 localhost 40002 <vmname.domain.local.monitor >>vmname.domain.local.local.console 2>&1 and VM is continuing, starting.

I test it on 2 virtual hypervisors (Ubuntu 12.04 and 14.04) with same result. Try add some sleep commands in init script, but with no results.

Status: 
Active

Comments

Submitted by allitsk on Sun, 08/24/2014 - 14:44

I test start manually VM startup command, but with same result:

root@domainvirthost:~# (sleep 20 ; ionice -n4 /usr/bin/kvm -name vmname.domain.local -m 1024 -drive file=/dev/datavg/vmname_domain_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/vmname_domain_local_swap,media=disk,index=1,cache=off,if=virtio -boot c -net tap,vlan=0,script=vmname.domain.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:B6:7C:2F,model=virtio -vnc :1,password -usbdevice tablet -monitor tcp:127.0.0.1:40002,server -smp 1) >vmname.domain.local.console 2>&1 </dev/null & sleep 3 ; /usr/bin/nc.pl --sleep 3 localhost 40002 <vmname.domain.local.monitor >>vmname.domain.local.console 2>&1 [1] 3455

root@domainvirthost:~# cat vmname.domain.local.console Failed to connect : Connection refused at /usr/bin/nc.pl line 19.

root@domainvirthost:~# /usr/bin/nc.pl --sleep 3 localhost 40002 >vmname.domain.local.local.console 2>&1 and VM is continuing, starting.

root@domainvirthost:~# cat vmname.domain.local.console qemu-kvm: boot=on|off is deprecated and will be ignored. Future versions will reject this parameter. Please update your scripts. QEMU 1.0 monitor - type 'help' for more information (qemu) change vnc password Password: ****** (qemu)

Submitted by JamieCameron on Sun, 08/24/2014 - 15:06

Can you start the second or later VMs from the Cloudmin UI after bootup completes?

Also, how many CPU cores does your system have? it is possible that startup is slow enough for the 2nd VM due to load from the 1st that it hasn't reached a state where the nc command can supply the VNC password fast enough.

Submitted by allitsk on Mon, 08/25/2014 - 02:39

System has slow Intel G3220 with two cores, but it was working before reinstalation of this system (used Cloudmin also, probably older version)

I test starting VM after full boot, no load. Start VM from Cloudmin web interface is working fine as to be. But start VM by same command that is in init script (see my comment #1) is not working and result is same, nc command can't reach console. No other VM is running!

I think that some sleep command is ignored.

Submitted by JamieCameron on Mon, 08/25/2014 - 13:20

What if you try starting it from the command line, but with --sleep 3 replaced by --sleep 30 ?

Submitted by allitsk on Mon, 08/25/2014 - 17:02

I ran this with same result, but executing this command take only 3 seconds, not >30 seconds.

(sleep 10 ; ionice -n4 /usr/bin/kvm -name vmname.domain.local -m 1024 -drive file=/dev/datavg/vmname_domain_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/vmname_domain_local_swap,media=disk,index=1,cache=off,if=virtio -boot c -net tap,vlan=0,script=vmname.domain.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:B6:7C:2F,model=virtio -vnc :1,password -usbdevice tablet -monitor tcp:127.0.0.1:40001,server -smp 1) >vmname.domain.local.console 2>&1 >vmname.domain.local.console 2>&1 </dev/null & sleep 3 ; /usr/bin/nc.pl --sleep 30 localhost 40001 <vmname.domain.local.monitor >>vmname.domain.local.console 2>&1

After that I try other sleep and this is working:

(sleep 10 ; ionice -n4 /usr/bin/kvm -name vmname.domain.local -m 1024 -drive file=/dev/datavg/vmname_domain_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/vmname_domain_local_swap,media=disk,index=1,cache=off,if=virtio -boot c -net tap,vlan=0,script=vmname.domain.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:B6:7C:2F,model=virtio -vnc :1,password -usbdevice tablet -monitor tcp:127.0.0.1:40001,server -smp 1) >vmname.domain.local.console 2>&1 >vmname.domain.local.console 2>&1 </dev/null & sleep 30 ; /usr/bin/nc.pl --sleep 3 localhost 40001 <vmname.domain.local.monitor >>vmname.domain.local.console 2>&1

But I think that first sleep (after bracket) is ignored, but this sleep is delay between VMs booting.

Submitted by JamieCameron on Mon, 08/25/2014 - 17:41

The sleep in that line is to introduce a delay between starting VMs.

Can you post the full /etc/init.d/cloudmin-kvm script from your system?

Submitted by allitsk on Tue, 08/26/2014 - 00:39

This one is NOT wokring (original):

#!/bin/sh
# chkconfig: 235 99 10
# description: Start KVM instances created by Cloudmin
#
### BEGIN INIT INFO
# Provides: cloudmin-kvm
# Required-Start: $network $syslog
# Required-Stop: $network
# Default-Start: 2 3 5
# Default-Stop: 0 1 6
# Description: Start KVM instances created by Cloudmin
### END INIT INFO

case "$1" in
'start')
        (sleep 10 ; ionice -n4 /usr/bin/kvm -name companyserver\.company\.local -m 512 -drive file=/dev/datavg/companyserver_company_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/companyserver_company_local_swap,media=disk,index=1,cache=off,if=virtio -drive index=2,media=disk,file=/dev/datavg/companyserver_company_local_3_img,if=virtio,cache=none -boot c -net tap,vlan=0,script=/virt/companyserver.company.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:8E:D4:48,model=virtio -vnc :2,password -usbdevice tablet -monitor tcp:127.0.0.1:40001,server -smp 2,cores=2) >/virt/companyserver.company.local.console 2>&1 </dev/null & sleep 3 ; /usr/bin/nc.pl --sleep 3 localhost 40001 </virt/companyserver.company.local.monitor >>/virt/companyserver.company.local.console 2>&1
        (sleep 20 ; ionice -n4 /usr/bin/kvm -name companyunifi\.company\.local -m 1024 -drive file=/dev/datavg/companyunifi_company_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/companyunifi_company_local_swap,media=disk,index=1,cache=off,if=virtio -boot c -net tap,vlan=0,script=/virt/companyunifi.company.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:B6:7C:2F,model=virtio -vnc :1,password -usbdevice tablet -monitor tcp:127.0.0.1:40002,server -smp 1) >/virt/companyunifi.company.local.console 2>&1 </dev/null & sleep 3 ; /usr/bin/nc.pl --sleep 3 localhost 40002 </virt/companyunifi.company.local.monitor >>/virt/companyunifi.company.local.console 2>&1
        RETVAL=$?
        ;;
'stop')
        killall qemu-kvm kvm
        RETVAL=$?
        ;;
'restart')
        $0 stop ; $0 start
        RETVAL=$?
        ;;
*)
        echo "Usage: $0 { start | stop | restart }"
        RETVAL=1
        ;;
esac
exit $RETVAL

Submitted by allitsk on Tue, 08/26/2014 - 00:44

This one is wokring (changed in sleep interval before nc command):

#!/bin/sh
# chkconfig: 235 99 10
# description: Start KVM instances created by Cloudmin
#
### BEGIN INIT INFO
# Provides: cloudmin-kvm
# Required-Start: $network $syslog
# Required-Stop: $network
# Default-Start: 2 3 5
# Default-Stop: 0 1 6
# Description: Start KVM instances created by Cloudmin
### END INIT INFO

case "$1" in
'start')
        (sleep 10 ; ionice -n4 /usr/bin/kvm -name companyserver\.company\.local -m 512 -drive file=/dev/datavg/companyserver_company_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/companyserver_company_local_swap,media=disk,index=1,cache=off,if=virtio -drive index=2,media=disk,file=/dev/datavg/companyserver_company_local_3_img,if=virtio,cache=none -boot c -net tap,vlan=0,script=/virt/companyserver.company.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:8E:D4:48,model=virtio -vnc :2,password -usbdevice tablet -monitor tcp:127.0.0.1:40001,server -smp 2,cores=2) >/virt/companyserver.company.local.console 2>&1 </dev/null & sleep 30 ; /usr/bin/nc.pl --sleep 3 localhost 40001 </virt/companyserver.company.local.monitor >>/virt/companyserver.company.local.console 2>&1
        (sleep 20 ; ionice -n4 /usr/bin/kvm -name companyunifi\.company\.local -m 1024 -drive file=/dev/datavg/companyunifi_company_local_img,media=disk,index=0,if=virtio,boot=on,cache=none -drive file=/dev/datavg/companyunifi_company_local_swap,media=disk,index=1,cache=off,if=virtio -boot c -net tap,vlan=0,script=/virt/companyunifi.company.local-eth0.sh -net nic,vlan=0,macaddr=02:54:00:B6:7C:2F,model=virtio -vnc :1,password -usbdevice tablet -monitor tcp:127.0.0.1:40002,server -smp 1) >/virt/companyunifi.company.local.console 2>&1 </dev/null & sleep 30 ; /usr/bin/nc.pl --sleep 3 localhost 40002 </virt/companyunifi.company.local.monitor >>/virt/companyunifi.company.local.console 2>&1
        RETVAL=$?
        ;;
'stop')
        killall qemu-kvm kvm
        RETVAL=$?
        ;;
'restart')
        $0 stop ; $0 start
        RETVAL=$?
        ;;
*)
        echo "Usage: $0 { start | stop | restart }"
        RETVAL=1
        ;;
esac
exit $RETVAL

Submitted by JamieCameron on Tue, 08/26/2014 - 16:58

Ok, I will address this in the next Cloudmin release by trying a bit longer to connect to the KVM monitor to send the VNC password command.

Submitted by Issues on Tue, 09/09/2014 - 17:01

Automatically closed -- issue fixed for 2 weeks with no activity.

Submitted by allitsk on Wed, 09/10/2014 - 00:42

Is it fixed? Probably not, but issue was closed automatically...

Submitted by andreychek on Wed, 09/10/2014 - 00:59

Jamie mentioned in comment #9 that the issue you brought up would be corrected in the next Cloudmin version. If you continue to see problems after upgrading to the next Cloudmin version, feel free to let us know and we can look into that further.