Xen - Error: Boot loader didn't return any data!

Submitted by RyanJohnson on Tue, 11/12/2013 - 15:07

Hi Guys,

I'm rebuilding my server and trialling Cloudmin. I've got over the first few hurdles but I've hit a wall and need some help.

I'm running Debian 7 64bit with Xen as the host and using the image "Debian Wheezy Xen instance with base OS" which I'm guessing is 32bit

When I first created the instance everything worked fine until I added some cores to the instance and rebooted it. Now I get the above error and can't boot the instance.

The output of /var/log/xen/xend.log is:

[2013-11-13 07:40:38 2553] DEBUG (XendDomainInfo:2498) XendDomainInfo.constructDomain
[2013-11-13 07:40:38 2553] DEBUG (balloon:187) Balloon: 2104996 KiB free; need 16384; done.
[2013-11-13 07:40:38 2553] DEBUG (XendDomain:476) Adding Domain: 17
[2013-11-13 07:40:38 2553] DEBUG (XendDomainInfo:2836) XendDomainInfo.initDomain: 17 256
[2013-11-13 07:40:38 29282] DEBUG (XendBootloader:113) Launching bootloader as ['/usr/lib/xen-4.1/bin/pygrub', '--output=/var/run/xend/boot/xenbl.16350', '-q', '/xen/duo.img'].
[2013-11-13 07:40:38 2553] ERROR (XendBootloader:214) Boot loader didn't return any data!
[2013-11-13 07:40:38 2553] ERROR (XendDomainInfo:488) VM start failed
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 474, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2838, in _initDomain
    self._configureBootloader()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 3285, in _configureBootloader
    bootloader_args, kernel, ramdisk, args)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendBootloader.py", line 215, in bootloader
    raise VmError, msg
VmError: Boot loader didn't return any data!
[2013-11-13 07:40:38 2553] DEBUG (XendDomainInfo:3071) XendDomainInfo.destroy: domid=17
[2013-11-13 07:40:38 2553] DEBUG (XendDomainInfo:2406) No device model
[2013-11-13 07:40:38 2553] DEBUG (XendDomainInfo:2408) Releasing devices
[2013-11-13 07:40:38 2553] ERROR (XendDomainInfo:108) Domain construction failed
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 106, in create
    vm.start()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 474, in start
    XendTask.log_progress(31, 60, self._initDomain)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 2838, in _initDomain
    self._configureBootloader()
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", line 3285, in _configureBootloader
    bootloader_args, kernel, ramdisk, args)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendBootloader.py", line 215, in bootloader
    raise VmError, msg
VmError: Boot loader didn't return any data!

I've tried switching the boot methods. When I tied the Pv-Grub I get change failed : Pv-Grub command /usr/lib/xen/boot/pv-grub-x86_32.gz was not found on the host system

These are the kernels reported by Cloudmin:

System type Xen Virtual system's kernel Linux version 3.2.0-4-686-pae for i686 Host system's kernel Linux version 3.2.0-4-amd64 for x86_64

I'm not sure of the exact steps that caused this .. butI did to an apt-get upgrade on both the host and guest prior

Status: 
Active

Comments

Submitted by JamieCameron on Tue, 11/12/2013 - 15:25

Which Xen version are you running there? We are currently looking into an issue that can trigger this with certain Xen 4.x releases.

Submitted by RyanJohnson on Tue, 11/12/2013 - 15:57

Thanks for getting back to me so quickly.

I'm not sure how to check the version as this is day two for me :) .. But these are some of the packages:

xen-hypervisor-4.1-amd64 4.1.4-3+deb7u1 xen-linux-system-3.2.0-4-amd64 3.2.51-1 xen-linux-system-amd64 3.2+46 xen-system-amd64 4.1.4-3+deb7u1

Do you know if there's a manual workaround for the time being? I was going to attempt compiling xen to get the /usr/lib/xen/boot/pv-grub-x86_32.gz file from this wiki http://wiki.xen.org/wiki/PvGrub But this may stuff things up ??

Ryan

Submitted by JamieCameron on Tue, 11/12/2013 - 16:15

I can send you a fix for this specific issue (of VM creation failing), but it doesn't completely solve the deeper issue - that is still being worked on. Let me know if you are interested.

Submitted by RyanJohnson on Tue, 11/12/2013 - 16:22

Yes please Jamie. I need to quickly get off my old relic as it's failing ;)

Submitted by RyanJohnson on Tue, 11/12/2013 - 17:37

BTW .. The creation works fine. I had an instance running all day until I decided to increase the cpu's and reboot it.

When you say deeper issue .. That sounds serious. Is this a xen related issue or Cloudmin or both? I've not found much out there about this particular error.

Submitted by RyanJohnson on Wed, 11/13/2013 - 20:23

Hi Jamie,

I've managed to to mount the xen image from Dom0. Do you mind pointing me in the right direction for getting the right kernel into the instance or a way to get the instance back up?

Submitted by JamieCameron on Thu, 11/14/2013 - 00:19

Sorry, I missed your comments on this bug. I can send you a patch, I just need to know which Linux distribution you are running your Cloudmin master on, and if you have the Pro or GPL version.

Submitted by RyanJohnson on Thu, 11/14/2013 - 00:24

Hi Jamie,

No probs!

I'm using Cloudmin GPL on Debian Wheezy 64bit .. The guest is Debian Wheezy 32bit

Submitted by JamieCameron on Thu, 11/14/2013 - 19:02

So one work-around to fix your currently broken VM is to shut down the VM, SSH in as root and run :

cd /xen
xm delete yourvmname
xm new yourvmname.cfg

This won't actually delete the VM, just re-sync it with the .cfg file. You should then be able to start it. Let me know how it goes..

Submitted by RyanJohnson on Thu, 11/14/2013 - 19:19

Not looking good .. First I did this:

root@xen:/xen# xm delete duo
Error: <Fault 3: 'duo'>
Usage: xm delete <DomainName>

And then this:

root@xen:/xen# xm new duo.cfg
Unexpected error: <type 'exceptions.ImportError'>

Please report to xen-devel@lists.xensource.com
Traceback (most recent call last):
  File "/usr/lib/xen-4.1/bin/xm", line 8, in <module>
    main.main(sys.argv)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xm/main.py", line 3983, in main
    _, rc = _run_cmd(cmd, cmd_name, args)
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xm/main.py", line 4007, in _run_cmd
    return True, cmd(args)
  File "<string>", line 1, in <lambda>
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xm/main.py", line 1518, in xm_importcommand
    cmd = __import__(command, globals(), locals(), 'xen.xm')
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xm/new.py", line 26, in <module>
    from xen.xm.xenapi_create import *
  File "/usr/lib/xen-4.1/bin/../lib/python/xen/xm/xenapi_create.py", line 24, in <module>
    from lxml import etree
ImportError: No module named lxml

This is my cfg:

root@xen:/xen# cat duo.cfg
memory = 2048
maxmem = 4096
name = 'duo'
vif = ['ip=MYIPADDRESS,mac=00:16:3e:86:93:CD']
address = 'MYIPADDRESS'
netmask = '255.255.255.0'
disk = ['file:/xen/duo.img,xvda1,w']
vnc = 1
vnclisten = "0.0.0.0"
vncunused = 1
vncpasswd = "MYPASSWORD"
vfb = ['type=vnc,vncunused=1,vncpasswd=MYPASSWORD,vnclisten=0.0.0.0']
vcpus = 2
bootloader = "/usr/lib/xen-4.1/bin/pygrub"
pae = 1

Submitted by JamieCameron on Thu, 11/14/2013 - 22:40

Does the Xen VM named duo actually exist? You can check by running the command xm list duo

Submitted by RyanJohnson on Thu, 11/14/2013 - 22:56

Scrub that last comment .. I needed to "apt-get install python-lxml" first.. but still no luck . Although at least now it shows up in xm list

This is the resulting errors

From Cloudmin

failed : Error: VM name 'duo' already exists Using config file "/xen/duo.cfg".

From xm start duo

xm start duo
Error: Boot loader didn't return any data!

Submitted by RyanJohnson on Fri, 11/15/2013 - 00:16

Yes . it looks like it's there:

Name                                        ID   Mem VCPUs      State   Time(s)
duo                                             2048     2                 0.0

When I list them all, duo is the only one without an ID and other things

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  7144     8     r-----   1215.5
duo                                             2048     2                 0.0
make                                         3   256     1     -b----     11.0

Submitted by JamieCameron on Fri, 11/15/2013 - 17:25

FYI, I am still working on the underlying issue here - I will update this bug when fixed.

Submitted by RyanJohnson on Mon, 11/18/2013 - 16:26

Thanks for letting me know .. in the meantime .. If I Switch to KVM will I have the same issues?

If I do switch, is removing Xen, Cloudmin and starting again possible?

Submitted by JamieCameron on Mon, 11/18/2013 - 17:08

Ok, I have a fix now for these Xen 4.x issues. Anyone who'd like to try it out, let me know if you are running Cloudmin GPL or Pro, and on what Linux distribution.

Submitted by RyanJohnson on Mon, 11/18/2013 - 17:16

I'll try it out please Jamie.

I'm using Cloudmin GPL + Debian 7 (wheezy) 64bit

Submitted by JamieCameron on Tue, 11/19/2013 - 00:25

Ok, updated package sent.