Creation of VM's via iscsi-server fails

Submitted by fix on Wed, 07/10/2013 - 06:42

When I try to create a new VM on a xen-host which uses an iscsi-server as store the creation of the VM fails. The reason is, that cloudmin tries to create the initial lvm volume (or image if choosed) but does not find the installation image under /var/cloudmin-images on the ISCSI-SERVER instead of the xen-host. By manually copy the installation image to /var/cloudmin-images the creation works

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Wed, 07/10/2013 - 08:02

Thanks for the bug report! I'm passing this over to Jamie to take a look at. He's out of the country at the moment though, so there may be a delay before he can respond.

Submitted by JamieCameron on Wed, 07/10/2013 - 08:57

That sounds like a bug - what exact error message are you getting in Cloudmin when creating the VM?

Submitted by fix on Wed, 07/10/2013 - 18:05

Hi Jamie

Thank's for your quick answer. The error Messages vary on the iscsi-servers settings. First I tried to create the VM's on a LVM VG as suggested. Then the creation run as follows:

---------- quote of output while creating VM on LVM Volume -------------------------

Comment: All hostnames are 'privated' out. DNS resolution works correct

Copying 435.12 MB image file to host system sin.xyz.xy ..
.. already in host cache
Creating virtual system with Xen ..
.. creation started.

Waiting for creation to complete ..
.. creation has completed successfully.

Comment: The completing just takes a few seconds. I think it's here, where the error occurs,
but without any error message

Removing missing disks from fstab file ..
.. cleanup failed : Failed to read fstab file : Failed to mount /dev/disk/by-path/ip-10.18.0.82:3260-iscsi-iqn.2013-07.xy.xyz.zen:test0003_0_img-lun-1 on /mnt/xen-test0003 : mount: you must specify the filesystem type

Expanding filesystem to 3 GB ..
.. failed to find primary disk

Mounting new instance's filesystem ..
.. failed : Failed to mount /dev/disk/by-path/ip-10.18.0.82:3260-iscsi-iqn.2013-07.xy.xyz.zen:test0003_0_img-lun-1 on /mnt/xen-test0003 : mount: you must specify the filesystem type

Configuring kernel for Xen instance ..
.. setup to boot Xen system's kernel with PyGrub

---------------- end quote output ------------------------------------------

After the quoted output cloudmin finishes the creation with the normal steps like it would create a working VM, but can't boot it because it can't find the boot drive. It took me hours to figure out if I had a wrong format on the iscsi Volume or if something else is wrong. After all I tried to change the iscsi-server settings to use a directory instead of the LVM VG and a new error message appeared which leaded me to the assumption that cloudmin could look after the installation image on the iscsi-server itself.

The output as follows:

---------- quote of output while creating VM on fixed directory -------------------------

Copying 435.12 MB image file to host system sin.xyz.xy ..
.. already in host cache
Creating virtual system with Xen ..
.. creation started.

Waiting for creation to complete ..... creation failed : Failed to extract filesystem : gunzip: /var/cloudmin-images/xen-64-centos5.4-base.ext3.gz: No such file or directory

Comment: The creation stops here. VM does not appear in VM list as against creating on iscsi's LVM VG

---------------- end quote output ------------------------------------------

There's one funny thing. When I try to create a VM on the iscsi-servers LVM VG (which ends with a new VM in the VM's list) I can reset the new VM from image and this part works as expected, ending with a running VM.

After all this, I copied the installation image to the iscsi-server and everything works as expected, but only, if the settings of the xen host is set to cache the installation image in /var/cloudmin-images (or maybe another fixed directory). If the xen host is set to not cache the images the installation fails with above errors.

When I first time setup a iscsi-server to use with cloudmin (probably while cloudmin version 6.5 or something, but still with all hosts on Centos 5.x) everthing worked as expected, which let me think there must be something wrong on my new raid or iscsi-server.

I hope, all this information is usefull while investigating this issue. If you need any further infos, just ask for it.

Thank you kindly & regards

Fix

Submitted by JamieCameron on Wed, 07/10/2013 - 19:05

Yes, it looks like Cloudmin isn't copying the image to the iSCSI server for extraction.

In the sample output above, is sin.xyz.xy the hostname of the iSCSI server, or the machine the VM will be hosted on?

Submitted by fix on Wed, 07/10/2013 - 19:35

sin.xyz.xy ist the xen host where the the VM should created at
zen.xyz.xy ist the iscsi-server
yen.xyz.xy ist the cloudmin server

All 3 running centos 5.9 with a xen-kernel.

regards

Fix

Submitted by JamieCameron on Wed, 07/10/2013 - 22:06

Which Cloudmin version are you running there? We have made some improvements in the way images are copied to iSCSI servers recently that may be relevant here.

Submitted by fix on Thu, 07/11/2013 - 01:49

I run Cloudmin 7.1 on Webmin 1.6.3

Submitted by JamieCameron on Thu, 07/11/2013 - 03:55

Ok, I found a bug that can cause this - let me know if you'd like a pre-release version of Cloudmin that includes the fix.

Submitted by fix on Thu, 07/11/2013 - 04:08

Hi Jamie

Of course. I would be happy to test this out. You can use the Emailaddress in my profile to reach me. Can you see that email?

Regards

Fix

Submitted by JamieCameron on Thu, 07/11/2013 - 06:07

Sure, I will send you an RPM.

Submitted by fix on Thu, 07/11/2013 - 06:55

Installed the rpm, created a VM, worked flawlessly!

Many thanks for the quick help. You made me a happy day.

Regards Fix

Submitted by JamieCameron on Thu, 07/11/2013 - 08:16

Great! This will go into the next Cloudmin release.

Submitted by Issues on Thu, 07/25/2013 - 11:41

Automatically closed -- issue fixed for 2 weeks with no activity.