My cloudmin install is quickly becoming unuseable.

Submitted by cruiskeen on Sun, 09/27/2015 - 15:08

I put in a request a while back that has been dangling along, partly due to my own fault of not responding very quickly, but over the last few days I'm having really serious cloudmin problems that are causing me serious client issues.

I cannot apparently build new servers - they always fail complaining about not being able to extract the filesystem, or they fail when they try to mount the file system and modify it, complaining that there is no root file system in /etc/fstab.

I thought perhaps my custom images were broken, so I tried to save a new one from the custom model images I maintain - they both run (centos 6 and centos 7 images) and will reboot, but I now cannot save them any more -- when I try to save an image I get

An image cannot be created from this system : No root filesystem found in /etc/fstab

Now there IS a root filesystem in /etc/fstab ---

/dev/vda1 / ext3 grpquota,usrquota,rw 0 1 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/vdb1 swap swap defaults 0 0

I am seriously at a loss on this and the whole system is now fairly useless since I cannot create any new virtual servers, and I REALLY need to be able to do that.

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Sun, 09/27/2015 - 15:45

Seems like the problem is that Cloudmin cannot mount the filesystems of the VMs.

If you shut down a VM, SSH into the master as root and run cloudmin mount-system --host your-vm-name , does it report any error?

If it succeeds, you will need to run cloudmin umount-system --host your-vm-name before the VM can be booted again.

Submitted by cruiskeen on Sun, 09/27/2015 - 15:51

Hi Jamie --- what you're getting here is the result of a sysadmin in panic mode. Now that I've calmed down a little more I see that this is not everywhere and appears to mainly be affecting my single CentOS 7 server.

At the moment I'm still furiously restoring clients from backups -- one of our servers just tanked completely this morning and I'm trying to get people on other machines. So now that I'm actually starting to build some VPS servers again on the other machines I'm putting THIS issue on the back burner. YOu'll probably hear more from me tomorrow after I've got some of the flames put out. In the meantime, sorry to bother you on a Sunday.

More info later.

Submitted by JamieCameron on Sun, 09/27/2015 - 16:47

Ok, if you'd like me to login to your Cloudmin system and take a look at why it can't access filesystems, please let me know.

Submitted by cruiskeen on Mon, 09/28/2015 - 09:24

Well of course the cloudmin command to remote mount the file system works just fine. I did it several times on one of the VPS's that's not in production yet and all of that just worked great. I confirrmed on the host system that it really did mount and unmount.

This is a little on my back burner at the moment as I'm still trying to recover one of my clients - which is a little difficult because they were also having backup problems. But at least I now have gotten some sleep.

Thanks - I'll let you know when and if I want to proceed on this.

Submitted by cruiskeen on Tue, 09/29/2015 - 11:16

A little more info on this. Now when I try to create a system on the CentOS 7 server I'm getting the following

Creating virtual system with KVM ..
.. creation started with IP 69.4.98.207.
 
Waiting for creation to complete ...........
.. creation has completed successfully.
 
Creating swap file of 4 GB ..
.. done
 
Removing missing disks from fstab file ..
.. cleanup failed : Failed to read fstab file : None of the partions in the disk image /dev/spacevol/newtestvps_cloudmin_cruiskeenconsulting_com_img could be mounted and contained the /etc directory. This may be due to the use of RAID or LVM.
 
Mounting new instance's filesystem ..
.. failed : None of the partions in the disk image /dev/spacevol/newtestvps_cloudmin_cruiskeenconsulting_com_img could be mounted and contained the /etc directory. This may be due to the use of RAID or LVM.
 
Adding DNS entry newtestvps.cloudmin.cruiskeenconsulting.com. for IP address 69.4.98.207 ..
.. done
 
Starting up new KVM instance ..
.. failed to start : KVM command was not found!

Submitted by JamieCameron on Tue, 09/29/2015 - 22:56

It seems like Cloudmin is having trouble SSHing into the host system. Are they on the same network? And is the host system perhaps overloaded?

Submitted by cruiskeen on Tue, 09/29/2015 - 23:08

Well --- probably at the time I was trying to do that the host system WAS under a fair amount of load --- we lost a server on Sunday and we had to do a lot of re-organizing -- one or 2 of the servers are way too busy for their own good, but I'm trying to level that out right at the moment.

I think that PARTICULAR problem was an anomaly, you're probably right, it's probably because the server was not responding fast enough.

I'm going to do a little more testing in the morning.

Submitted by JamieCameron on Wed, 09/30/2015 - 00:08

It would be interesting to check Cloudmin's history of access to the host system, on the Edit System page under Status Change History, to see if it was ever detected as down or unreachable?

Submitted by cruiskeen on Thu, 10/01/2015 - 12:55

As mentioned in another ticket, all of the Cloudmin issues I've been experiencing seem to have been resolved, though I am still not completely sure why they started and stopped. Please close this