New images from running systems

Submitted by ghomem on Wed, 07/09/2014 - 12:25

How can a new image be created out of a running system?

I've seen the new image from upload file on the UI.

Status: 
Closed (fixed)

Comments

Submitted by ghomem on Thu, 07/10/2014 - 04:33

I'm getting this error on "Create Image":

An image cannot be created from this system : Root filesystem in /etc/fstab is mounted from UUID=1c6c4a3e-0cdd-4db9-865d-d60df8dda555, not a real device path. This will cause systems created from the image to use incorrect device paths

what's wrong?

Submitted by andreychek on Thu, 07/10/2014 - 10:05

In the /etc/fstab file, filesystems can be mounted using either the device name, or using the device UUID.

The device name is remains the same even if the VPS is cloned, but the device UUID is specific to that particular installation.

What you would need to do is to modify the /etc/fstab file, and change that entry to use the actual device path (ie, something like /dev/sda1, though this is just an example) rather than the UUID.

Submitted by ghomem on Wed, 07/16/2014 - 14:13

Hi,

We are going to low level detail here.

We've moved from VMWare to Cloudmin. VMWare supported spnapshots which were a quick way to save a known good state of a VM. I understand that snapshots are only supported with LVM and we are not using LVM.

What is the way to backup a machine using Cloudmin, independently from the guest OS?

Thanks!

Submitted by JamieCameron on Wed, 07/16/2014 - 14:40

Cloudmin does support VM backups and images, but snapshots are only supported when using LVM (as Linux has no way of snapshotting a regular file).

Submitted by ghomem on Wed, 07/16/2014 - 14:43

So,

Instead of snapshots, what is the way to backup a virtual machine using Cloudmin, independently from the guest OS?

Submitted by JamieCameron on Wed, 07/16/2014 - 21:23

The backup functionality built into Cloudmin - it is accessible at Backup and Restore -> System Backups.

Submitted by ghomem on Sat, 09/13/2014 - 12:41

I backed up a VM on a machine, then copied the backup file to the default backup destination on another Cloudmin host... but I can't find an option to restore it now.

What I am looking for is an OS independent way to copy machines from one cloudmin host to the other.

This is a typical modus operandi for us: create a machine somewhere, teste it and then copy it to an independent cloudmin host where we want it to run in production.

Submitted by JamieCameron on Sat, 09/13/2014 - 12:47

You should be able to restore it at Backup and Restore -> Restore Backup. Just make sure you copy all the backup files to the new system.

Submitted by ghomem on Sat, 09/13/2014 - 14:43

All the 4 files are on the host default backup location:

vdi02.cloudmin.intranet.disks
vdi02.cloudmin.intranet.gz
vdi02.cloudmin.intranet.gz.1
vdi02.cloudmin.intranet.serv

and I don't see any restorable backup on Cloudmin.

Submitted by JamieCameron on Sat, 09/13/2014 - 17:47

If you go to the "Restore Backup" page, you should be able to enter the server and path to restore those files from.

Submitted by JamieCameron on Sat, 09/13/2014 - 17:49

Oh, wait .. I forgot that page only allows restoring of existing systems.

What you should do instead is use the restore-system API command. Just run the following as root on the new Cloudmin master :

cloudmin restore-systems --source /backup --host vdi02.cloudmin.intranet

(assuming those files are in /backup on the master)

Submitted by ghomem on Sat, 09/13/2014 - 18:47

Doesn't work!

[root@vdidemo ~]# cloudmin restore-systems --source /home/virtual/BACKUP/ --host vdi02.cloudmin.intranet

Finding systems to restore ..

.. found 0 systems

Working out backup sources ..

.. found 0 usable sources

[root@vdidemo ~]# ls /home/virtual/BACKUP/

vdi02.cloudmin.intranet.disks vdi02.cloudmin.intranet.gz vdi02.cloudmin.intranet.gz.1
vdi02.cloudmin.intranet.serv

[root@vdidemo ~]#

Please support us restoring this system, as our work is blocked on this.

Submitted by ghomem on Sat, 09/13/2014 - 18:47

Besides the command not working, if the "Restore Backup" page is only meant to restore existing systems from their own backups, then we should have an "Import Cloudmin System" under the "New System" section that would import the system to Cloudmin, from a given local or remote directory, without creating a backup entry.

Submitted by JamieCameron on Sat, 09/13/2014 - 19:05

My mistake, I forgot that you need to supply the --recreate flag to restore-systems in this situation.

Submitted by ghomem on Sat, 09/13/2014 - 19:22

Doesn't seem to make any difference:

[root@vdidemo ~]# cloudmin restore-systems --recreate --source /home/virtual/BACKUP/ --host vdi02.cloudmin.intranet Finding systems to restore .. .. found 0 systems

Working out backup sources .. .. found 0 usable sources

[root@vdidemo ~]#

Submitted by JamieCameron on Sat, 09/13/2014 - 20:44

Ok, I just realized there is another issue - when using a backup to do a migration like this to another Cloudmin master, it will fail as the original host ID is unknown to the new master.

We will add support for this in the next release, but until then you can perform the following work-around :

  1. Find the ID of the new host system, by running the following command on the Cloudmin master cloudmin list-systems --host new-host-name --id-only
  2. Edit the /home/virtual/BACKUP/vdi02.cloudmin.intranet.serv file and change the numeric ID in the kvm_on= line to the one from step 1.
  3. Re-try the restore.

Submitted by ghomem on Sun, 09/14/2014 - 06:16

The id is 0. Tried to restore with and without --recreate. Still doesn't work.

[root@vdidemo ~]# cloudmin list-systems --host vdidemo --id-only
0
[root@vdidemo ~]# vi /home/virtual/BACKUP/vdi02.cloudmin.intranet.serv
[root@vdidemo ~]# cloudmin restore-systems --recreate --source /home/virtual/BACKUP/ --host vdi02.cloudmin.intranet Finding systems to restore .. .. found 1 systems

Working out backup sources .. .. found 1 usable sources

Re-creating missing system vdi02.cloudmin.intranet .. Use of uninitialized value in split at /usr/libexec/webmin/acl/acl-lib.pl line 47. .. not possible : The original system was not created from an image, or the image is no longer available

[root@vdidemo ~]# cloudmin restore-systems --source /home/virtual/BACKUP/ --host vdi02.cloudmin.intranet Finding systems to restore .. .. found 1 systems

Working out backup sources .. .. found 1 usable sources

Re-creating missing system vdi02.cloudmin.intranet .. Use of uninitialized value in split at /usr/libexec/webmin/acl/acl-lib.pl line 47. .. not possible : The original system was not created from an image, or the image is no longer available

[root@vdidemo ~]#

Submitted by ghomem on Sun, 09/14/2014 - 06:47

I need your help to get this sytem running. Meanwhile I will open a separate ticket for a feature request.

Submitted by JamieCameron on Sun, 09/14/2014 - 11:22

Ok, to fix that issue you need to make sure the new Cloudmin master has the image that the system was originally created from. Images are stored under /var/webmin/server-manager

Submitted by ghomem on Sun, 09/14/2014 - 11:33

We can't create an image because fstab has UUIDs, plus we need to move windows systems as well. We had already concluded that creating images is a low-level procedure - we want to move the full systems without internal detail knowledge.

Submitted by JamieCameron on Sun, 09/14/2014 - 12:26

Yeah, the image requirement is admittedly pretty dumb - I will remove that in the next release.

Submitted by ghomem on Sun, 09/14/2014 - 13:02

I need a patch for this as I have a demo next week (which includes Cloudmin) and I can't move the system from the test server do the demo server. And I can't reinstall the VM as it is too detailed.

Please send me a patch that allows for the backup of this VM.

Submitted by JamieCameron on Sun, 09/14/2014 - 14:52

If you need a quick one-off solution, here's what I'd recommend :

  1. Create an image of the VM you want to move.
  2. Copy the files for that image (the ones that start with the image ID you chose) from /var/webmin/server-manager from the old Cloudmin master to the new.
  3. On the new master, create a new system with the same name and resources as the original from the image.

Submitted by ghomem on Sun, 09/14/2014 - 14:59

Jamie,

I told you on comment #21 why I can't create an image. I have to move the backups and get them running.

Can you recommend a manual way to make backups run on a different system?

Submitted by JamieCameron on Sun, 09/14/2014 - 15:05

Sorry, I missed that. The solution is to edit /etc/fstab and change the UUID=1c6c4a3e-0cdd-4db9-865d-d60df8dda555 to whatever the correct device file path is, like /dev/sda1 (as shown by the mount command).

This isn't needed on Windows systems.

Submitted by ghomem on Sun, 09/14/2014 - 15:43

Would you be able to provide a solution that doesn't force me to change the machines in say, 24h? I'd rather wait for a cleaner and less risky procedure if that is possible.

Note: I don't know how complicated is to enable the manual restore. If it is impossible I'll change the fstabs by hand.

Submitted by JamieCameron on Sun, 09/14/2014 - 15:50

It will be unlikely that I can come up with a full tested restore solution in 24h, so I'd recommend going with the image method I described in #24.

Submitted by ghomem on Sun, 09/14/2014 - 16:58

OK, I changed the fstab to use sda instead of UUIDs and the machine mounts the device ok after rebooting.

Tried to create an image and got this error:

Creating temporary directory for image files .. .. done

Shutting down system .. .. done

Checking filesystem on first disk ..

fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) fsck.ext2: Superblock invalid, trying backup blocks...

The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193

fsck.ext2: Bad magic number in super-block while trying to open /dev/loop1

.. un-fixable filesystem errors found!

I'm sorry but I have to politely insist that we should transfer the system without messing with the internals.

Submitted by JamieCameron on Sun, 09/14/2014 - 19:21

I'm afraid that creating an image is the only supported way of moving a VM between systems right now. This area hasn't gotten a lot of focus in Cloudmin in the past, as typically most sites have only a single master that manages multiple host systems - and moves between hosts are already supported.

So I'd suggest that getting imaging working is the best bet. Does the filesystem on this VM actually use ext2 format, as mentioned in the error from fsck? That seems unlikely, as ext2 is pretty old. I'd be interested to see the /etc/fstab file from the VM, and the output of the mount command when it is running.

Submitted by ghomem on Sun, 09/14/2014 - 20:03

The system is based on Ubuntu 12.04 and nothing of this kind of low level stuff was changed from the defaults. Fstab was only changed after your suggestion of replacing UUIDs by device names.

as typically most sites have only a single master that manages multiple host systems - and moves >between hosts are already supported.

On ticket 34238 I explained that we have multiple independent masters, and our (yours too) customer base will expand like that. The typical "move" operation between connected masters doesn't work because our masters are not connected.

But right now we need urgently to get this done - I never expected to have problem copying systems accross masters.

Information below.

FSTAB:

root@vdi02:~# cat /etc/fstab

/etc/fstab: static file system information.

#

Use 'blkid' to print the universally unique identifier for a device; this may be used with UUID= as a more robust way to name devices that works even if disks are added and removed. See fstab(5).

#

proc /proc proc nodev,noexec,nosuid 0 0

/ was on /dev/sda1 during installation

/dev/sda1 / ext4 errors=remount-ro 0 1

swap was on /dev/sda5 during installation

UUID=4e622dee-9755-4792-927f-f8116c0f68f4 none swap sw 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0 root@vdi02:~#

Output from mount:

/dev/sda1 on / type ext4 (rw,errors=remount-ro) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev)

Submitted by JamieCameron on Sun, 09/14/2014 - 20:49

Sorry, it seems like you are running into one Cloudmin bug after another here. The problem in this case is that it expects that the command e4fsck exist on your host system to check EXT4 format disks, but I am guessing that command doesn't actually exist - which causes it to fall back to trying e2fsck .

Fortunately, there is a solution - if you use the command-line cloudmin create-image tool to create an image, fsck errors can be ignored. It has to be run on the master system as root, like so :

cloudmin create-image --host your-vm-name --id your-image-id --desc "Image for transfer" --ignore-fsck

Submitted by ghomem on Sun, 09/14/2014 - 21:19

Doesn't work.

cloudmin create-image --host vdi02 --id vdi02img --desc "Image for transfer" --ignore-fsck

Unknown parameter --ignore-fsck

Converts an existing virtual system into an image.

cloudmin create-image --host name --desc "image description" --id image-id [--overwrite] [--version number | --inc-version] [--no-serial] [--no-compress] [--storage spec]

Submitted by ghomem on Sun, 09/14/2014 - 21:40

Can I create a new machine on the target cloudmin server, with the same virtual hardware and just import the virtual disk from the backup? Would that work? Which steps should I take?

Submitted by JamieCameron on Sun, 09/14/2014 - 22:03

My mistake, that --ignore-fsck flag hasn't been included in the released version yet.

So another option would be to create a VM of the same size and name as the original on the new host system. Then shut it down, copy across the compressed disk image from a backup, and un-compress it over the file or LVM device of the primary disk you created.

Submitted by JamieCameron on Sun, 09/14/2014 - 22:05

Alternately, you can replace the file /usr/libexec/webmin/server-manager/create-image.pl on the master system with the one attached to this bug report - that includes support for the --ignore-fsck flag.

Submitted by ghomem on Mon, 09/15/2014 - 05:29

I have this large file on the live machine

/home/virtual/VM/vdi02.cloudmin.intranet.img

and these two files on the backup

/home/virtual/BACKUP/vdi02.cloudmin.intranet.gz

/home/virtual/BACKUP/vdi02.cloudmin.intranet.gz.1

The machine only has one disk,

How do I assemble the two backup gzipped files into a single disk?

Submitted by JamieCameron on Mon, 09/15/2014 - 09:23

In a Cloudmin backup, each disk gets its own file - so vdi02.cloudmin.intranet.gz is the first disk, and vdi02.cloudmin.intranet.gz.1 is the second (both gzip compressed). The backup process simply compresses the original .img file into a .gz file.

If you only used one disk, you should be able to just uncompress the vdi02.cloudmin.intranet.gz on the new host, rename it over the .img file , and then start up the VM.

Submitted by ghomem on Mon, 09/15/2014 - 20:43

OK, the second drive was the CDROM ISO, not important.

After a lot of stress I managed to have the system running on a different master. I'm going to close this ticket and suggest that we continue on

https://www.virtualmin.com/node/34238

I never expected to spend 1.5 days on a VM tranfer. Please let's get this fixed.

Submitted by JamieCameron on Thu, 09/18/2014 - 15:26

FYI, the next CLoudmin release will properly support VM migration using backups and restores.

Submitted by ghomem on Thu, 09/18/2014 - 15:35

Thanks for the feedback.

Before releasing, can you check that deleting backups (backup logs as you call it) is working? I tried to delete one and the process hung forever - I had no time do dig further.

Submitted by JamieCameron on Thu, 09/18/2014 - 17:20

I tested this again, and deletion of old backups works fine.