Moving VM from Ubuntu Host to CENTOS7 Host

Submitted by Franco Nogarin on Fri, 12/18/2015 - 14:11 Pro Licensee

Get this iodd error when I try and move a VM to a new CENTOS7 Host, the VM is Linux 3.13.0-71-generic on x86_64 (ubuntu 14.04) The CENTOS7 Host is new, and the VM is normally hosted on a Ubuntu HOST. This looks like other issues in the past that have been resolved. Here is what I did:

CHose Move VM and set it to shutdown first, not to assigne new IP and not already moved, the results:

Moving KVM system from smvmbackup.hardware to smvmcloud.hardware ..

Shutting down on original host system ..
.. done
Copying disk images to new host system ..

Copying disk /dev/lg_iscsi/nms_virtual_img .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ....
Re-fetching system status ..
.. done. New status is : Down
Refreshing status of host system smvmbackup.hardware ..
.. done. New status is : Webmin down

Refreshing status of host system smvmcloud.hardware ..
.. done. New status is : Webmin down



.. copy failed : Checksum mismatch after file transfer : beb4f9408f071de1c29d6f08193fdffe != 179a97ee31564358bdc877fcb2de52de
Status: 
Active

Comments

Submitted by andreychek on Fri, 12/18/2015 - 15:26

Howdy -- it looks like the image got corrupted in the transfer.

Just to verify, is there enough disk space for the image on your new server?

What output does this command produce:

df -h

Submitted by Franco Nogarin on Fri, 12/18/2015 - 18:01 Pro Licensee

Hi Eric, Yes thats what it looks like, but that does not make any sense in that there is no reason the transfer should be corrupted every time. so a couple more points....

1) The host uses a dedicate Logical Volume to provide Logical volumes for the VMs, so running DF - h on this host will not show you the free space on the LVG since it is not used as a drive by the OS.

So the free space on the volume group is:
> vgdisplay
  --- Volume group ---
  VG Name               vg_vmdata
  System ID            
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  9
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               530.00 GiB
  PE Size               4.00 MiB
  Total PE              135680
  Alloc PE / Size       11520 / 45.00 GiB
  Free  PE / Size       124160 / 485.00 GiB
  VG UUID               h1yqfF-LUV4-N4cX-H6KI-FSCI-U7dD-PadVQ1

2) this happens to any VM i try to move. it reports a checksum error.

Submitted by Franco Nogarin on Fri, 12/18/2015 - 18:21 Pro Licensee

and I misunderstoofd what you were asking, sorry, here is the output of df -h on the HOST, lots of room.

[root@smvmcloud ~]# df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_smvmcloud-root  122G  1.6G  120G   2% /
devtmpfs                        95G     0   95G   0% /dev
tmpfs                           95G     0   95G   0% /dev/shm
tmpfs                           95G   33M   95G   1% /run
tmpfs                           95G     0   95G   0% /sys/fs/cgroup
/dev/sda1                      946M  154M  793M  17% /boot
tmpfs                           19G     0   19G   0% /run/user/1000
tmpfs                           19G     0   19G   0% /run/user/0

Submitted by Franco Nogarin on Fri, 12/18/2015 - 18:30 Pro Licensee

Further testing shows similar failure when cloning a vm to this host:

Creating clone of KVM system server-2015.virtual named clone-test ..

    Allocating new IP addresses ..
    .. allocated 192.168.80.22

    Copying virtual disks ..
        Copying disk /dev/data/server-2015_virtual_img ..
        .. copy failed : Checksum mismatch after file transfer : 186777044d431dd196e5b94a30d86e29 != 4b079960df1c922ec41a4953899d024a

        Refreshing status of original system ..
        .. done. New status is : Down

    .. failed : Copy of critical disk failed : Checksum mismatch after file transfer : 186777044d431dd196e5b94a30d86e29 != 4b079960df1c922ec41a4953899d024a

space on the host of the origianl VM in my firt post:

root@smvmbackup:~# df -h
Filesystem                             Size  Used Avail Use% Mounted on
/dev/mapper/smvmbackup64-root           15G  2.3G   12G  17% /
udev                                    16G   12K   16G   1% /dev
tmpfs                                  3.2G  344K  3.2G   1% /run
none                                   5.0M  4.0K  5.0M   1% /run/lock
none                                    16G     0   16G   0% /run/shm
/dev/mapper/smvmbackup64-lv_root_temp   30G  178M   28G   1% /tmp
/dev/cciss/c0d0p1                      228M   27M  189M  13% /boot

space on the host of the copy job above

root@smvmmaster:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/smvmmaster--vg-root   52G  9.1G   41G  19% /
udev                              95G   12K   95G   1% /dev
tmpfs                             19G  404K   19G   1% /run
none                             5.0M     0  5.0M   0% /run/lock
none                              95G     0   95G   0% /run/shm
/dev/sda1                        237M   20M  205M   9% /boot
/dev/mapper/smvmmaster--vg-home   79G   56M   75G   1% /home

Submitted by JamieCameron on Fri, 12/18/2015 - 23:47

How close are the source and destination systems, network-wise? It's possible that corruption could be happening on the network.

Also, is the VM shut down when you are moving it?

Submitted by Franco Nogarin on Sat, 12/19/2015 - 00:30 Pro Licensee

they are sitting beside each other plugged into the same gigabit switch, the problem occurs regardless of up down state, i tried both.

Submitted by JamieCameron on Sat, 12/19/2015 - 00:53

Are both host systems configured to use the same iSCSI server in Cloudmin?

Submitted by Franco Nogarin on Sat, 12/19/2015 - 11:23 Pro Licensee

Both are set up to use LVM not iSCSI.

Submitted by JamieCameron on Sat, 12/19/2015 - 15:24

Ok, I was confused by the appearance of lg_iscsi in the LV path in your original post.

chadwick89's picture
Submitted by chadwick89 on Wed, 04/13/2016 - 11:30

Same problem here.

I am working with 3 servers in this scenario:

Server #1 - Running Cloudmin Server #2 - KVM Host (Ubuntu 12.04LTS) Server #3 - KVM Host (Centos7 Community Edition 7.15.1511)

I am trying to clone a Ubuntu 14.04LTS VM (in shutdown state) from Server#2 to Server#3. When I instantiate this process, I see that Server#1 begins to quickly lose its available (/) drive space. There are 9.5GB free when the cloning starts, but once it gets rolling I see this in the /tmp/.webmin folder:

9.5GiB [##########] 989460_15494_4_clone.cgi

About 5-7 minutes after I see this, the cloning process fails with:

reating clone of KVM system server-2015.virtual named ubutestclone1 ..

Allocating new IP addresses .. .. allocated 192.168.80.33 Copying virtual disks ..

Copying disk /dev/data/server-2015_virtual_img .. .. copy failed : Checksum mismatch after file transfer : 556516304b91c6a43a020545e24b0af2 != 132964435a685eb0b437ba2e4f0160e4 Refreshing status of original system .. .. done. New status is : Down

.. failed : Copy of critical disk failed : Checksum mismatch after file transfer : 556516304b91c6a43a020545e24b0af2 != 132964435a685eb0b437ba2e4f0160e4

Both servers #2 and #3 are using LVM for VMs, and both have plenty of space on their drives. All servers involved are connected to the same LAN segment, so packet loss (and other TCP/IP issues) should be minimal. We haven't had issues cloning in other circumstances.

Any additional assistance here would be greatly appreciated.

Submitted by JamieCameron on Wed, 04/13/2016 - 23:46

For anyone who is seeing this, it would be really useful for us to login to your Cloudmin master system (if that's possible)

Submitted by Franco Nogarin on Fri, 04/15/2016 - 08:11 Pro Licensee

Jaimie, I am on the road at the moment, and we have several other cloudmin emergencies that we are dealing with when i get back, but as soon as I can I will send over credentials and full set of symptoms - many of the priority issue I am facing require local access via VNC (vm is stalled at boot without ip address) and i need hands on to evaluate all the problems. I will likely send over credentials on monday.
Thanks!
Franco

Submitted by Franco Nogarin on Mon, 04/18/2016 - 17:13 Pro Licensee

Done, emails with info sent.... Thanks again Jamie!
Franco