Moving LVM based KVM VM's between physical hosts

Submitted by mcordas on Fri, 10/22/2010 - 17:31 Pro Licensee

Hi,

We will soon have to move a LVM based KVM VM from a physical KVM host to another. Both are managed by Cloudmin Pro.

The problem is that on the first host, we don't have sufficient disk space to make a LVM snapshot of it (it's 35 Gb large). We can have access to a network storage, though.

What would be your advice, as the available technique in Cloudmin we should use for that move ?

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Fri, 10/22/2010 - 18:01

So you shouldn't actually need any space on the source system, as Cloudmin does the transfer via a direct dd / ssh pipeline from the LV on the old system to a new LV on the new system.

I would recommend doing a test with a small test KVM instance first though, just to make sure that nothing is blocking the move from working, like network issues or KVM version compatibilities..

Submitted by mcordas on Fri, 10/22/2010 - 18:07 Pro Licensee

OK. I wasn't sure about that, as when we moved virtual domains, Cloudmin used gzip/gunzip. I was thinking about doing the DD via ssh pipeline myself. But if Cloudmin does it... :-)

Sure enough, we will test it before doing it on a live system.

The advantage of snapshots is that it's fast, and then we can DD wile the VM is up and running. But as we don't have that space, I guess we have no choice than being offline during the transfer.

Thank you for the answer !

Submitted by JamieCameron on Fri, 10/22/2010 - 18:12

Cloudmin doesn't actually support snapshotting of KVM instances when moving anyway - it can only move downed VMs.

Submitted by mcordas on Fri, 10/22/2010 - 18:15 Pro Licensee

OK. Thank you for the precision.

Submitted by mcordas on Mon, 10/25/2010 - 10:12 Pro Licensee

Just two additional questions :

1) If we move a LVM based KVM VM from a xxx.xxx.xxx.2 server (hostname xxx.company.com) to a xxx.xxx.xxx.3 (hostname yyy.company.com) server, how will the VM behave : Will it keep it's IP address ? Does the fact that it's host's IP and hostname will change affect the VM somehow ?

2) The theorical transfer rate between two servers with 10k SAS disks, linked through a good Gigabit Switch, should be around 80MB/s, or around 2 minutes for 10 GB. It that what we should expect, more or less, from Cloudmin when dd'ing a LVM based VM from a host to another, or should we expect a slower rate due to some additional operations/limitations ?

Thanks in advance for the precisions.

Submitted by JamieCameron on Mon, 10/25/2010 - 10:48

  1. A new IP will be allocated if the current IP is outoside new host's IP allocation range. For two hosts on the same LAN, this is unlikely .. but if you are moving a VM across the country, a new IP will generally be allocated (because otherwise the VM would no longer be accessible).

  2. There may be some overhead from the SSH encryption, but unless your system is heavily loaded or has a slow CPU that is unlikely to slow down the transfer. That said, I haven't personally tested this extensively..

Submitted by mcordas on Tue, 10/26/2010 - 05:43 Pro Licensee

Thanks for question 1.

Concerning Question 2 :

i did test dd between the two hosts : - via netcat : Host2 : nc -l -p 2000 >/dev/null Host1: dd if=/dev/zero bs=1M count=2000 | nc xxx.xxx.xxx.xxx 2000

Result : 57.3 MB/s

  • via SSH : dd if=/dev/zero bs=1M count=2000 | ssh xxx.xxx.xxx.xxx dd of=/dev/null

Result : 22MB/s ! It's definitely slower via SSH. Both hosts aren't heavily loaded, and have a large proportion of CPU adn RAM ressources free.

So here's another question : does Cloudmin compress data when dd'ing a VM from a host to another to speed up transfer ? Or does it a raw dd over SSH ?

Submitted by JamieCameron on Tue, 10/26/2010 - 11:17

SSH can do compression, but Cloudmin doesn't enable it.

How high is the CPU load on the source and destination systems when you transfer via SSH? Are the ssh and sshd processing using a lot of CPU time?

Submitted by JamieCameron on Tue, 10/26/2010 - 11:21

SSH can do compression, but Cloudmin doesn't enable it.

How high is the CPU load on the source and destination systems when you transfer via SSH? Are the ssh and sshd processing using a lot of CPU time?

Submitted by mcordas on Tue, 10/26/2010 - 13:54 Pro Licensee

For info, here's the output of the test :

Host 1 and 2 are both multicore with a lot of free RAM.

Host 1 (several VMs) :

top - 20:33:17 up 4 days, 6:01, 1 user, load average: 1.01, 1.19, 1.25 Tasks: 152 total, 1 running, 151 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5%us, 6.2%sy, 0.0%ni, 92.6%id, 0.1%wa, 0.0%hi, 0.6%si, 0.0%st Mem: 18476996k total, 18259304k used, 217692k free, 6655348k buffers Swap: 8152948k total, 176k used, 8152772k free, 142656k cached

Host 2 (no service at the moment):

top - 20:33:46 up 120 days, 10:57, 1 user, load average: 0.01, 0.04, 0.02 Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0% us, 0.2% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 10229280k total, 9560996k used, 668284k free, 158552k buffers Swap: 4194296k total, 208k used, 4194088k free, 7412092k cached

Test from Host 1 :

dd if=/dev/zero bs=1M count=1000 | ssh xxx.xxx.xxx.xxx dd of=/dev/null

Result :

1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 49.9468 seconds, 21.0 MB/s 2047889+222 records in 2048000+0 records out

Host 1 :

top - 20:40:05 up 4 days, 6:08, 2 users, load average: 1.64, 1.31, 1.24 Tasks: 165 total, 2 running, 163 sleeping, 0 stopped, 0 zombie Cpu(s): 9.7%us, 27.7%sy, 0.0%ni, 61.0%id, 0.3%wa, 0.0%hi, 1.2%si, 0.0%st Mem: 18476996k total, 18353556k used, 123440k free, 6680888k buffers Swap: 8152948k total, 176k used, 8152772k free, 142996k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19352 root 15 0 58520 3296 2560 R 49.9 0.0 0:21.44 ssh

Host 2 :

top - 20:40:19 up 120 days, 11:03, 1 user, load average: 2.55, 0.76, 0.28 Tasks: 118 total, 3 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 14.7% us, 22.0% sy, 18.7% ni, 25.4% id, 3.3% wa, 0.0% hi, 15.9% si Mem: 10229280k total, 9774704k used, 454576k free, 158812k buffers Swap: 4194296k total, 208k used, 4194088k free, 7619028k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25698 root 16 0 37696 3164 2108 R 54.5 0.0 0:07.96 sshd

Submitted by JamieCameron on Tue, 10/26/2010 - 15:00

So if I am reading that right, it looks like ssh was using ~50% of CPU on both hosts during the transfer .. which is why it would be slower than an un-encrypted nc connection.

The only way this could be avoided would be to drop the encryption, but I'm not sure how good an idea that would be if transferring across an un-trusted network.

Submitted by mcordas on Tue, 10/26/2010 - 15:35 Pro Licensee

Both hosts are on the same subnet. We control the switch and normaly, this trafic shouldn't be "sniffable" from outside. We could maybe open a port on the firewall (restricted to the IP of the 1st host).

Would it be possible to configure Cloudmin to do such an unencrypted dd over ethernet, or should we do it manually ?

Submitted by JamieCameron on Tue, 10/26/2010 - 18:49

I looked into this some more, and found that SSH doesn't actually have an option to turn off encryption :-(

However, it does have a configurable encryption type, and some (like blowfish) are faster than others. The next release will allow you to choose the SSH encryption flag.

Let me know if you'd like to try out a pre-release version with this feature.

Submitted by mcordas on Wed, 10/27/2010 - 01:51 Pro Licensee

Yes, that would be great, if there's no hassle when upgrading to release version afterwards.

Submitted by JamieCameron on Wed, 10/27/2010 - 15:01

Ok, I have just emailed you an updated RPM. Upgrading to the next major release will work fine ..

Submitted by Issues on Wed, 11/10/2010 - 15:20

Automatically closed -- issue fixed for 2 weeks with no activity.