Cloudmin Backup System

Submitted by george.asenov on Fri, 12/04/2015 - 08:05

Hello,

We decide to try the backup system option to backup whole virtual server.

We set:

"Remote SSH server:"

"SSH host:"

"SSH username:"

"SSH password:"

"Directory:"

and to use LVM snapshot if possible.

But after hitting button "Backup now" almost nothing happens we only see:

Finding systems to backup ..

.. found 1 systems

Working out backup destinations ..

.. found 1 usable destinations

And browser stops loading and no new processes on the server - nothing.

Do we need to create LVM snapshot beforehand or we are missing something else.

Status: 
Active

Comments

Submitted by JamieCameron on Fri, 12/04/2015 - 10:03

No, it should happen automatically.

Can you check if the host system for the VM is up, and that the remote destination is reachable via SSH?

Submitted by george.asenov on Mon, 12/07/2015 - 03:12

Hi Jamie,

Now when you mention it. I checked and cloudmin do not have SSH access. So I fixed it and tried again. The result was exactly the same.

Submitted by JamieCameron on Mon, 12/07/2015 - 22:31

Can you check when the backup is hung at that stage, what scp or ssh processes are running?

Submitted by george.asenov on Tue, 12/08/2015 - 02:02

Hi Jamie,

As I said there are no additional processes running for the backup. The backup process didn't start at all. Only in the interface we see:

Finding systems to backup ..

.. found 1 systems

Working out backup destinations ..

.. found 1 usable destinations

And nothing else after. Also the browser stop loading. Then nothing. No new processes, no SCP no SSH nothing, no new LVM snapshot....

Submitted by JamieCameron on Tue, 12/08/2015 - 22:37

Any chance we could login to your master system to see what is going wrong with the backup here?

Submitted by george.asenov on Wed, 12/09/2015 - 04:34

Hi Jamie,

Sorry Jamie but unfortunately this is a production server holding very important virtual machines so I'm unable to give access to it.

If you need some info (logs, settings, config files) I will make my best to provide it.

Submitted by george.asenov on Wed, 12/09/2015 - 08:17

Hi Again.

We manage to make it work. From the debug log we new that there is a backup log which says that there is a problem with the host key (we recently change the backup machine). So now it is working but need to be improved to show error when needed.

One additional question: Is it possible this backups to be restored on other machine with different hostname and with GPL version (if not. Is this possible with the payed version)

And if we will need the license version on which machine it should be if we need to restore these backups on other machine if for some reason this host machine is dead.

In other words we are searching for a way to have full backups on all (or only important) VPSes set in a way where is possible to bring them back online as fast as possible if the worst happen (dead hardware for example)

Submitted by JamieCameron on Wed, 12/09/2015 - 19:26

Interesting ... what was the error message in the backup log?

And yes, it is possible to restore on another system, even one running Cloudmin GPL.

Submitted by george.asenov on Thu, 12/10/2015 - 03:15

Here is the backup log:

======================================

msg=Failed to upload test file to destination system domain.com : Invalid host key in /root/.ssh/known_hosts at line 2 dests=ssh:username:password@domain.com/home/backups/s2/vpshost.com.gz dest=ssh:username:password@domain.com/home/backups/s2/vpshost.com.gz time=1449657453 base_dest=ssh:username:password@domain.com/home/backups/s2 size= host=vpshost.com ok=0 id=1449657453_1426671690178310 server=1426671690178310 empty= backup=

==========================================

*domains, usernames and passwords are replaced.

Submitted by JamieCameron on Thu, 12/10/2015 - 22:17

Ok, that looks like the cause - the question is, why doesn't this error appear in the backup output?

Do you have the option selected on the backup form to do multiple systems in parallel?

Submitted by george.asenov on Fri, 12/11/2015 - 06:33

Hi Jamie,

I do not see such an option in "system state" > "backup system" page.

Submitted by JamieCameron on Fri, 12/11/2015 - 23:00

Does anything appear on the backup page after the ".. found 1 usable destinations" message, such as a footer saying "return to backups" ?

It's rather unusual that the entire backup fails silently at this point, even though it does record the failure in the logs.

Submitted by george.asenov on Mon, 12/14/2015 - 01:40

No, nothing after that.

Submitted by JamieCameron on Mon, 12/14/2015 - 23:08

Hmmm .. I think I would need access to your system to debug this further, sorry.

Submitted by george.asenov on Mon, 01/18/2016 - 05:31

Hi Jamie.

Just to let you know: We have tried to restore one KVM instance on another Host and it fails because of the LVM group is different. Creating virtual system with KVM .. .. creation failed : LVM logical volume creation for swap failed : Volume group "vg_s2_kvm" not found.

By default Centos make the VG names from the hostname so if we need to restore a backed up machine from one host to another we need to have the same LVM Group on it.

Submitted by JamieCameron on Mon, 01/18/2016 - 14:49

george.asenov - are you restoring this backup on a completely different Cloudmin master system than the original?

Note that if you want to just move VMs between hosts, it can be done by registering both hosts with the same Cloudmin master, and using the built-in move feature.

Submitted by george.asenov on Wed, 01/20/2016 - 02:19

Hi Jamie,

We do not have more than one host machine at the moment. And we are trying to find the best way to restore all the VMs to another machine if the original crashes (broken hardware for example). In this situation migrating it with the standard move option will be impossible. So we need to restore the backup made from one host machine to completely new machine without the need of the old one (because in case of hardware failure it will be impossible).

Submitted by JamieCameron on Thu, 01/21/2016 - 01:19

Ok - this looks like a Cloudmin bug then, which we should be able to fix in the next release.

The work-around until then is to edit the .serv file in the same directory as the backup, and fix all references to the old VG name.

Submitted by george.asenov on Thu, 01/21/2016 - 02:15

We already test that but it doesn't work. Probably there is some other place where it is stored. As I can understand there is some information in the .disk file. But it is hard to read but i see the name of the LVM group there.

Submitted by JamieCameron on Thu, 01/21/2016 - 11:55

The .disk file is in a format that can be safely edited with a search and replace.

Submitted by JamieCameron on Sun, 01/24/2016 - 12:44

Status: Active » Fixed

Ok, this issue with the restore will be properly fixed in the next Cloudmin release.

Submitted by george.asenov on Tue, 02/16/2016 - 01:38

Status: Closed (fixed) » Active

Submitted by george.asenov on Tue, 02/16/2016 - 01:46

Hi In the mean time we have create lvm group with the same name like on the original machine and try again. But unfortunately it fails again. We have try using SSH server for source and choose new virtual machine and fill out the original FQDN machine name also Host, Username,Password,Directory. But submitting instantly give us the following error:

Restore failed : No backup for system v4.s2.trafficplanethosting.com was found at the selected source

v4.s2.trafficplanethosting.com is the original name of the machine. That way is in the backup and in in the field "new system named".

Submitted by JamieCameron on Tue, 02/16/2016 - 11:38

That "no backup" message means that no files starting with that prefix were found in the directory you are restoring from.

Submitted by george.asenov on Tue, 02/16/2016 - 15:13

Hi Jamie,

But the files are there. We tried many different ways with the same result. The only way we found working was to copy the files locally and then the restore happen without issues. There is some issue in the way that cloudmin is searching for the files. Probably it searching with different prefix like with the new hostname as a suffix.