Backup problems with iSCSI based VMs [#29918]

Submitted by Franco Nogarin on Tue, 09/10/2013 - 15:46 Pro Licensee

It would appear that the group backup function is failing while backing up a single VM works.

Last 2 days of automated cloudmin backups failed for iSCSI based VMs
/c/backup/virtual/2013-09-10/observium.virtual.gz on SSH server 192.168.80.233 as root 2013-09-10 02:00 observium.virtual Failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!

so i tried backing up a single VM that had failed and it worked fine,
/c/backup/virtual/2013-09-10/observium.virtual.gz on SSH server 192.168.80.233 as root 2013-09-10 08:46 observium.virtual Complete : 1.78 GB

so I tried all the ones that failed as a group and it failed again:
Backing up 6 systems to destinations from host systems ..

Finding systems to backup ..
.. found 6 systems
Working out backup destinations ..
.. found 6 usable destinations

Backing up geoserver.sparcsonline.com to /c/backup/virtual/2013-09-10/geoserver.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for geoserver.sparcsonline.com via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ .....
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backing up groundwork.virtual to /c/backup/virtual/2013-09-10/groundwork.virtual.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for groundwork.virtual via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ...........................................................
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backing up icinga.sparcsonline.com to /c/backup/virtual/2013-09-10/icinga.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for icinga.sparcsonline.com via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ......
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backing up otrs.sparcsonline.com to /c/backup/virtual/2013-09-10/otrs.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for otrs.sparcsonline.com via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ..............................................................
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backing up mysqlcluster1.sparcsonline.com to /c/backup/virtual/2013-09-10/mysqlcluster1.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for mysqlcluster1.sparcsonline.com via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ........
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backing up mysqlcluster2.sparcsonline.com to /c/backup/virtual/2013-09-10/mysqlcluster2.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for mysqlcluster2.sparcsonline.com via SSH to 192.168.80.233 .................................................................................. ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ...............................................................
.. backup failed : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
Backups of 0 systems completed, but 6 failed!
geoserver.sparcsonline.com : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
groundwork.virtual : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
icinga.sparcsonline.com : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
otrs.sparcsonline.com : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
mysqlcluster1.sparcsonline.com : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!
mysqlcluster2.sparcsonline.com : Failed to create copy of LVM disk : Disk is on iSCSI, but the underlying device or file was not found!

However just backing up a single vm worked fine before and now:

Backing up 1 systems to destinations from host systems ..

Finding systems to backup ..
.. found 1 systems
Working out backup destinations ..
.. found 1 usable destinations

close inspection of the VM shows:

the VM is hosted on the same host as the iSCSI is served from (SMVMCLOUD). as was the VM that backed up correctly when done singley. so from webmin on smvmcloud I checked hardware> iSCSI client>connections:
192.168.80.236 3260 wtf iqn.2013-08.smvmcloud geoserver.sparcsonline.com_0_img None /dev/sdg 192.168.80.236 3260 wtf iqn.2013-08.smvmcloud geoserver.sparcsonline.com_1_img None /dev/sdh

All good there, so checked the targets:
iqn.2013-08.smvmcloud:geoserver.sparcsonline.com_0_img LVM VG master, LV geoserver.sparcsonline.com_0_img 15 GB geoserver.sparcsonline.com iqn.2013-08.smvmcloud:geoserver.sparcsonline.com_1_img LVM VG master, LV geoserver.sparcsonline.com_1_img 5 GB geoserver.sparcsonline.com

also good, so i am very confused as to why its failing.

Franco

Status:

Closed (fixed)

Comments

Submitted by JamieCameron on Tue, 09/10/2013 - 18:32 Comment #1

It actually looks like it failed for a single server backup as well.

Part of the problem here is that due to a Cloudmin bug, the reason why it is failing is not being shown. To correct that ,replace the file /usr/share/webmin/server-manager/backups-lib.pl with the file attached to this bug report, run /etc/webmin/restart , re-try a backup and let us know what error message it displays.

Submitted by Franco Nogarin on Tue, 09/10/2013 - 19:05 Pro Licensee Comment #2

Wow i guessed it might be 2 bugs! Ok, I did as instructed and got:
Backing up 1 systems to destinations from host systems ..

Finding systems to backup ..
.. found 1 systems
Working out backup destinations ..
.. found 1 usable destinations

Submitted by JamieCameron on Tue, 09/10/2013 - 23:34 Comment #3

Ok, that's more useful.

Does the VM being backed up have only iSCSI disks, or does it have some in local files as well?

Submitted by Franco Nogarin on Wed, 09/11/2013 - 08:15 Pro Licensee Comment #4

all iSCSI.

Submitted by Franco Nogarin on Wed, 09/11/2013 - 08:26 Pro Licensee Comment #5

I only installed that file on the cloudmin master, was it needed on the KVM hosts as well? Franco

Submitted by JamieCameron on Wed, 09/11/2013 - 15:34 Comment #6

So it looks like the Cloudmin master was unable to contact the iSCSI server system.

Do you see the same error if you perform a backup using the Cloudmin command line tools? Try running something like the following as root on the master system :

cloudmin backup-systems --host geoserver.sparcsonline.com --dest ssh://root:pass@192.168.80.233/c/backup/virtual/2013-09-10

Submitted by Franco Nogarin on Thu, 09/12/2013 - 08:18 Pro Licensee Comment #7

Same results :(

Finding systems to backup ..
.. found 1 systems

Working out backup destinations ..
.. found 1 usable destinations

Backing up geoserver.sparcsonline.com to /c/backup/virtual/2013-09-12/geoserver.sparcsonline.com.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for geoserver.sparcsonline.com via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi : 

geoserver.sparcsonline.com: FAILED Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Submitted by JamieCameron on Thu, 09/12/2013 - 15:19 Comment #8

On the iSCSI server system, does anything get logged to the Webmin error log /var/webmin/miniserv.error when you try a backup?

Also, is the Webmin version on the iSCSI server up to date?

Submitted by Franco Nogarin on Fri, 09/13/2013 - 08:49 Pro Licensee Comment #9

Up to date, but Nothing related to the backup in the log but it does have a bunch of this:

Error: Module server-manager does not exist
Error
-----
Module server-manager does not exist
-----
Error: Module server-manager does not exist
Error
-----
Module server-manager does not exist
-----
Error: Module server-manager does not exist
Error
-----
Module server-manager does not exist
-----
Error: Module server-manager does not exist
Error
-----
Module server-manager does not exist
-----

Submitted by JamieCameron on Fri, 09/13/2013 - 13:49 Comment #10

That could be a problem - does a new entry like that get logged every time you try an iSCSI backup?

Submitted by Franco Nogarin on Fri, 09/13/2013 - 13:50 Pro Licensee Comment #11

I am unable to tell since there are tons of them (way more than my backup attempts) and they ar enot dated or timestamped in the log :(

Submitted by JamieCameron on Fri, 09/13/2013 - 14:43 Comment #12

You could try running tail -f /var/webmin/miniserv.error to see update to that file in real time, and then try a backup in another window.

Submitted by Franco Nogarin on Sat, 09/14/2013 - 18:09 Pro Licensee Comment #13

I have discovered all kinds of corruption in my cloumin system related to old .serv files in /etc/webmin/servers and have done the unthinkable and deleted them all. It took me all night to rebuild the serv files from backups, and cloudmin_kvm startup scripts, and I found many things that were contributing including a mixture of iscsi and non iscsi attributes used to define the iscsi drives, I hand edited everything and low and behold, 26/27 VMs are behaving as expected. The last one has issues becuase of iSCSI caching, and may not be restorable (But it was a very new VM with small effort invested)

So for now, rather than sending you on a further wild goosechase looking for bugs that may not exist, I will find the real issues which are related to two approaches in the serv file to defining iscsi paramaters, i am convinced this also causes the unlimited disk size error too.... I will post back soon. Franco

Submitted by JamieCameron on Sun, 09/15/2013 - 21:40 Comment #14

OK .. I'd be more than happy to login to your system myself to debug this further, if you are still seeing problems.

Submitted by Franco Nogarin on Mon, 09/16/2013 - 09:38 Pro Licensee Comment #15

I was able to back up several of those VMS yesterday for the first time, but the automated backup last night failed with the typical
Base destination directory /c/backup/virtual/2013-09-16 on SSH server 192.168.80.233 as root Backup files mysqlcluster1.virtual.gz Date and time completed 2013-09-16 02:00 Backed up system mysqlcluster1.virtual Final status Failed : Failed to create copy of LVM disk : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi : Backup ID 1379318403_137911087936061

I am now thinking that the "unlimited" disk size issue is related to an unpopulated disk size variable in the serverid.serv files. I will make a comprehensive report today and if you feel it will help I can set you up with access for a visit into the system.

Submitted by JamieCameron on Mon, 09/16/2013 - 11:14 Comment #16

Thanks - remote access would be very useful.

Submitted by Franco Nogarin on Mon, 09/16/2013 - 11:58 Pro Licensee Comment #17

Hi Jaimie, I am now 99% convinced (from observing my systems) that

iSCSI backup failures

VMs stuck on >grub rescue prompt

inability to transfer images from iscsi to iscsi

Inability to create VMs on HostA which puts disks on iscsi HOSTB

inability to move from files or LVM to iSCSI on anytyhing other than iSCSI host

Are all related to the true issues behind the seeming misleading error : fail to read Fastrpc business.
My 192.168.80.0 network is fast and reliable un hindered by firewalls and has no reason to fail any network comms.

The backups seem to backup the harddisk just fine but when it tries to backup the serv files it fails? I have manually restored the disks left on the backup drive after the backup failure so the backup seems to be done correctly but the checking or reporting of it fails so it stops.

The errors related to server manager on the iscsi hosts are leftover from when the iscsi host has cloudmin installed and was acting as cloudmin replication slave.

I also have an odd ISCSI interface on SMVMCLOUD named wtf that most of my vms use by default instead of the default, and then sometimes the iscsi connections on the iscsi server are made with ip 127.0.1.1 instead of the 192.168.80.236 IP in use by that host, this causes no end of headaches as it results in the failure of a vm and subsequent >grub rescue issue (same result different cause I think)

I can see this work by deleting a valid connection and immediately the invalid connection is waiting to take its place and all of a sudden cache on the host cant read or write to the drive but it can see the drive, of course this is all a theory but look at the attachment showing all those invalid 127.0.1.1 connections waiting in the background to pop up and do harm,

I have gone into the iscsi etc files and manually modified everything (deleted the connection references for the wtf interface and all the bad references to 127.0.0.1 and 127.0.1.1 and then rebooted the server, this did away with the confusing interface and all the local ip records sabataging the the iscsi system in cloudmin.

In some cases where the failure of the underlying iscsi causes corruption to the lvm for the VM I must manually recreate the original iscsi connections and restore from backup to solve the >grub recsue issues.

Franco

Submitted by Franco Nogarin on Mon, 09/16/2013 - 12:20 Pro Licensee Comment #18

remote access info sent to your @webmin.com account! and thanks! Franco

Submitted by JamieCameron on Mon, 09/16/2013 - 15:23 Comment #19

Thanks for the login. Is there a VM that I can do a test backup of?

Submitted by Franco Nogarin on Mon, 09/16/2013 - 20:15 Pro Licensee Comment #20

sure, icinga.virtual seems to fail every night try that, I rebuilt otrs.virtual from the ground up as a iSCSI vm and it backed up perfectly, I guess tonights backup will tell the tale for that one.

I should mention that I removed two HOST servers from the system and purged the remnats of cloudmin from SMVMcloud so the system only has 2 servers now SMVMHOST (master) and SMVMCLOUD (iSCSI) both are sent to make images on iSCSI on SMVMCLOUD.

Also, I noticed that you cannot make an image from an iSCSI based VM. I that because of my config or is that how cloudmin normally behaves?

Thanks again Jamie! Franco

Submitted by JamieCameron on Mon, 09/16/2013 - 23:53 Comment #21

Ok, I am trying a test backup now of icinga.virtual to /tmp on the master system. And it just completed with no issues.

Next I'll try a backup to the SSH server 192.168.80.233

Submitted by JamieCameron on Tue, 09/17/2013 - 00:05 Comment #22

Ok, that worked too - the command I used was :

cloudmin backup-systems --host icinga.virtual --dest ssh:root:XXX@192.168.80.233/c/backup/virtual/2013-09-16

does that work for you?

Submitted by Franco Nogarin on Tue, 09/17/2013 - 09:10 Pro Licensee Comment #23

Manual backups seem to work fine, I see you backed up icinga two ways and both worked, and then at 2am the automated backup for icinga and 5 other VMs failed with

Failed :
Failed to create copy of LVM disk :
Disk is on iSCSI, but an error occurred finding the underlying device or file :
Error loading iscsi-target :
Error reading response length from fastrpc.cgi :

the backup are

Systems hosted on smvmcloud From host system Every day at 2:00

Systems hosted on smvmhost From host system Every day at 2:00

I am going to rebuild the backup jobs based on group and see if it has an effect.

Submitted by Franco Nogarin on Tue, 09/17/2013 - 09:18 Pro Licensee Comment #24

here is last nights backup report i got in email, notice the odd "Disk is - "

System               Result                                        Size     
-------------------- --------------------------------------------- ----------
backtrack.virtual    OK                                            7.79 GB  
geoserver.virtual    Failed to create copy of LVM disk  : Disk is  -        
icinga.virtual       Failed to create copy of LVM disk  : Disk is  -        
mysqlcluster1.virtua Failed to create copy of LVM disk  : Disk is  -        
mysqlcluster2.virtua Failed to create copy of LVM disk  : Disk is  -        
observium.virtual    Failed to create copy of LVM disk  : Disk is  -        
otrs.virtual         Failed to create copy of LVM disk  : Disk is  -        

Finding systems to backup ..
.. found 7 systems

Working out backup destinations ..
.. found 7 usable destinations

Backing up backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for backtrack.virtual via SSH to 192.168.80.233 ..
.. created backup of 7.79 GB

Saving details of system backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backing up geoserver.virtual to /c/backup/virtual/2013-09-17/geoserver.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for geoserver.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Backing up icinga.virtual to /c/backup/virtual/2013-09-17/icinga.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for icinga.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Backing up mysqlcluster1.virtual to /c/backup/virtual/2013-09-17/mysqlcluster1.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for mysqlcluster1.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Backing up mysqlcluster2.virtual to /c/backup/virtual/2013-09-17/mysqlcluster2.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for mysqlcluster2.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Backing up observium.virtual to /c/backup/virtual/2013-09-17/observium.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for observium.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Backing up otrs.virtual to /c/backup/virtual/2013-09-17/otrs.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for otrs.virtual via SSH to 192.168.80.233 ..
.. backup failed : Failed to create copy of LVM disk  : Disk is on iSCSI, but an error occurred finding the underlying device or file : Error loading iscsi-target : Error reading response length from fastrpc.cgi :

Submitted by Franco Nogarin on Tue, 09/17/2013 - 13:21 Pro Licensee Comment #25

I made a backup job that backs up all iSCSI vms, at the same time, and ran it manualy and it worked great:

Backing up 5 systems to destinations from host systems ..

Finding systems to backup ..
.. found 5 systems
Working out backup destinations ..
.. found 5 usable destinations

Backing up backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.gz on SSH server 192.168.80.233 as root ..
Backing up mysqlcluster1.virtual to /c/backup/virtual/2013-09-17/mysqlcluster1.virtual.gz on SSH server 192.168.80.233 as root ..
Backing up observium.virtual to /c/backup/virtual/2013-09-17/observium.virtual.gz on SSH server 192.168.80.233 as root ..
Backing up icinga.virtual to /c/backup/virtual/2013-09-17/icinga.virtual.gz on SSH server 192.168.80.233 as root ..
Backing up mysqlcluster2.virtual to /c/backup/virtual/2013-09-17/mysqlcluster2.virtual.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for observium.virtual via SSH to 192.168.80.233 ....................Compressing disk files for mysqlcluster1.virtual via SSH to 192.168.80.233 ..Compressing disk files for icinga.virtual via SSH to 192.168.80.233 ..Compressing disk files for mysqlcluster2.virtual via SSH to 192.168.80.233 .....Compressing disk files for backtrack.virtual via SSH to 192.168.80.233 ................................................................................................................................................................................................................................................................................................................ ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... .............................................................................................................................................................................................................................................................................................................. ........................................................................................... . .. .... ................................................................................................................................................................................................................................................................
.....
.. created backup of 1.90 GB
Saving details of system icinga.virtual to /c/backup/virtual/2013-09-17/icinga.virtual.serv on SSH server 192.168.80.233 as root ..
......... done

.....Saving list of disks for icinga.virtual to /c/backup/virtual/2013-09-17/icinga.virtual.disks on SSH server 192.168.80.233 as root ..
............... done

....... ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... .................................................................................................................................................................................................................................................. ......................................................................... .. ... ..............................................................
...

.. created backup of 1.33 GB
Saving details of system observium.virtual to /c/backup/virtual/2013-09-17/observium.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

...Saving list of disks for observium.virtual to /c/backup/virtual/2013-09-17/observium.virtual.disks on SSH server 192.168.80.233 as root ..
..... done

..................................................................................................................................................................................... .. .. ............................................................................................................................................................................................................................................ .. .. ............................................................................................................................................................................................................................................ .. .. ............................................................................................................................................................................................................................................ .. .. ............................................................................................................................................................................................................................................ .. .. ............................................................................................................................................................................................................................................ .. .. ..................................................................................................................................................................................................................................
.. created backup of 3.03 GB
Saving details of system mysqlcluster2.virtual to /c/backup/virtual/2013-09-17/mysqlcluster2.virtual.serv on SSH server 192.168.80.233 as root ..
.... done

..Saving list of disks for mysqlcluster2.virtual to /c/backup/virtual/2013-09-17/mysqlcluster2.virtual.disks on SSH server 192.168.80.233 as root ..
... done

.. ... ............................................................................................................................................................. ... ............................................................................................................................................................. ... .............
.. created backup of 7.79 GB
Saving details of system backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.serv on SSH server 192.168.80.233 as root ..
... done

.Saving list of disks for backtrack.virtual to /c/backup/virtual/2013-09-17/backtrack.virtual.disks on SSH server 192.168.80.233 as root ..
... done

..................................................................... ................................................................................ ......................
.. created backup of 2.80 GB
Saving details of system mysqlcluster1.virtual to /c/backup/virtual/2013-09-17/mysqlcluster1.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for mysqlcluster1.virtual to /c/backup/virtual/2013-09-17/mysqlcluster1.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backups of 5 systems completed successfully.

I will set it to schedule for to night, and if it fails, it may be related to the automation process... Franco

Submitted by JamieCameron on Tue, 09/17/2013 - 13:27 Comment #26

I wonder if the issue is that the backups are being run in parallel, and are overloading the iSCSI server system. On the scheduled backup form, try changing "Backup order" from "All at once" to "One by one".

Submitted by Franco Nogarin on Tue, 09/17/2013 - 14:50 Pro Licensee Comment #27

they were always on one-by-one, this was the first time i tried all at once, and it was during working hours and worked like a charm.

They were always by host, one job per host, the new backups are as follows

Job at 2am by group all iscsi at once
job at 3am by group all LVM one-by-one

Submitted by JamieCameron on Tue, 09/17/2013 - 15:53 Comment #28

So it works when they are all done in parallel, but not one by one?!

Submitted by Franco Nogarin on Wed, 09/18/2013 - 08:09 Pro Licensee Comment #29

Its not consistent, last night only one failed:

System               Result                                        Size     
-------------------- --------------------------------------------- ----------
backtrack.virtual    OK                                            7.79 GB  
geoserver.virtual    Failed to create copy of LVM disk  : Disk is  -        
icinga.virtual       OK                                            1.92 GB  
mysqlcluster1.virtua OK                                            2.90 GB  
mysqlcluster2.virtua OK                                            3.04 GB  
observium.virtual    OK                                            1.33 GB  

Finding systems to backup ..
.. found 6 systems

Working out backup destinations ..
.. found 6 usable destinations

Backing up backtrack.virtual to 
/c/backup/virtual/2013-09-18/backtrack.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for backtrack.virtual via SSH to 192.168.80.233 ..
.. created backup of 7.79 GB

Saving details of system backtrack.virtual to 
/c/backup/virtual/2013-09-18/backtrack.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for backtrack.virtual to 
/c/backup/virtual/2013-09-18/backtrack.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backing up geoserver.virtual to 
/c/backup/virtual/2013-09-18/geoserver.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for geoserver.virtual via SSH to 192.168.80.233 ..
.. backup failed : 
Failed to create copy of LVM disk  : 
Disk is on iSCSI, but an error occurred finding the underlying device or file : 
Error loading iscsi-target :
 Error reading response length from fastrpc.cgi :

Backing up icinga.virtual to 
/c/backup/virtual/2013-09-18/icinga.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for icinga.virtual via SSH to 192.168.80.233 ..
.. created backup of 1.92 GB

Saving details of system icinga.virtual to 
/c/backup/virtual/2013-09-18/icinga.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for icinga.virtual to 
/c/backup/virtual/2013-09-18/icinga.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backing up mysqlcluster1.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster1.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for mysqlcluster1.virtual via SSH to 192.168.80.233 ..
.. created backup of 2.90 GB

Saving details of system mysqlcluster1.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster1.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for mysqlcluster1.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster1.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backing up mysqlcluster2.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster2.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for mysqlcluster2.virtual via SSH to 192.168.80.233 ..
.. created backup of 3.04 GB

Saving details of system mysqlcluster2.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster2.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for mysqlcluster2.virtual to 
/c/backup/virtual/2013-09-18/mysqlcluster2.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

Backing up observium.virtual to 
/c/backup/virtual/2013-09-18/observium.virtual.gz on SSH server 192.168.80.233 as root ..
    Compressing disk files for observium.virtual via SSH to 192.168.80.233 ..
.. created backup of 1.33 GB

Saving details of system observium.virtual to 
/c/backup/virtual/2013-09-18/observium.virtual.serv on SSH server 192.168.80.233 as root ..
.. done

Saving list of disks for observium.virtual to 
/c/backup/virtual/2013-09-18/observium.virtual.disks on SSH server 192.168.80.233 as root ..
.. done

I tried to back it up manually but it failed, so this one consistently cannot be backed up. its working fine otherwise.

Backing up 1 systems to destinations from host systems ..

Finding systems to backup ..
.. found 1 systems
Working out backup destinations ..
.. found 1 usable destinations

Backing up geoserver.virtual to 
/c/backup/virtual/2013-09-18/geoserver.virtual.gz on SSH server 192.168.80.233 as root ..
Compressing disk files for geoserver.virtual via SSH to 192.168.80.233 
.................................................................................. 
................................................................................
 ................................................................................
 ................................................................................ 
...................................................
.. backup failed : Failed to create copy of LVM disk : 
Disk is on iSCSI, but an error occurred finding the underlying device or file : 
Error loading iscsi-target : 
Error reading response length from fastrpc.cgi :
Backups of 0 systems completed, but 1 failed!
geoserver.virtual :   
Failed to create copy of LVM disk : 
Disk is on iSCSI, but an error occurred finding the underlying device or file : 
Error loading iscsi-target :
 Error reading response length from fastrpc.cgi :

Submitted by JamieCameron on Wed, 09/18/2013 - 17:09 Comment #30

Ok, I will try a manual backup of that VM.

Submitted by JamieCameron on Wed, 09/18/2013 - 17:59 Comment #31

Ok, I think I have a fix now! The reason I didn't detect it before was that it only happens for VMs that have multiple large disks on the iSCSI server, which trips a timeout in the remote Webmin connection.

I have applied a patch to your system, and will include the fix in the next Cloudmin release.

Submitted by Franco Nogarin on Wed, 09/18/2013 - 18:14 Pro Licensee Comment #32

Nice! :D Thanks so much Jamie!!! Hopefully this will solve the grub rescue issue!!!! (I am confident it will!) Franco

Submitted by Franco Nogarin on Thu, 09/19/2013 - 09:47 Pro Licensee Comment #33

No Backup failures overnight, Nice work and Thanks again! Franco

Submitted by Issues on Thu, 10/03/2013 - 16:11 Comment #34

Automatically closed -- issue fixed for 2 weeks with no activity.