Hit and Miss Rackspace Backups

Submitted by xtremeservices on Fri, 08/23/2013 - 02:44

Hi Guys,

I hate to bother you again but my backups are failing randomly at Rackspace.

The message that I see every so often is

    Uploading archive to Rackspace's Cloud Files service ..
    .. upload failed! Empty HTTP response

This does not happen every night or even every backups but it tends to happen 2 to 3 times per month and is random throughout the individual virtual servers being backed up.

Here is an example of the latest failure log :

Backup failed! See the progress output above for the reason why. Total backup time was 14 minutes, 14 seconds.

Virtual servers that failed :
    savetheolympictheatre.org
    mukilteochamber.com
    arlingtonflyin.org
    hopecounselingllc.com
    kirklandrvsales.com
    webdev.fannit.com
    xtremeservices.net
    northpacificpropertymanagement.com
    vital-soft.com
    wilzdezign.com
    mukilteo.org
    wiki.vital-soft.com
    joesglassco.com
    brockcomputers.biz
    status.xtremeservices.net
    dev.vital-soft.com
    soggyart.com
    arlingtonmtw.com
    mukilteochamber.org
    clinics.vital-soft.com
    fannit.com
    vital-soft.co
Sent by Virtualmin at: https://andromeda.xtsuniverse.net:10000

Creating backup for virtual server savetheolympictheatre.org ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database savetheolympictheatre_org ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 10 seconds

Creating backup for virtual server mukilteochamber.com ..
    Copying virtual server configuration ..
    .. done

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache aliases ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 3 seconds

Creating backup for virtual server arlingtonflyin.org ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database arlingtonflyin_org ..
    .. done

    Dumping MySQL database arlingtonflyin_org_wordpress ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 12 seconds

Creating backup for virtual server hopecounselingllc.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database hopecounselingllc_com ..
    .. done

    Dumping MySQL database hopecounselingllc_com_wordpress ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 8 seconds

Creating backup for virtual server kirklandrvsales.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database kirklandrvsales_com ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. upload failed! Empty HTTP response

.. completed in 1 minutes, 44 seconds

Creating backup for virtual server webdev.fannit.com ..
    Copying virtual server configuration ..
    .. done

    Copying records in DNS domain ..
    .. done

    Copying Apache aliases ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 3 seconds

Creating backup for virtual server xtremeservices.net ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database xtremeservices_net ..
    .. done

    Dumping MySQL database xtremeservices_net_wordpress ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 11 seconds

Creating backup for virtual server northpacificpropertymanagement.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database northpacificpropertymanagement_com ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. upload failed! Empty HTTP response

.. completed in 1 minutes, 10 seconds

Creating backup for virtual server vital-soft.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. done

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database vital_soft_com ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating backup of Mailman mailing lists ..
    .. done

    Backing up Subversion repositories and configuration files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 1 minutes, 16 seconds

Creating backup for virtual server wilzdezign.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database wilzdezign_com ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. upload failed! Empty HTTP response

.. completed in 1 minutes, 34 seconds

Creating backup for virtual server mukilteo.org ..
    Copying virtual server configuration ..
    .. done

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache aliases ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 3 seconds

Creating backup for virtual server wiki.vital-soft.com ..
    Copying virtual server configuration ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database wiki_vital_soft_com ..
    .. done

    Dumping MySQL database wiki_vital_soft_com_mediawiki ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 12 seconds

Creating backup for virtual server joesglassco.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database joesglassco_com ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 4 seconds

Creating backup for virtual server brockcomputers.biz ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. done

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database brockcomputers_biz ..
    .. done

    Dumping MySQL database brockcomputers_biz_roundcube ..
    .. done

    Dumping MySQL database brockcomputers_biz_wordpress ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 12 seconds

Creating backup for virtual server status.xtremeservices.net ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database status_xtremeservices_net ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 4 seconds

Creating backup for virtual server dev.vital-soft.com ..
    Copying virtual server configuration ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database dev_vital_soft_com ..
    .. done

    Dumping MySQL database dev_vital_soft_com_wordpress ..
    .. done

    Dumping MySQL database dev_vitalsoft_com_wp_new ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating backup of Mailman mailing lists ..
    .. no lists found

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 10 seconds

Creating backup for virtual server soggyart.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database soggyart_com ..
    .. done

    Dumping MySQL database soggyart_com_wordpress ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. upload failed! Empty HTTP response

.. completed in 1 minutes, 5 seconds

Creating backup for virtual server arlingtonmtw.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database arlingtonmtw_com ..
    .. done

    Dumping MySQL database arlingtonmtw ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 11 seconds

Creating backup for virtual server mukilteochamber.org ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database mukilteochamber_org ..
    .. done

    Copying Procmail and SpamAssassin configuration files ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 35 seconds

Creating backup for virtual server clinics.vital-soft.com ..
    Copying virtual server configuration ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Dumping MySQL database clinics_vital_soft_com ..
    .. done

    Dumping MySQL database clinics_vital_soft_com_osticket ..
    .. done

    Dumping MySQL database clinics_vital_soft_com_wordpress ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 9 seconds

Creating backup for virtual server fannit.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database fannit_com ..
    .. done

    Dumping MySQL database fannit_ver02 ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWstats configuration file ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 21 seconds

Creating backup for virtual server vital-soft.co ..
    Copying virtual server configuration ..
    .. done

    Copying records in DNS domain ..
    .. done

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache aliases ..
    .. done

    Creating incremental TAR file of home directory ..
    .. done

    Uploading archive to Rackspace's Cloud Files service ..
    .. done

.. completed in 6 seconds

Deleting backups from andromeda/incremental/%Y-%m-%d-%a in Rackspace container Backups older than 30 days ..
.. no backups to delete were found

My Failures of Full and Incremental backups so far this year : 4/30,5/9, 5/10,5/11,6/12,6/23,6/25,7/6,7/24,8/3,8/23

Please let me know if you need any more information.

Thx,

~Jeremy

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Fri, 08/23/2013 - 11:12

Does rackspace offer any logs to perhaps report why these upload operations are failing?

Also, roughly how large are these backups? Rackspace requires that files above 200M (by default) be split into multiple chunks, which uses a different upload code path in Virtualmin.

Submitted by xtremeservices on Fri, 08/23/2013 - 15:24

Hi Jamie,

Rackspace really need to get their act together with logging the API. They do not currently offer any sort of logging for this as far as I have been able to access through their higher up Tech Support.

This issue is hit and miss with regard to size. Last night's failure was for an incremental backup with a total size for all virtual servers of 179.51MB.

~Jeremy

Submitted by JamieCameron on Fri, 08/23/2013 - 19:13

I think the correct fix here is for Virtualmin to re-try rackspace uploads in cases of transient failures like this. I will implement that in the next Virtualmin release.

Submitted by Issues on Fri, 09/06/2013 - 19:21

Automatically closed -- issue fixed for 2 weeks with no activity.