Hi guys,
I have found a bug of sorts that's REALLY bad. I went out to my Rackspace Cloud Files Backups container to grab a Full backup from 2 Sundays ago and low and behold all my Full Backups for July and August are gone. The only file left in there is the file that Virtualmin mentioned it wanted to Delete due to my older than 30 days policy in last night's full backup (which failed).
22 servers backed up successfully, 0 had errors.
9 Virtualmin configuration settings backed up successfully.
Deleting backups from andromeda/full/%Y-%m-%d-%a in Rackspace container Backups older than 30 days ..
Deleting file andromeda/full/2013-06-30-Sun/kirklandrvsales.com.tar.bz2, which is 42 days old ..
.. deletion failed : Failed to delete part andromeda/incremental/2013-08-08-Thu/arlingtonflyin.org.tar.bz2 : Empty HTTP response
.. no backups to delete were found
That output strikes me as very strange since I am running a Full backup and not the incremental. The "no backups to delete were found" seems to be output from bad if logic when the deletion of the kirklandrvsales.com failed due to what looks like a pointer to a completely different file.
According to the full backup log on July 28th
22 servers backed up successfully, 0 had errors.
9 Virtualmin configuration settings backed up successfully.
Deleting backups from andromeda/full/%Y-%m-%d-%a in Rackspace container Backups older than 30 days ..
.. no backups to delete were found
So the mass purge appears to have happened during the August 11th backup, including the August 11th backup. I have no log entry for August 4th which is why July 28th and Aug 11th are the dates I am looking at.
I have opened a ticket with Rackspace to see if I can get the Api log of exactly what happened this past Sunday during the Full backups.
In the mean time, is there a log that I might have locally from Virtualmin that would show in more detail what it did during the failed Sunday backup Aug 11th?
Hope you all are having a great day,
~Jeremy
Comments
Submitted by xtremeservices on Mon, 08/12/2013 - 17:56 Comment #1
BTW - I am running Virtualmin Pro 4.01
Submitted by JamieCameron on Mon, 08/12/2013 - 18:24 Comment #2
That would be really bad if Virtualmin was deleting the wrong files!
The only logs we have are in the backup report emails, which should tell you what directories were cleared. The only way I can imagine that incorrect backups were deleted is if they had the wrong timestamps, making Virtualmin think they were older than 30 days. But that is hard to check now that they have been deleted :-(
Submitted by xtremeservices on Mon, 08/12/2013 - 18:32 Comment #3
Hi Jamie,
Can you explain the following log :
22 servers backed up successfully, 0 had errors.
9 Virtualmin configuration settings backed up successfully.
Deleting backups from andromeda/full/%Y-%m-%d-%a in Rackspace container Backups older than 30 days ..
Deleting file andromeda/full/2013-06-30-Sun/kirklandrvsales.com.tar.bz2, which is 42 days old ..
.. deletion failed : Failed to delete part andromeda/incremental/2013-08-08-Thu/arlingtonflyin.org.tar.bz2 : Empty HTTP response
.. no backups to delete were found
I am curious how the file andromeda/full/2013-06-30-Sun/kirklandrvsales.com.tar.bz2 had any connection to the file andromeda/incremental/2013-08-08-Thu/arlingtonflyin.org.tar.bz2 .
Thx,
Submitted by xtremeservices on Mon, 08/12/2013 - 18:55 Comment #4
Jamie can you confirm at which level in the following backup path Virtualmin searches for old files to delete?
Backups/andromeda/incremental/%Y-%m-%d-%a Backups/andromeda/full/%Y-%m-%d-%a
I was under the assumption that Virtualmin would only look in the parent folder of the strftime generated folder (incremental) and (full).
My full backup which runs on Sunday at 12am is set to purge files older than 30 days and my incremental which runs every other day of the week at 12am is also set to 30 days. I am at a complete loss as to why Virtualmin tried to delete andromeda/incremental/2013-08-08-Thu/arlingtonflyin.org.tar.bz2 during the full backup.
Sorry I am a bit chatty, I am just a little frustrated with the loss of my backups and hope we can get to the bottom of this.
Submitted by JamieCameron on Mon, 08/12/2013 - 19:23 Comment #5
Looking at the Virtualmin code, it seems that if the file being deleted does not have a properly formatted X-Object-Manifest header, it could cause this kind of deletion of additional files in the container :-(
Even though the rackspace docs at http://docs.rackspace.com/files/api/v1/cf-devguide/content/Large_Object_... say that this header should be in container/file format, I am guessing that for some reason it wasn't in this case. That would explain the message you saw, and why the files were deleted.
The next Virtualmin release will include a fix for this issue.
Submitted by xtremeservices on Mon, 08/12/2013 - 19:29 Comment #6
Thank you for the quick response Jaime.
When is the next release scheduled? Can you email me the patched file(s) if the release is a ways out?
Thx again!
Submitted by JamieCameron on Tue, 08/13/2013 - 00:06 Comment #7
You can get the fix now by saving the attached file as
/usr/share/webmin/virtual-server/rs-lib.pl, and then running/etc/webmin/restartSubmitted by xtremeservices on Tue, 08/13/2013 - 11:34 Comment #8
Thx again!
I have applied the patch and will see how things go.