Deleting old backups - bad math?

Submitted by arjones85 on Mon, 03/18/2013 - 13:40

I was looking at the output of a backup job, and noticed the following:

Started at 18/Mar/2013 05:00

Deleting backups from local file /data/webserver_backup/incremental/%A-%Y-%m-%d older than 15 days ..
Deleting directory /data/webserver_backup/incremental/Saturday-2013-03-02, which is 15 days old ..
.. deleted 7.47 GB.

Deleting directory /data/webserver_backup/incremental/Friday-2013-03-01, which is 16 days old ..
.. deleted 7.47 GB.

.. deleted 2 old backups

I may not be understanding how Virtualmin counts its days, but there is 16 days from the 2nd to the 18th, not 15.

In addition, the output before the first deletion says it's deleting backups OLDER than 15 days, then deletes a backup it claims is only 15 days old, not older.

Thoughts?

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Mon, 03/18/2013 - 15:24

The deletion actually works by looking at the timestamp on the directory, comparing that to the current time, and checking if it is more than 15 days. That could explain the behavior you are seeing..

Submitted by arjones85 on Mon, 03/18/2013 - 15:31

Thanks Jamie,

Still slightly confused on if the output we are seeing here is correct or not. So, let me step through the process, please correct me where I am wrong so I can better understand and properly configure my backups:

Start date of backup job (per the start date in the logs I provided up above): 18/Mar/2013

First directory we see that is deletable: Saturday-2013-03-02
Virtualmin says it is: 15 days old
It's actually: 16 days old (18 - 2 = 16)

Next directory: Friday-2013-03-01
Virtualmin says it is: 16 days old
It's actually: 17 days old (18 - 01 = 17)

Edit: I re-re-read your reply, are you saying it's also looking at the actual last modification time as opposed to just the name of the directory, and if it's less than a whole day it rounds down?

Submitted by JamieCameron on Mon, 03/18/2013 - 19:30

That's correct - the name of the directory doesn't actually matter at all. What matters is the last modification timestamp on the directory.

Submitted by arjones85 on Mon, 03/18/2013 - 21:30

Got it, thanks. I really do appreciate your willingness to help.

One final question to make sure my backup policy is sane, since deletion is working differently than what I thought I understood.

I have found that in order to get Virtualmin to do a 7 day backup rotation, I have to actually set the delete after number of days field to "6" instead of "7". This will catch the previous week's backup for that day. However, in certain circumstances it appears it could potentially catch the day before that backup as well. For example:

From and including: Tuesday, March 12, 2013 at 2:01:51 AM <=== mtime of the day before the backup we actually want deleted
To, but not including : Monday, March 18, 2013 at 2:02:00 AM <=== Our backup today ran a bit longer than normal

The duration is 6 days, 0 hours, 0 minutes and 9 seconds = over 6 days, so this day will be deleted too.

How do I prevent this? Would the proper prevention be to set the number of days to delete at 7, and just allow that an 8th day will usually be present in our backup set as well? This extra day can be shown by:

From and including: Monday, March 11, 2013 at 2:04:15 AM
To, but not including : Monday, March 18, 2013 at 2:02:02 AM

The duration is 6 days, 23 hours, 57 minutes and 47 seconds = Not over 7 days old yet, so we now have a full 8 days of backups in our backup folder.

Submitted by JamieCameron on Mon, 03/18/2013 - 23:37

Because the mtime of the backup directory will be changed as each file is written into it, a backup started at say 2am will have an mtime of maybe 2:05am. However, purging is done based on the start time of the next backup - so if you enter 7 days, you might actually get 8 days worth of backups.

The work-around is to enter something like 7.5 days, which will ensure that you'll only have 7 days worth.

Submitted by arjones85 on Tue, 03/19/2013 - 09:39

Thanks Jamie,

Just for clarification, did you mean enter "6.5" in "Days to delete"? Entering 7.5 would make the problem worse (I would think) and lead to a greater chance of there being 8 days of backups, instead of 7.

Submitted by JamieCameron on Tue, 03/19/2013 - 12:39

Yes, I meant to enter 6.5.

Submitted by arjones85 on Sun, 03/24/2013 - 02:45

Thanks, I do appreciate your help! I have edited my backup policy.