Backups not excluding

About two updates ago, my backups started freaking out. I've got regular rsync backups running on large files, so I use the Virtualmin backup feature to grab databases and configs. Now, when a backup runs it doesn't exclude items in the "Files to exclude from each domain" field. The effect was a completely full hard drive and consequently a denial of service for my clients. Clients of which have a contractual service level agreements. How do I get my backups working right again?

Status: 
Closed (fixed)

Comments

What paths are you entering in the exclude list - full paths, or just relative to the domain's home directory?

Also, did the excludes stop working when you upgraded to Virtualmin 3.92?

wohlford's picture
Submitted by wohlford on Mon, 05/21/2012 - 18:31 Pro Licensee

The paths in this field are:

cgi-bin
domains
etc
fcgi-bin
logs
tmp
users
www

Correct. Updating to Virtualmin 3.92 is when I encountered the problem. I could have sworn it was in the update prior, but nonetheless it happened at the same time the New Feature Log mentioned "Encrypted backups."

Ok, that should work.

Which of those directories contained the files that you wanted to exclude?

wohlford's picture
Submitted by wohlford on Mon, 05/21/2012 - 19:45 Pro Licensee

All of them. None of them are currently working. There were before the recent update.

Please note, I'm using the virtualmin backup routines for database dumps and configs. Everything else gets rsync-ed regularly.

Are you using the default tar.gz backup format, or have you switched to something like ZIP or bzip2?

wohlford's picture
Submitted by wohlford on Mon, 05/21/2012 - 23:16 Pro Licensee

My default is 'pbzip2 -p3', but I also tried the normal bzip2 binary.

That should work fine..

Is the backup including public_html in top-level virtual servers, in sub-servers, or both?

wohlford's picture
Submitted by wohlford on Tue, 05/22/2012 - 08:02 Pro Licensee

The 'public_html' folder has been renamed 'www' on my server. As you can see above, 'www' has been excluded.

wohlford's picture
Submitted by wohlford on Tue, 05/22/2012 - 08:17 Pro Licensee

I'm not sure why or how, but I seem to have gotten my backups working again. It stopped working with the last update, but after flipping switches trying to diagnose this issue it's working like it was previously. There is definitely a bug, I just don't know what it is. Wish I could be of more help.

Feel free to close this issue as you see fit. If I can provide further details, I'm available to assist.

Ok, I'm glad it is working again now. Unfortunately I couldn't re-produce this, or see how any code changes could cause it.

By the way, a better way to make backups when you are planning to use rsync to copy home directories is as follows :

  1. On the backup schedule form, set the backup format to "One file per server (old format)"
  2. In the "Features and settings" section, select all features except for "Server's home directory and web page"

The resulting backup will include Apache configs, MySQL, DNS and so on, but nothing under each domains' home directory.

wohlford's picture
Submitted by wohlford on Thu, 05/24/2012 - 08:46 Pro Licensee

Thanks for the suggestion on excluding home and web directories. I'll definitely have to check that out.

It is possible that the 'Excluded Directories' for each virtual server got wiped? Some of my home directories have data in them that is specifically excluded above and beyond the general exclusion list. This is data like photo gallery files not in the web directory. For such instances, I've added those specific directories in [virtual server] -> Administration Options -> Excluded Directories. In my trying to figure this bug out, I noticed that nothing was in those fields. I believe once I corrected that, everything started to work like it had in the past. Plausible?

That page being empty would certainly explain the problem.

Do you happen to know roughly when the exclude lists got emptied - perhaps when you upgraded?

wohlford's picture
Submitted by wohlford on Thu, 05/24/2012 - 12:03 Pro Licensee

My backups started randomly failing after the upgrade. Then eventually my disk ran out of space as the backups consumed the hard drive. It looks like the exclude lists for each virtual server got emptied when I upgraded to the latest version.

I did a test upgrade on a system with some excluded directories set, but I wasn't able to re-produce the problem of the excludes disappearing..

wohlford's picture
Submitted by wohlford on Thu, 05/24/2012 - 14:47 Pro Licensee

If a tree falls in a forest and no one is around to hear it, does it make a sound? I guess the tech version is "If a bug fails to reproduce, does it exist?

On the test system, where you using general exclude directories or specific virtual server excludes?

Specific virtual server excludes. Which ones were you setting?

wohlford's picture
Submitted by wohlford on Thu, 05/24/2012 - 15:19 Pro Licensee

Pull up one of the virtual servers (e.g. example.org, example.com, etc). Flip down Administration Options. Then go to Excluded Directories. Add something in the field labeled "Directories under home to skip." Then try an upgrade. After the upgrade is complete, is there still data in that field?

Yes, that's exactly what I did.

wohlford's picture
Submitted by wohlford on Thu, 06/21/2012 - 17:33 Pro Licensee

Bug can't be reproduced. Closing ticket.