Incremental Backups

It turns out to be faster and takes up less space. Did you look over that script I emailed to Joe and you a while back? I don't know exactly what he uses but its fast. That's why Chris Wik does it that way at ANU internet. I'm happy to beta test anything you want.

Compress it.

Courtesy of Chris Wik. Perhaps it'll help.

Here's my script, sold as seen, no guarantees :-) It has been working great for us for many years with only minor tweaks here and there. It requires the perl-Linux-LVM and gcc packages, the former can be found in rpmforge.

The config file contents are as follows, I put it in /etc/backup.conf but you could change that to suit:

$::dry_run   = '';           # change to '' for live run, '-n' for dry
$::stats    = '';       # change to '--stats' for stats output
$::snap_pc  = 5;
$::sleep    = 5;
$::backup_host  = '10.0.0.21';
@::exclude      = ('coop2_backups');
@::vgs      = ('Xen','FastXen');
 
Should be mostly self explanatory. Put the name of your volume group(s) in the vgs array, any LVs you want to exclude go in the exclude array. backup_host is the hostname or IP of your backup server.
 
You'll need a folder /mnt/backup which is where it'll mount the LVM snapshots.
 
Script: backup.pl
 
#!/usr/bin/perl -w -P
#include "/etc/backup.conf"
 
use Linux::LVM;
 
# Find LVMs
foreach $vg(@vgs) {
    print "Scanning VG $vg...\n\n";
    foreach $lv(`ls /dev/mapper/$vg-*`) {
        # Get LV info
        chomp($lv);
        %lv = get_lv_info($lv);
        @lv_name = split(/\//, $lv{'lv_name'});
        $lv_name = pop @lv_name;
        ###print "Name\t\t\t" . $lv_name . "\n";
 
        # Check if it's in the exclude list
        $match = 0;
        foreach $ex(@exclude) {
             if($lv_name =~ $ex) { $match = 1; }
        }
        if($match == 1) { next; }
 
        # Find out if it has an ext filesystem
        $type   = `file -s -L $lv`;
        if($type =~ /ext/) { $fs = 'ext'; } else { $fs = 'other'; }
        ##print "Filesystem\t\t" . $fs . "\n";
 
        # If it's ext, continue, otherwise go to next in list
        if($fs ne 'ext') {
            ##print "Not an ext fs\n";
            next;
        } else {
            print "Name\t\t\t" . $lv_name . "\n";
        }
 
        # How big is the LV?
        ##print "LV Size\t\t\t" . $lv{'size'} . $lv{'size_unit'} . "\n";
        $snap_size = ($lv{'size'} / 100 * $snap_pc) . $lv{'size_unit'};
        ##print "Snapshot size\t\t" . $snap_size . "\n";
 
        # Create snapshot
        $lvcreate = "/sbin/lvcreate -s -L " . $snap_size . " -n backup_snap /dev/" . $vg . "/" . $lv{'lv_name'};
        ##print "Creating LV\t\t" . $lvcreate . "\n";
        print `$lvcreate`;
 
        # Mount snapshot
        $mount = "mount /dev/$vg/backup_snap /mnt/backup";
        ##print "Mounting snapshot\t" . $mount . "\n";
        print `$mount`;
 
        # Check for rsync.exclude file
        if(-r '/mnt/backup/etc/rsync.exclude') { $rsync_exclude = '/mnt/backup/etc/rsync.exclude'; }
        else { $rsync_exclude = ''; }
 
        # Perform backup
        $rsync_args = "$dry_run $stats -aH --numeric-ids --delete --ignore-errors --exclude-from=".$rsync_exclude." --password-file=/etc/rsync.scrt";
        $rsync = "rsync $rsync_args /mnt/backup rsync\@".$backup_host."::backups/".$lv_name;
        ##print "Perform backup\t\t" . $rsync . "\n";
        print `$rsync`;
 
        # Unmount snapshot
        $umount = "umount /mnt/backup";
        ##print "Unmounting snapshot\t" . $umount . "\n";
        print `$umount`;
 
        # Remove snapshot
        $lvremove = "/sbin/lvremove -f /dev/$vg/backup_snap";
        ##print "Removing snapshot\t" . $lvremove . "\n";
        print `$lvremove`;
 
        # Remove leftover symlinks in /dev/mapper (required on CentOS 6)
        print `rm -f /dev/mapper/*-real /dev/mapper/*-cow`;
 
        # sleep before next backup
        ##print "Sleeping $sleep secs...\n";
        sleep $sleep;
    }
}

This assumes you are running an rsync server, if you want to use ssh for transport change $rsync_args accordingly. If you're using a public network for transferring the backups, use SSH!

Chris has been using it and it seems to work well. This could afford a good revert to point in time feature rather than the whole image from the nightly backup.

Jamie, has there been any thought given to how to speed this up? It's taking over 9 hours to do these backups. Any solutions you suggest?