Submitted by arjones85 on Tue, 08/27/2013 - 14:37
I think I've reported this before, but it's driving me nuts. Are there any sort of special flags that Virtualmin creates a bzip2 backup with that would make it no able to extract on Windows?
I can extract the backup fine on the server it is created on. But if I try to extract it on Windows with 7zip, Winrar, or Winzip, all 3 report corruption.
Any ideas?
Status:
Active
Comments
Submitted by andreychek on Tue, 08/27/2013 - 14:43 Comment #1
Unfortunately, the options being used by default are all standard options.
Out of curiosity, do you see similar problems when using gzip for compression rather than bzip2?
Also, if you go into System Settings -> Virtualmin Config -> Backup and Restore, what is "Extra command-line parameters for compression command" and "Bzip2 compression command" set to?
Submitted by arjones85 on Tue, 08/27/2013 - 21:12 Comment #2
Thanks for the help!
Extra command line parameters is set to "None".
I've dug a little bit more into it, and both smaller gzip and bzip2 archives seem to extract fine on Windows. Just the larger bzip2 I created doesn't. Virtualmin did not report any errors when creating the backup.
I currently have the backups being uploaded to S3, and I am then pulling the files down via both S3 Browser and an S3 CLI tool on Linux. To rule out S3 Browser doing something weird with my troubled backup file I moved the copy I pulled down on Linux to a public http folder, downloaded it to my Windows workstation, and it still reported as corrupted.
I am recreating a large backup with gzip instead of bzip2 and will give that a shot and report back.
Thanks again for the help and advice!
Submitted by andreychek on Tue, 08/27/2013 - 21:44 Comment #3
Yeah, it'd be great to hear the results of your gzip test.
How large are the backup archives you're working with that are having these problems?
Submitted by arjones85 on Tue, 08/27/2013 - 22:03 Comment #4
They are ~6 GB. The gzip is done being created, currently testing extracting on the Linux server and it's also downloading on Windows desktop now via S3 Browser. I have also started a bzip2 backup to compare - will get you results tomorrow.
Submitted by arjones85 on Wed, 08/28/2013 - 20:04 Comment #5
Tested both gzip, bzip2, and pbzip2. In all tests Virtualmin reported success with the backups. Results:
gzip - Extracts on Windows and Linux fine
bzip2 - Extracts on Windows and Linux fine
pbzip2 - Does *not* extract on Windows. Extracts on Linux fine.
I then recalled a previous issue I put in about pbzip2 making tar stall. This happened with version 1.1.1. I have 1.1.9 on this server, but went ahead and tried downgrading to 1.0. Tried the backup again but same thing, will not extract on Windows.
Really frustrating, but I am leaning towards pbzip2 doing something strange with the archives. I don't know what, but it seems pretty clear that's what is happening. I wanted to be able to utilize parallel compression because it cuts my backup time in half.
Any chance we can get support for pigz (http://zlib.net/pigz/)?
Submitted by andreychek on Wed, 08/28/2013 - 20:00 Comment #6
Out of curiosity, what is the backup time you see when using Gzip vs Pbzip2?
We've typically found them to be similar, but with pbzip2 using vastly more resources while performing the backup (that is, the advantage of gzip over pbzip2 is that websites are more responsive, but the backup time is about the same).
I'll do some research into Pigz though.
Submitted by arjones85 on Wed, 08/28/2013 - 22:52 Comment #7
Here's each one:
Gzip - 18 min
Bzip2 - 56 min
Pbzip2 - 16 min
Zip - 16 min
Not terrible for about a 6 gig backup. (Zip actually had the backup at 8 gigs). Uploading to S3 adds another ~1 hour though. I guess S3 upload is slow even though this box has a 250mbit upload speed. Sort of strange.
Given that there's not a whole lot of space savings between gzip and bzip2, I'll just move over to gzip for my backups. Would be interested in seeing how pigz fairs though in this mix :)