Restoring of zip backups fails for big zips

Submitted by flo on Tue, 12/08/2015 - 14:08 Pro Licensee

VIrtualmin says during restore:

4294967296 extra bytes at beginning or within zipfile

But i can unzip the archive on the same machine with unzip ...without problems.

How can i restore from the unzipped files?

Status: 
Active

Comments

Submitted by andreychek on Tue, 12/08/2015 - 14:27

Howdy -- hmm, what is the output of this command:

unzip -t BACKUP_FILE_NAME.zip

That will perform a check on the file to see if there's any issues detected in the archive.

Submitted by flo on Tue, 12/08/2015 - 14:42 Pro Licensee

HI Andreycheck

everything except one mail should be ok I've removed thousands of lines ending with OK Here's the rest:

Archive: dehn.in.zip warning [dehn.in.zip]: 4294967296 extra bytes at beginning or within zipfile (attempting to process anyway) file #1: bad zipfile offset (local header sig): 4294967296 (attempting to re-compensate) testing: homes/sandeep.dwivedi/Maildir/.trash/cur/1442842859.P17287Q36M327357.new.extdehn.de:2,S bad CRC 451d335c (should be 1931ba86) file #25717: bad zipfile offset (local header sig): 1785944 (attempting to re-compensate) At least one error was detected in dehn.in.zip.

Submitted by andreychek on Tue, 12/08/2015 - 14:46

Okay, so there actually is a problem with the .zip file, but it appears to be a very minor one.

However, since there was an error returned in the unzip process, Virtualmin does stop the restore process.

In the restore screen, does it help to check the "Ignore errors" option? Does that cause the restore to ignore that issue and continue?

Submitted by flo on Tue, 12/08/2015 - 14:53 Pro Licensee

sadly not... if i get it extracted in the filesystem can i tar it and try to restore it from the tar?

Submitted by andreychek on Tue, 12/08/2015 - 14:59

Yes you could regenerate the archive, using either .tar or .zip, either one should allow Virtualmin to continue.

Even if you used a .zip, simply regenerating that .zip should correct the problem that unzip is detecting.

Submitted by flo on Tue, 12/08/2015 - 15:32 Pro Licensee

gnargh id did a zip -FF --out fixed.dehn.in.zip dehn.in.zip

zip -T fixed.dehn.in.zip homes/sandeep.dwivedi/Maildir/.trash/cur/1442842859.P17287Q36M327357.new.extdehn.de:2,S bad CRC 451d335c (should be 1931ba86) test of fixed.dehn.in.zip FAILED

so an error remains ... virtualmin still fails

(but i can unpack it with unzip and get about 7 gb of files) it seems to be a problem with zips greater than 4 GB

What can i do now?

Submitted by andreychek on Tue, 12/08/2015 - 16:15

That's actually the same file as earlier, which is odd.

Just to verify -- are you certain that's a newly created .zip file, rather than the old one?

Also, you may want to try making a .tar.gz file, just to see if that helps.

It should be no problem to have backups of any size, so that size itself shouldn't be the problem.

What output do you see if you run this command:

dmesg | tail -30

That will show if the kernel has produced any error messages recently, which could occur if there were a disk issue.

Submitted by flo on Tue, 12/08/2015 - 16:41 Pro Licensee

Hm now i came a little further i tried to restore the fixed.dehn.in.zip (without renaming it to dehn.in.zip) he started but it failed at the home directories - which is supposed to be the CRC error in the zip ... (see output_restore_...) then i tried to extract it on the command line and zip the extracted files again ... but i did not have luck with these

For the background: the original server has crashed filesystem broke down .. all i have are these zip files :-(

is there a way to extract and tar these files to be able to restore them as tar?

Submitted by andreychek on Tue, 12/08/2015 - 18:20

Since it looks like one of the issues is one particular file, what if you delete this one first:

homes/sandeep.dwivedi/Maildir/.trash/cur/1442842859.P17287Q36M327357.new.extdehn.de:2,S

And then, after that's deleted, re-create the archive without it.

Are you then able to restore the resulting backup archive?

Submitted by flo on Tue, 12/08/2015 - 19:23 Pro Licensee

Hi thank you yes I could repair and restore this one but now I'm stuck with the next:

[root@dehn4 dehn-usa.com]# zip -T dehn-usa.com.zip zip warning: unexpected signature on disk 0 at 1102260042

zip warning: archive not in correct format: dehn-us.com.zip

after a while he comes along with: copying: homes/frank.basciano/Maildir/.trash/cur/1408143321.P8366Q27.new.extdehn.de:2,S fcopy: write error

zip warning: no end of stream entry found: homes/frank.basciano/Maildir/.trash/cur/1408143321.P8366Q27.new.extdehn.de:2,S
zip warning: rewinding and scanning for later entries
zip warning: unexpected signature 50 4b 07 08 on disk 0 at 3398136652

the fixed file is little smaller but also broken :-(

Submitted by andreychek on Tue, 12/08/2015 - 20:31

You're seeing some really unusual issues there.

If you're generating a new archive on a new server, you shouldn't be seeing archive errors, unless they occurred when creating the new archive... which is a sign of other issues.

This particular system -- is it a dedicated server, or a VPS?

If it's a VPS, what kind of VPS is it?

Submitted by flo on Tue, 12/08/2015 - 23:56 Pro Licensee

It's a dedicated server running centos 7 with 32gb of ram

Submitted by andreychek on Wed, 12/09/2015 - 09:38

Would it be possible for me to log in and take a look at this backup file, and attempt a restore?

Is the backup file now located on the server it should be restored to?

The thing that's curious to me, is that even if there was a problem with the old archive, as soon as you create a new one, you should no longer see any of those problems.

The idea that you're seeing problems with new archives makes me concerned that you're experiencing a problem with something on this new server.

However, I can take a look if you like, and see what I can figure out.

Submitted by flo on Thu, 12/10/2015 - 02:46 Pro Licensee

Hi

Sorry my company guidelines don't allow external access. But I got some facts. All backups > than 4 GB in format ZIP are faulty on centos. If they are less than 8 gb in size they are sometimes recoverable. Zip does not report errors when failures occur while writing (!). Our incremental backups had the same size than the full backups which looks like virtualmin could not read the full backups --> but I did not get an error message here I think the incremental backup should report an error or at least a warning when the full backup is unreadable.

So i suppose to remove the ZIP option on centos or let backups fail when a single file or the whole zip is bigger than 4 GB & add the warning for unreadable backups (or maybe a verify function for backups)

We could reproduce these errors on centos 5 6 and 7

so long and thanks for your help & thanks for virtualmin I'm still a fan ;-) Flo

Submitted by andreychek on Thu, 12/10/2015 - 10:21

Sorry that you're seeing so many problem when making archives on CentOS!

Unfortunately, I don't seem to be able to reproduce those issues though.

In fact, we have quite a few people using CentOS, many of whom have extremely large sites, and we haven't received any reports of issues with zip file corruption.

I did some testing on that --

What I did is log into CentOS 7... the new server Joe is setting up for virtualmin.com, and I generated a backup of our home directory, which is currently 9GB.

The resulting .zip file is a little over 6GB:

ls -lh homes.zip
-rw-r--r-- 1 root root 6.3G Dec 10 10:55 homes.zip

And then, I used unzip -t to test:

unzip -t homes.zip | grep -v OK  
Archive:  homes.zip
No errors detected in compressed data of homes.zip.

Our resulting .zip file above appears to be correct.

My concern here is that something else may be going on with your setup that's causing some problems.

Also, I think we saw similar problems above when we tried making a .tar version of your backup, it wasn't able to restore that either.

I understand if you aren't able to have us to log in, but I'd definitely recommend having someone look deeper into that to try and figure out what's going on.

I'm not sure that we're going to be able to permenantly modify Virtualmin to allow it to restore corrupted archives... though it's possible we could temporarily tweak things in your case just to get what you have restored.

However, I'll talk to Jamie to see what he thinks.

Submitted by andreychek on Thu, 12/10/2015 - 12:05

Just to verify -- what is the output of this command on your server:

zip -v | grep large

It should show "LARGE_FILE_SUPPORT" and "ZIP64_SUPPORT", which are needed for larger files.

Both of those are enabled by default on CentOS though (I tested CentOS 6 and 7).

However, even if that doesn't work, re-creating the archive using .tar should work.

If you can't create a working archive on your server using .tar or .zip, it definitely seems like something is awry.

As a workaround, you may want to try creating the archive on a different system... perhaps a desktop.