Backup to AWS S3 fails

Submitted by tarnschaf on Mon, 04/20/2015 - 03:00

I have some problems backing up my virtual servers to S3.

Without awscli installed

.. upload failed! Empty response to HTTP request. Headers were :

with awscli installed (is it still recommended? I read about this in an old changelog)

No errors are reported, however if I check the contents of my bucket in Managment Console, no files have been uploaded. Additoinally, after the backup the following shows up:

Deleting backups from virtualmin/%Y%m%d-%H%M in Amazon S3 bucket chitai2backup older than 14 days .. .. failed to list S3 files :

When creating a bucket from Virtualmin - or when trying to delete backups older than 14 days, the following message is shown:

Failed to save bucket : Failed to save bucket ACL : The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

Status: 
Active

Comments

Submitted by JamieCameron on Mon, 04/20/2015 - 21:01

Is the aws command installed on your system ... and if so, in which directory?

Submitted by tarnschaf on Tue, 04/21/2015 - 00:10

/usr/bin/aws

aws --version aws-cli/1.2.9 Python/3.4.0 Linux/3.13.0-34-generic

Submitted by JamieCameron on Tue, 04/21/2015 - 00:19

Ok, that should work.

Which region is the bucket in that you're trying to backup to?

Submitted by tarnschaf on Tue, 04/21/2015 - 00:36

The last bucket I tried belongs to the rather new Frankfurt region. Before I used a bucket in Ireland - it worked except with large files. It appears that since installing aws-command the backups are fast, but nothing is inside the buckets afterwards..

Submitted by JamieCameron on Tue, 04/21/2015 - 00:41

So backups are failing even to buckets that are not in Frankfurt ?

Submitted by tarnschaf on Mon, 04/27/2015 - 03:17

I tried to backup to a bucket in another region but suddenly have other problems. One probably because I have a error in permissions:

.. upload failed! upload failed: ../../../../tmp/.webmin/23983-backup-20150427-0929/ccct.tar.gz to s3://yyy/virtualmin/backup-20150427-0929/xxx.gz A client error (AccessDenied) occurred when calling the PutObject operation: Access Denied

Altough I set both user and resource permissions to allow the IAM to PutObjects there.. policy simulator confirms the access.

At one time even a connection error - however it didn't appear again when I retried. Error - Perl execution failed

File does not exist: Can't connect to s3.amazonaws.com:443

LWP::Protocol::https::Socket: SSL connect attempt failed because of handshake problems SSL wants a read first at /usr/share/perl5/LWP/Protocol/http.pm line 41. at S3/ListBucketResponse.pm line 26.

So probably I have to sort out other things before I can provide more info on this ticket.

Submitted by JamieCameron on Mon, 04/27/2015 - 22:13

If S3 is sending back a "permission denied" error, it is probably because the ACL on the bucket is wrong. I'd suggest testing with another S3 client to see if the same problem occurs, and also double-checking bucket permissions in the S3 console.

Submitted by tarnschaf on Mon, 05/04/2015 - 00:12

I tried with a new bucket in Norther Carolina (Ireland once worked, as well), with the following result:

  • FIles a created, backup seems successful
  • When trying to delete old backups afterwards, it says

Deleting backups from virtualmin/backup-%Y%m%d-%H%M in Amazon S3 bucket XXX older than 14 days .. .. failed to list S3 files :

Submitted by JamieCameron on Mon, 05/04/2015 - 18:07

Is there no other information after "failed to list S3 files" ?

Submitted by tarnschaf on Tue, 05/05/2015 - 02:44

No, this is actually the tail of the log output, it ends with the colon.

Submitted by JamieCameron on Tue, 05/05/2015 - 23:42

Have you tried upgrading to version 4.17 of Virtualmin yet? It should be out now, and includes some improvements in the area of S3 backups.

Submitted by tarnschaf on Wed, 05/06/2015 - 02:33

Hi,

same output after updating to 4.17.

Is this the correct way to delete old backups? I saw a similar option in S3 bucket management...

Submitted by JamieCameron on Wed, 05/06/2015 - 21:37

Yes, this should work as a backup deletion method.

The issue may be that Virtualmn doesn't support backups to multi-level subdirectories. What is backup destination are you using exactly?

Submitted by tarnschaf on Thu, 05/07/2015 - 10:17

The value in "Bucket and path" is

bucketname/virtualmin/backup-%Y%m%d-%H%M

  • strftime replacement: checked
  • one file per server
  • create destination directory: checked

Submitted by JamieCameron on Thu, 05/07/2015 - 23:54

So I did some tests using a similar path with two levels of subdirectories, but deletion of old backups worked fine ..

virtualmin backup-domain --domain example.com --dest s3://mybucket/foo/%Y-%M-%d --newformat --all-features --strftime --purge 0.001

Submitted by acosonic on Fri, 01/17/2020 - 08:29

I think that there should be warning if awscli is not installed.

For example I've ended up with Perl execution failed

Actually: virtualmin s3 https::Socket: getaddrinfo: Name or service not known

And it worked after installing awscli...

(I guess :D) Uploading archive to Amazon's S3 service .. .. upload failed! upload failed: ../../../../tmp/.webmin/29795-/ourserver.tar.gz A client error (AccessDenied) occurred when calling the CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate.