Submitted by cerebrum on Thu, 09/03/2015 - 21:44
Virtualmin fails to access AWS buckets if the name includes dots. However the following is a valid name according to AWS documentation: backup.webmin.com
Virtualmin shows the following error message when trying to access this bucket (or any other bucket with dots in the name):
HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Fri, 4 Sep 2015 02:27:36 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed
File does not exist: Can't connect to backup.webmin.com.s3.amazonaws.com:443
at S3/ListBucketResponse.pm line 26.
The bucket is listed under Virtualmin -> Backup and Restore -> Amazon S3 Buckets, but accessing it results in the error shown above (same if I use this bucket as a backup storage).
If I create a bucket name such as "1441333849" (same access and secret key), everything works perfectly fine... same system, same access/secret key, etc.
Cheers cerebrum
Status:
Closed (fixed)
Comments
Submitted by JamieCameron on Fri, 09/04/2015 - 16:18 Comment #1
Does Amazon allow creation of buckets with dots in their names via their AWS console?
The way the API is implemented, I can't see how this could work..
Submitted by cerebrum on Sat, 09/05/2015 - 01:13 Comment #2
Yes, you can create buckets with dots in their name via the AWS Console (web) as well as via AWS cli tools, etc. According to the official AWS documentation (see link below) bucket names must comply with DNS naming conventions, which explicitly allows dots:
[...] Bucket names must be at least 3 and no more than 63 characters long. Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number [...].
Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
Submitted by JamieCameron on Sun, 09/06/2015 - 01:07 Comment #3
You're right, buckets with dots in their names are supported. I just did a test backup to a bucket named foo.bar.jamie, and it actually worked fine - however, this was in the US region. Which region is your bucket in?
Submitted by cerebrum on Sun, 09/06/2015 - 06:55 Comment #4
I tried using buckets in US as well as eu-central-1 (also see issue 37935). Both are not working, but different error messages.
When you tested with the US bucket - did you install AWS's CLI? https://aws.amazon.com/cli/
Submitted by JamieCameron on Sun, 09/06/2015 - 17:32 Comment #5
I tested on a system without the
aws
command installed, as the error you are seeing appears to come from Virtualmin's built-in code. I assume your system also doesn't have theaws
command?Submitted by cerebrum on Tue, 09/08/2015 - 08:28 Comment #6
Hi Jamie,
Thanks for testing and checking this. It's much appreciated. Please find my tests below.
Bucket name: 1441455056, bucket location: us-east-1
Location checked with:
aws s3api get-bucket-location --bucket 1441455056
and:
s3cmd info s3://1441455056
AWS access credentials for "aws CLI" are configured using "aws configure". Bucket is accessible using:
aws s3 ls --recursive s3://1441455056/
Nothing is configured under: Virtualmin -> Backup and Restore -> Cloud Storage Providers (output below)
Provider name: Amazon S3
Current state: Not yet configured
Backup users: Used by 1 backups
However: Virtualmin -> Backup and Restore -> Amazon S3 Buckets ...shows the list of all buckets available.
A scheduled backup is configured as:
Backup destinations: Amazon S3 bucket
Access key ID: <provided>
Secret key ID: <provided>
Bucket and path: 1441455056/virtualmin/%Y%m%d-%H%M-full
If I trigger a backup via: Virtualmin -> Backup and Restore -> Scheduled Backups -> Action: Backup -> Backup Now, this works perfectly fine.
Then, I created two new buckets:
aws --region eu-central-1 s3 mb s3://eu.bucket.virtualmin
aws --region us-east-1 s3 mb s3://us.bucket.virtualmin
I changed the configuration: Virtualmin -> Backup and Restore -> Scheduled Backups -> edit the existing one
Bucket and path: us.bucket.virtualmin/virtualmin/jessie/%Y%m%d-%H%M-full
-> Save Schedule -> Actions: Backup -> Backup Now
No errors, all good. I update the configuration again: Virtualmin -> Backup and Restore -> Scheduled Backups -> edit the existing one
Bucket and path: eu.bucket.virtualmin/virtualmin/jessie/%Y%m%d-%H%M-full
-> Save Schedule -> Actions: Backup -> Backup Now
Error (it looks like virtualmin does not use aws CLI for the backup):
Starting backup of 1 domains to virtualmin/jessie/%Y%m%d-%H%M-full in Amazon S3 bucket eu.bucket.virtualmin ..
HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Tue, 8 Sep 2015 12:48:07 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed
File does not exist: Can't connect to eu.bucket.virtualmin.s3.amazonaws.com:443
at S3/ListBucketResponse.pm line 26.
Then, I created a third bucket in eu-central-1:
aws --region eu-central-1 s3 mb s3://eu1441717636
...and changed the config to use this bucket:
Bucket and path: eu1441717636/virtualmin/jessie/%Y%m%d-%H%M-full
...triggered the backup and this works, too. So my conclusion is that it's the dot in the bucket name (see issue 37935). Do you agree?
The server is still in "testing mode" and not in production. So, feel free to tell me what you want me to do if there is anything I can do to help you.
Submitted by JamieCameron on Tue, 09/08/2015 - 17:36 Comment #7
I wonder if you are seeing an issue with DNS caching making newly created bucket hostnames (like eu.bucket.virtualmin.s3.amazonaws.com) unavailable?
If you try another backup to that same bucket after waiting a while after creating it, does it still fail?
Submitted by cerebrum on Thu, 09/17/2015 - 19:09 Comment #8
Today I had the time to investigate this issue further. I removed all previous configuration and started from scratch again. It definitely looks like the dot in the bucket name causes an issue (at least in the European region - I have not tested other regions).
Step 1: everything is clean
Virtualmin -> Backup and Restore -> Amazon S3 Buckets: No scheduled backups to S3 have been created yet, and no default S3 keys have been set.
Virtualmin -> Backup and Restore -> Cloud Storage Providers: "Not yet configured" and "Not used by any backups" (both AWS and Rackspace)
Virtualmin -> Backup and Restore -> Scheduled Backups: No scheduled backups have been created yet.
Step 2: test aws-cli (command line)
Create a new bucket and upload a test file:
# aws --region eu-central-1 s3 mb s3://eu.virtualmin.test
make_bucket: s3://eu.virtualmin.test/
# aws --region eu-central-1 s3 cp test.txt s3://eu.virtualmin.test/
upload: ./test.txt to s3://eu.virtualmin.test/test.txt
# aws --region eu-central-1 s3 ls s3://eu.virtualmin.test/
2015-09-17 23:38:07 29 test.txt
Step 3: configure backup in virtualmin
Virtualmin -> Backup and Restore -> Scheduled Backups -> Add a new backup schedule
Destination:
Amazon S3 bucket
Access key ID:
**********
Secret key ID:
**********
Bucket and path:
eu.virtualmin.test/backup
-> Create Schedule
Step 4: trigger backup
Virtualmin -> Backup and Restore -> Scheduled Backups
Under Actions -> Backup.. -> Backup Now
Starting backup of 1 domains to backup in Amazon S3 bucket eu.virtualmin.test ..
HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Thu, 17 Sep 2015 23:47:36 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed
File does not exist: Can't connect to eu.virtualmin.test.s3.amazonaws.com:443
at S3/ListBucketResponse.pm line 26.
Step 5: use bucket name without a dot
Execute exactly the same commands as in step 2, but use
euvirtualmintest
rather thaneu.virtualmin.test
.In Virtualmin -> Backup and Restore -> Scheduled Backups -> ...
Change "Bucket and path" to:
euvirtualmintest/backup
and save.Finally, trigger backup as in step 4 again.
Backup runs through without any problems and the following command lists the backup files created:
# aws --region eu-central-1 s3 ls s3://euvirtualmintest/backup/
Submitted by JamieCameron on Thu, 09/17/2015 - 22:30 Comment #9
Ok, I found the bug that is causing this to fail - it is due to the
aws
command not being used in all cases. It will be fixed in the next Virtualmin release (4.19).Submitted by cerebrum on Fri, 09/18/2015 - 09:03 Comment #10
Awesome!! Many thanks.
Submitted by Issues on Fri, 10/02/2015 - 09:04 Comment #11
Automatically closed -- issue fixed for 2 weeks with no activity.
Submitted by Issues on Fri, 10/02/2015 - 09:04 Comment #12
Automatically closed -- issue fixed for 2 weeks with no activity.