Amazon S3 bucket names with dots

Virtualmin fails to access AWS buckets if the name includes dots. However the following is a valid name according to AWS documentation: backup.webmin.com

Virtualmin shows the following error message when trying to access this bucket (or any other bucket with dots in the name):

HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Fri, 4 Sep 2015 02:27:36 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed

File does not exist: Can't connect to backup.webmin.com.s3.amazonaws.com:443

at S3/ListBucketResponse.pm line 26.

The bucket is listed under Virtualmin -> Backup and Restore -> Amazon S3 Buckets, but accessing it results in the error shown above (same if I use this bucket as a backup storage).

If I create a bucket name such as "1441333849" (same access and secret key), everything works perfectly fine... same system, same access/secret key, etc.

Cheers cerebrum

Status: 
Closed (fixed)

Comments

Does Amazon allow creation of buckets with dots in their names via their AWS console?

The way the API is implemented, I can't see how this could work..

Yes, you can create buckets with dots in their name via the AWS Console (web) as well as via AWS cli tools, etc. According to the official AWS documentation (see link below) bucket names must comply with DNS naming conventions, which explicitly allows dots:

[...] Bucket names must be at least 3 and no more than 63 characters long. Bucket names must be a series of one or more labels. Adjacent labels are separated by a single period (.). Bucket names can contain lowercase letters, numbers, and hyphens. Each label must start and end with a lowercase letter or a number [...].

Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

You're right, buckets with dots in their names are supported. I just did a test backup to a bucket named foo.bar.jamie, and it actually worked fine - however, this was in the US region. Which region is your bucket in?

I tried using buckets in US as well as eu-central-1 (also see issue 37935). Both are not working, but different error messages.

When you tested with the US bucket - did you install AWS's CLI? https://aws.amazon.com/cli/

I tested on a system without the aws command installed, as the error you are seeing appears to come from Virtualmin's built-in code. I assume your system also doesn't have the aws command?

Hi Jamie,

Thanks for testing and checking this. It's much appreciated. Please find my tests below.

Bucket name: 1441455056, bucket location: us-east-1

Location checked with:

aws s3api get-bucket-location --bucket 1441455056

and:

s3cmd info s3://1441455056

AWS access credentials for "aws CLI" are configured using "aws configure". Bucket is accessible using:

aws s3 ls --recursive s3://1441455056/

Nothing is configured under: Virtualmin -> Backup and Restore -> Cloud Storage Providers (output below)

Provider name: Amazon S3
Current state: Not yet configured
Backup users: Used by 1 backups

However: Virtualmin -> Backup and Restore -> Amazon S3 Buckets ...shows the list of all buckets available.

A scheduled backup is configured as:

Backup destinations: Amazon S3 bucket
Access key ID: <provided>
Secret key ID: <provided>
Bucket and path: 1441455056/virtualmin/%Y%m%d-%H%M-full

If I trigger a backup via: Virtualmin -> Backup and Restore -> Scheduled Backups -> Action: Backup -> Backup Now, this works perfectly fine.

Then, I created two new buckets:

aws --region eu-central-1 s3 mb s3://eu.bucket.virtualmin
aws --region us-east-1 s3 mb s3://us.bucket.virtualmin

I changed the configuration: Virtualmin -> Backup and Restore -> Scheduled Backups -> edit the existing one

Bucket and path: us.bucket.virtualmin/virtualmin/jessie/%Y%m%d-%H%M-full

-> Save Schedule -> Actions: Backup -> Backup Now

No errors, all good. I update the configuration again: Virtualmin -> Backup and Restore -> Scheduled Backups -> edit the existing one

Bucket and path: eu.bucket.virtualmin/virtualmin/jessie/%Y%m%d-%H%M-full

-> Save Schedule -> Actions: Backup -> Backup Now

Error (it looks like virtualmin does not use aws CLI for the backup):

Starting backup of 1 domains to virtualmin/jessie/%Y%m%d-%H%M-full in Amazon S3 bucket eu.bucket.virtualmin ..
HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Tue, 8 Sep 2015 12:48:07 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed
File does not exist: Can't connect to eu.bucket.virtualmin.s3.amazonaws.com:443
at S3/ListBucketResponse.pm line 26.

Then, I created a third bucket in eu-central-1:

aws --region eu-central-1 s3 mb s3://eu1441717636

...and changed the config to use this bucket:

Bucket and path: eu1441717636/virtualmin/jessie/%Y%m%d-%H%M-full

...triggered the backup and this works, too. So my conclusion is that it's the dot in the bucket name (see issue 37935). Do you agree?

The server is still in "testing mode" and not in production. So, feel free to tell me what you want me to do if there is anything I can do to help you.

I wonder if you are seeing an issue with DNS caching making newly created bucket hostnames (like eu.bucket.virtualmin.s3.amazonaws.com) unavailable?

If you try another backup to that same bucket after waiting a while after creating it, does it still fail?

Today I had the time to investigate this issue further. I removed all previous configuration and started from scratch again. It definitely looks like the dot in the bucket name causes an issue (at least in the European region - I have not tested other regions).

Step 1: everything is clean

Virtualmin -> Backup and Restore -> Amazon S3 Buckets: No scheduled backups to S3 have been created yet, and no default S3 keys have been set.

Virtualmin -> Backup and Restore -> Cloud Storage Providers: "Not yet configured" and "Not used by any backups" (both AWS and Rackspace)

Virtualmin -> Backup and Restore -> Scheduled Backups: No scheduled backups have been created yet.

Step 2: test aws-cli (command line)

Create a new bucket and upload a test file:

# aws --region eu-central-1 s3 mb s3://eu.virtualmin.test
make_bucket: s3://eu.virtualmin.test/
# aws --region eu-central-1 s3 cp test.txt s3://eu.virtualmin.test/
upload: ./test.txt to s3://eu.virtualmin.test/test.txt
# aws --region eu-central-1 s3 ls s3://eu.virtualmin.test/
2015-09-17 23:38:07         29 test.txt

Step 3: configure backup in virtualmin

Virtualmin -> Backup and Restore -> Scheduled Backups -> Add a new backup schedule

Destination: Amazon S3 bucket

Access key ID: **********

Secret key ID: **********

Bucket and path: eu.virtualmin.test/backup

-> Create Schedule

Step 4: trigger backup

Virtualmin -> Backup and Restore -> Scheduled Backups

Under Actions -> Backup.. -> Backup Now

Starting backup of 1 domains to backup in Amazon S3 bucket eu.virtualmin.test ..
HTTP/1.0 500 Perl execution failed Server: MiniServ/1.760 Date: Thu, 17 Sep 2015 23:47:36 GMT Content-type: text/html; Charset=iso-8859-1 Connection: close
Error - Perl execution failed
File does not exist: Can't connect to eu.virtualmin.test.s3.amazonaws.com:443
at S3/ListBucketResponse.pm line 26.

Step 5: use bucket name without a dot

Execute exactly the same commands as in step 2, but use euvirtualmintest rather than eu.virtualmin.test.

In Virtualmin -> Backup and Restore -> Scheduled Backups -> ...

Change "Bucket and path" to: euvirtualmintest/backup and save.

Finally, trigger backup as in step 4 again.

Backup runs through without any problems and the following command lists the backup files created:

# aws --region eu-central-1 s3 ls s3://euvirtualmintest/backup/

Ok, I found the bug that is causing this to fail - it is due to the aws command not being used in all cases. It will be fixed in the next Virtualmin release (4.19).

Automatically closed -- issue fixed for 2 weeks with no activity.

Automatically closed -- issue fixed for 2 weeks with no activity.