S3 Backup fails due to invalid key/secret when listing buckets

Trying to backup an individual domain, eventually all domains, to a home S3 compatible server (minio) which has wildcard SSL and DNS. Tried the forum here: https://www.virtualmin.com/node/68068 (contains all server info and a longer description of the problem/what I've tried)

Using virtualmin backup interface I get told there was an error listing buckets, or with the command-line version it reports: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

Using aws cli works fine.

[root@lon ~]# aws s3 ls --endpoint-url https://s3.home.mortgages4all.co.uk
2019-12-15 18:41:15 bucket2
2019-12-16 07:14:03 test-bucket
Status: 
Active

Comments

Howdy -- thanks for contacting us!

You may want to go into Backup and Restore -> Cloud Service Providers, and there double-check the details of your account. It's possible that doesn't have the same info that the "aws" command has stored within it.

Thanks, sadly I've triple checked, they are the same. I've also tried removing them (using clear details and also checking if any scheduled backups) before using the same credentials for the virtualmin CLI. The real aws command has the same details. You can verify in the linked post.

Hmm, I'm unfortunately not sure what might cause that.

We're not seeing that issue with other buckets at the moment, it's possible that there's some unusual issues going on with that particular bucket or key.

Note that the error in question isn't being returned by Virtualmin itself, but from the S3 libraries it uses.

What you may want to try, is to generate a new set of keys for that particular bucket, and see if that resolves your issue.

If not, we may need to get Jamie involved to see if he has any thoughts, but re-generating those keys would be a good first step.

Oh, I meant to ask -- which Webmin and Virtualmin versions is it that you're using there?

Changed keys with no joy, excuse the user/pass terminology... user: tyeth pass: password123

Operating system Ubuntu Linux 18.04.2 Webmin version 1.932 Usermin version 1.780 Virtualmin version 6.08 Cloudmin version 9.4 Pro

So to elaborate, initially I realised I had an improper setup as the s3cmd run on the server reported the SSL cert for my default domain on the same server. I therefore realised I had dns issues (my server thought it had responsibility for the zone but cloudflare did so resolution of some subdomains resolved differently on my server), but I resolved this by removing the dns for the mortgages4all.co.uk domain from my server and leaving it only in cloudmin. I also setup wildcard dns and wildcard ssl to accomodate the non-path based s3 buckets.

Looking at tcpdump as I try to browse the buckets page in virtualmin I think it's using the wrong host still according to the following:

00:22:01.875087 IP lon.mortgages4all.co.uk.41565 > u2.amazonaws.com.domain: 7142 [1au] A? s3-1.amazonaws.com. (59)
00:22:01.875575 IP lon.mortgages4all.co.uk.45521 > u2.amazonaws.com.domain: 52747 [1au] AAAA? s3-1.amazonaws.com. (59)
00:22:01.876142 IP localhost.46382 > localhost.domain: 26630+ PTR? 10.65.154.156.in-addr.arpa. (44)
00:22:01.880983 IP lon.mortgages4all.co.uk.40032 > arin.authdns.ripe.net.domain: 33444 [1au] PTR? 10.65.154.156.in-addr.arpa. (67)
00:22:01.881942 IP arin.authdns.ripe.net.domain > lon.mortgages4all.co.uk.40032: 33444- 0/8/1 (464)
00:22:01.889529 IP u2.amazonaws.com.domain > lon.mortgages4all.co.uk.41565: 7142- 0/4/1 (184)
00:22:01.931292 IP localhost.41717 > localhost.domain: 43167+ PTR? 10.9.0.193.in-addr.arpa. (41)
00:22:01.931464 IP ns-343.awsdns-42.com.domain > lon.mortgages4all.co.uk.60980: 59073*- 1/4/9 AAAA 2600:9000:5306:be00::1 (350)
00:22:01.932766 IP lon.mortgages4all.co.uk.58259 > ns3.afrinic.net.domain: 26882 [1au] PTR? 10.9.0.193.in-addr.arpa. (64)
00:22:01.941307 IP localhost.45791 > localhost.domain: 19459+ PTR? 87.193.251.205.in-addr.arpa. (45)
00:22:01.945644 IP lon.mortgages4all.co.uk.50774 > c.in-addr-servers.arpa.domain: 36760 [1au] PTR? 87.193.251.205.in-addr.arpa. (68)
00:22:01.968438 IP ns1.arin.net.domain > lon.mortgages4all.co.uk.49971: 718*- 2/0/1 A 199.71.0.63, RRSIG (251)
00:22:01.970642 IP ns1.arin.net.domain > lon.mortgages4all.co.uk.33598: 8726*- 2/0/1 AAAA 2001:500:31::63, RRSIG (263)
00:22:03.510691 IP localhost.40648 > localhost.domain: 50256+ PTR? 10.169.216.196.in-addr.arpa. (45)
00:22:03.511918 IP lon.mortgages4all.co.uk.52638 > ns3.afrinic.net.domain: 4636 [1au] PTR? 10.169.216.196.in-addr.arpa. (68)
00:22:04.481657 IP localhost.60318 > localhost.domain: 46506+ A? s3.amazonaws.com. (34)
00:22:04.482255 IP localhost.domain > localhost.60318: 46506 2/4/0 CNAME s3-1.amazonaws.com., A 52.216.179.133 (206)
00:22:04.492986 IP localhost.60318 > localhost.domain: 23993+ AAAA? s3.amazonaws.com. (34)
00:22:04.493410 IP localhost.domain > localhost.60318: 23993 1/1/0 CNAME s3-1.amazonaws.com. (137)
00:22:08.286839 IP localhost.53047 > localhost.domain: 61201+ PTR? 210.218.234.185.in-addr.arpa. (46)
00:22:08.288209 IP lon.mortgages4all.co.uk.57059 > ns3.afrinic.net.domain: 54158 [1au] PTR? 210.218.234.185.in-addr.arpa. (69)
00:22:08.288848 IP ns3.afrinic.net.domain > lon.mortgages4all.co.uk.57059: 54158 NXDomain*- 0/6/1 (746)
00:22:08.293387 IP localhost.domain > localhost.53047: 61201 NXDomain 0/1/0 (106)

second cleaner version:

00:31:14.780652 IP localhost.47453 > localhost.domain: 54114+ A? s3.amazonaws.com. (34)
00:31:14.780702 IP localhost.47453 > localhost.domain: 16242+ AAAA? s3.amazonaws.com. (34)
00:31:14.782138 IP localhost.domain > localhost.47453: 16242 1/1/0 CNAME s3-1.amazonaws.com. (137)
00:31:14.785888 IP lon.mortgages4all.co.uk.35839 > ns-482.awsdns-60.com.domain: 61784 [1au] A? s3-1.amazonaws.com. (59)
00:31:14.786451 IP localhost.56150 > localhost.domain: 22314+ PTR? 226.193.251.205.in-addr.arpa. (46)
00:31:14.787637 IP lon.mortgages4all.co.uk.55776 > ns4.p31.dynect.net.domain: 44007 [1au] PTR? 226.193.251.205.in-addr.arpa. (69)
00:31:14.788729 IP ns4.p31.dynect.net.domain > lon.mortgages4all.co.uk.55776: 44007*- 1/4/1 PTR ns-482.awsdns-60.com. (228)
00:31:14.790255 IP localhost.domain > localhost.56150: 22314 1/5/0 PTR ns-482.awsdns-60.com. (203)
00:31:14.793662 IP ns-482.awsdns-60.com.domain > lon.mortgages4all.co.uk.35839: 61784*- 1/4/1 A 52.216.232.125 (200)
00:31:14.794573 IP localhost.domain > localhost.47453: 54114 2/4/0 CNAME s3-1.amazonaws.com., A 52.216.232.125 (206)
00:31:15.052736 IP localhost.60039 > localhost.domain: 19731+ PTR? 146.144.38.46.in-addr.arpa. (44)
00:31:15.054397 IP localhost.domain > localhost.60039: 19731 NXDomain 0/1/0 (104)
00:31:16.743543 IP localhost.43090 > localhost.domain: 16233+ A? s3.amazonaws.com. (34)
00:31:16.744383 IP localhost.domain > localhost.43090: 16233 2/4/0 CNAME s3-1.amazonaws.com., A 52.216.232.125 (206)
00:31:16.744477 IP localhost.43090 > localhost.domain: 45948+ AAAA? s3.amazonaws.com. (34)
00:31:16.744746 IP localhost.domain > localhost.43090: 45948 1/1/0 CNAME s3-1.amazonaws.com. (137)

aws is installed via apt but runs on python3. aws-cli/1.16.303 Python/3.6.9 Linux/4.15.0-72-generic botocore/1.13.39 I also have the same version installed for python2.

I also tried deleting ~/.aws/config and credentials