I have 3 cloud servers at RackSpace UK which backup to RackSpace UK cloud servers over the local network.
Backups started failing on two of the three servers a few days ago, with the following log entry:
Starting backup of 84 domains to webX.myhostname.com/%Y/%m/%d in Rackspace container Backups .. Authentication at https://lon.auth.api.rackspacecloud.com/v1.0 failed : Backup failed! See the progress output above for the reason why.
I have checked the RS account details are correct, and via SSH if I run
virtualmin list-rs-containers
I get a correct list of containers back, so I believe the RS authentication details are correct.
I assume there should be something after the "failed: " line which provides the error message, but this appears to be missing. Is there a way to find out why the authentication is failing?
All servers (even the one which works) are running the following:
CentOS Linux 7.5.1804 Webmin version 1.881 Usermin version 1.741 Virtualmin version 6.03 Theme version Authentic Theme 19.16 Kernel and CPU Linux 3.10.0-693.11.6.el7.x86_64 on x86_64
Comments
Submitted by JamieCameron on Mon, 05/14/2018 - 23:20 Comment #1
It looks like the HTTP request timed out, but due to a separate bug that error message didn't appear properly.
Does this only happen for background scheduled backups?
Submitted by wheeler on Tue, 05/15/2018 - 05:25 Comment #2
This happens if I try a manual backup also (from the scheduled backup template). If it is a timeout it is a very short timeout as when I try it manually it appears in under 1 second of clicking the link.
Timestamping the backup logs would also be a nice feature :)
Submitted by JamieCameron on Wed, 05/16/2018 - 19:28 Comment #3
Sounds like I need to test this myself.
How did you configure Virtualmin (or your backup destination) to use Rackspace UK?
Submitted by wheeler on Thu, 05/17/2018 - 05:55 Comment #4
I have configured RackSpace as a 'Cloud Storage Provider' in the virtualmin Backup and Restore module.
Username and API Key are entered and correct
Rackspace server location: UK Default
Use internal Rackspace network servers: Yes (I have also tried with this set to No, and get the same error)
(The cloud servers are on the internal network)
Multipart chink size: default 200MB
Allow use of provider by (Resellers: No, Domain owners: No)
Use by backups :
[redacted]/%Y/%m/%d in Rackspace container Backups All virtual servers
The backup schedule is mostly default settings, no commands run before or after backup etc.
Starting a manual backup from scratch also gets the same error.
Submitted by wheeler on Thu, 05/17/2018 - 05:59 Comment #5
I have been doing some other reading on this, it seems that authentication can fail if an expired authentication token is given. Does virtualmin cache the authentication tokens anywhere? if so can I clear the cache?
This could explain why one of my servers still backs up OK but the other two do not, or I could be barking up the wrong tree :)
Submitted by JamieCameron on Fri, 05/18/2018 - 00:59 Comment #6
Yes, you can clear the token at Backup and Restore -> Cloud Backup Providers.
Submitted by wheeler on Fri, 05/18/2018 - 04:52 Comment #7
I have looked at this and can only see a 'Clear All Settings' button - I have tried this and still get the same error.
I've also remove and re-added the Cloud Storage Provider settings, and removed and recreated the scheduled backup. Still same error.
I have opened a ticked with RackSpace and they need to see the full logs from Virtualmin to be able to help.
Submitted by JamieCameron on Sat, 05/19/2018 - 16:48 Comment #8
If you remove and re-add the rackspace storage provider, are you prompted to re-authenticate? That should reset any existing token.
Submitted by JamieCameron on Sat, 05/19/2018 - 16:51 Comment #9
Actually, there appears to be a Virtualmin bug in the code that clears the rackspace account. Try editing
/etc/webmin/virtual-server/configand deleting thers_user=andrs_key=lines.Submitted by wheeler on Mon, 05/21/2018 - 07:10 Comment #10
I'm not sure what you mean by being prompted to re-authenticate? I am entering the rackspace username and api key - I don't believe there is anything else required to authenticate with the rackspace Cloud Files APIs.
I have cleared those two (rs_user/rs_key) lines from the config file and re-entered them via the Cloud Storage Providers screen and still get the same error.
The "virtualmin list-rs-files" command does still work when I have those values set correctly.
If I intentionally set the username to be incorrect, I get the following output:
[root@web virtual-server]# virtualmin list-rs-files --container Backups ERROR: Authentication at https://lon.auth.api.rackspacecloud.com/v1.0 failed : Invalid HTTP response : HTTP/1.1 401 Unauthorized
Is there a way to trigger a backup via the command line so I can see what the authentication error is?
Submitted by wheeler on Mon, 05/21/2018 - 07:58 Comment #11
Hello - I have now resolved this problem. I realised that both servers which were unable to backup hadn't been rebooted since the CentOS 7.5 update, whereas the server which was working had.
Rebooting the servers which weren't working fixed whatever the problem was.
Thanks for your help on this. It would be great if we could get a more detailed error if something goes wrong in the authentication stage, but my immediate problem is now solved.
Submitted by JamieCameron on Wed, 05/23/2018 - 00:26 Comment #12
The next release will add more details on why the rackspace operation failed. However, I'm not sure why a reboot would have fixed it!
Submitted by wheeler on Wed, 05/23/2018 - 05:59 Comment #13
Me either. Very odd. I also have a few servers on AWS (but backing up to Could Files on RS UK) and they had exactly the same problem. It seems something in the CentOS 7.4 -> 7.5 update broke the virtualmin backup process, but is fixed by a reboot.
Submitted by JamieCameron on Wed, 05/23/2018 - 23:12 Comment #14
It's possible that some shared library (ie. for SSL) was cached in the webmin server process, and was only reloaded when you rebooted.