AWS S3 backup error ; HTTP/1.1 503 Slow Down

6 posts / 0 new
Last post
#1 Thu, 09/14/2017 - 18:40
ckwsp101

AWS S3 backup error ; HTTP/1.1 503 Slow Down

hi,

I filed a feature request for improved S3 backup.

Intermittently my scheduled S3 backups fail with the error

Uploading archive to Amazon's S3 service .. .. upload failed! Invalid HTTP response : HTTP/1.1 503 Slow Down

All virtual servers in the schedule are failed with the 503 Slow Down message.

Other times the problem backup schedule completes.

Thu, 09/14/2017 - 19:34
Joe
Joe's picture

S3 was, maybe still is, experiencing rolling outages/delays. I don't know any details, but it's out of our control.

--

Check out the forum guidelines!

Thu, 09/14/2017 - 19:35
Joe
Joe's picture

Or, do you mean this is a persistent issue not just in the past day or two?

--

Check out the forum guidelines!

Thu, 09/14/2017 - 19:47
ckwsp101

Joe,

Running on AWS LightSail VM. In last 48 hours 1 in 20 S3 backups failed. Prior to that 1 in 400 backups failed with the 503 event.

The hourly scheduled backup typically under 50 MB.

If the virtualmin backup code already has retry logic suggested by AWS document, the issue is with Amazon. I will open an Amazon ticket if the situation continues.

Thu, 09/14/2017 - 19:51
Joe
Joe's picture

I don't know for sure. Jamie wrote that code...since it's not just happening today, it may be something we need to look at. I assumed we were gracefully handling problems and I don't think we've had other similar reports lately, but it's worth looking into.

--

Check out the forum guidelines!

Thu, 09/14/2017 - 20:01
ckwsp101

Joe,

My LightSail VM instance talking to AWS region which had(?) S3 storage issue. Maybe worth reviewing code to take into account AWS notion of degraded service latency. Document when error thrown to check S3 health

12:31 PM PDT We are investigating increased error rates for Storage Gateway read/write operations in the US-EAST-1 Region.
12:53 PM PDT We are seeing recovery for Storage Gateway read/write operations in the US-EAST-1 Region.
1:10 PM PDT Between 11:40 AM and 12:56 PM PDT we experienced increased latencies for read/write operations in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.
Topic locked