Let Encrypt

Let's Encrypt needs a renewal every 90 days, so Virtualmin starts trying at 60, great!

But, if something goes wrong, like this: ValueError: Error requesting challenges: 429 { "type": "urn:acme:error:rateLimited", "detail": "Error creating new authz :: Too many currently pending authorizations.", "status": 429

Virtualmin continues to try. Over and over and over, every 5 minutes. This basically means that it will never be able to get the authorization, ever, because there will always be "too many" in the queue (20 per week per domain are all that are allowed)

And it snowballs of course, until no domain on the server can get a cert. We look like we are trying to DOS the cert server.

So, we had the error seen on ticket https://www.virtualmin.com/node/40975 which stopped the creation of a certificate, but the thing kept trying until it created the above error on a completely different domain (on the same server)

Basically I was shooting myself in the foot with all the retrys.

Help?

Status: 
Closed (fixed)

Comments

We'll fix this in the next Virtualmin release by adding a random delay of up to 24 hours before re-trying the renewal.

In your case, you could turn off auto-renewal by editing the Virtualmin config files manually. You'd need to edit each file in /etc/webmin/virtual-server/domains with a letsencrypt_renew line, and either remove it or increase the number (which is in months).

Just for fun, figured this out (the time for it to fail)

The first day, for almost exactly 24 hours, I got the "can't download..." error every 5 minutes. Then it changed to the "too many current..." and stayed there for now 13 days. Thats a lot of errors in the queue, might be a while before I can create another cert :-)

I'll try it again in a week.

Is there any kind of fix for this? I assume it affects all Virtualhosts? i have a lot of certs expiring around the same time so this is really bad for us. The first site cert is already broken.

That fix is in the new Virtualmin release that went out a few days ago.

Give that a try, and let us know if that does the trick for you.

I'll do a snapshot tonight, upgrade, and give it whirl. Thanks!

Did the upgrade to the latest Webmin, however, I'm still getting the following error:

Requesting a certificate for hosting.cloudapps.northwestern.edu from Let's Encrypt .. .. request failed : Failed to request certificate :

Parsing account key... Parsing CSR... Registering account... Already registered! Verifying hosting.cloudapps.northwestern.edu... Traceback (most recent call last): File "/usr/share/webmin/webmin/acme_tiny.py", line 202, in main(sys.argv[1:]) File "/usr/share/webmin/webmin/acme_tiny.py", line 198, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca) File "/usr/share/webmin/webmin/acme_tiny.py", line 109, in get_crt raise ValueError("Error requesting challenges: {0} {1}".format(code, result)) ValueError: Error requesting challenges: 429 { "type": "urn:acme:error:rateLimited", "detail": "Error creating new authz :: Too many currently pending authorizations.", "status": 429 }

Is there anything else I can try?

It may be necessary to wait 24 hours prior to being able to do that.

Let's Encrypt rate limits the connections to N per 24 hours I believe... so the key would be to let the existing ones expire before trying another.

https://letsencrypt.org/docs/rate-limits/

"Clearing Pending Authorizations

If you have a large number of pending authorization objects and are getting a rate limiting error, you can trigger a validation attempt for those authorization objects by submitting a JWS-signed POST to one of its challenges, as described in the ACME spec. The pending authorization objects are represented by URLs of the form https://acme-v01.api.letsencrypt.org/acme/authz/XYZ, and should show up in your client logs. Note that it doesn’t matter whether validation succeeds or fails. Either will take the authorization out of ‘pending’ state. If you do not have logs containing the relevant authorization URLs, you need to wait for the rate limit to expire. As described above, there is a sliding window, so this may take less than a week depending on your pattern of issuance.

Note that having a large number of pending authorizations is generally the result of a buggy client. If you’re hitting this rate limit frequently you should double-check your client code."

If anyone is still getting messages from Virtualmin indicating that it is re-trying cert requests rapidly, please let us know.

I'm still getting it.

Requesting a certificate for hosting.cloudapps.northwestern.edu from Let's Encrypt .. .. request failed : Failed to request certificate :

Parsing account key... Parsing CSR... Registering account... Already registered! Verifying hosting.cloudapps.northwestern.edu... Traceback (most recent call last): File "/usr/share/webmin/webmin/acme_tiny.py", line 202, in main(sys.argv[1:]) File "/usr/share/webmin/webmin/acme_tiny.py", line 198, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca) File "/usr/share/webmin/webmin/acme_tiny.py", line 109, in get_crt raise ValueError("Error requesting challenges: {0} {1}".format(code, result)) ValueError: Error requesting challenges: 429 { "type": "urn:acme:error:rateLimited", "detail": "Error creating new authz :: Too many currently pending authorizations.", "status": 429 }

You may have to just wait for a while (like a couple of days)

Still not working, and I need to get this fixed.

Should I stop the Webmin service all together? Would that fix the issue so that Webmin doesn't send anymore requests to Let's Encrypt?

Requesting a certificate for hosting.cloudapps.northwestern.edu from Let's Encrypt .. .. request failed : Failed to request certificate :

Parsing account key... Parsing CSR... Registering account... Already registered! Verifying hosting.cloudapps.northwestern.edu... Traceback (most recent call last): File "/usr/share/webmin/webmin/acme_tiny.py", line 202, in main(sys.argv[1:]) File "/usr/share/webmin/webmin/acme_tiny.py", line 198, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca) File "/usr/share/webmin/webmin/acme_tiny.py", line 109, in get_crt raise ValueError("Error requesting challenges: {0} {1}".format(code, result)) ValueError: Error requesting challenges: 429 { "type": "urn:acme:error:rateLimited", "detail": "Error creating new authz :: Too many currently pending authorizations.", "status": 429 }

Jamie, for folks updating to the new Webmin, would the fixes included with it resolve existing SSL cert schedules, in addition to newly created ones?

Should I just remove the auto-renewals for now?

Also, I created a new site in Virtualmin, and when I go to request a cert, I'm getting the same error.

Sorry for all the trouble there! Jamie should have some additional thoughts for us regarding that, I'm waiting to see what he thinks about all that.

Yes, a new Virtualmin install should apply this backoff to all existing domains and renewals.

You can change the renewal period (or turn it off) for existing domains by SSHing in and using a command like :

virtualmin modify-web --domain example.com --letsencrypt-renew 4

to change the renewal period to 4 months, or

virtualmin modify-web --domain example.com --no-letsencrypt-renew

I wanted to try this one more time and now I'm getting a different error, something that relates to Virtualmin's default port:

Requesting a certificate for hosting.cloudapps.northwestern.edu, cloudapps.northwestern.edu from Let's Encrypt .. .. request failed : Failed to request certificate :

Parsing account key... Parsing CSR... Registering account... Already registered! Verifying hosting.cloudapps.northwestern.edu... Traceback (most recent call last): File "/usr/share/webmin/webmin/acme_tiny.py", line 202, in main(sys.argv[1:]) File "/usr/share/webmin/webmin/acme_tiny.py", line 198, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca) File "/usr/share/webmin/webmin/acme_tiny.py", line 122, in get_crt resp = urlopen(wellknown_url) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 467, in error result = self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "/usr/lib/python2.7/urllib2.py", line 429, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 447, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open context=self._context) File "/usr/lib/python2.7/urllib2.py", line 1167, in do_open h = http_class(host, timeout=req.timeout, **http_conn_args) File "/usr/lib/python2.7/httplib.py", line 1258, in __init__ source_address) File "/usr/lib/python2.7/httplib.py", line 751, in __init__ (self.host, self.port) = self._get_hostport(host, port) File "/usr/lib/python2.7/httplib.py", line 792, in _get_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: '10000.well-known'

sespit - what domain name are you requesting a cert for?

hosting.cloudapps.northwestern.edu, which is the URL we use for Webmin/Virtualmin

That is very very odd - Virtualmin never requests a cert for port 10000, only the default HTTPS port 443.

Did you make any local modifications to acme_tiny.py ?

I have just started getting this error, which is quite weird as the bug was supposedly fixed 6 months ago!

I've only noticed because I have just created a new site with SSL and requested a LE cert, which failed with the "Error creating new authz :: Too many currently pending authorizations" error.

Further investigation shows a test site I set up about 8 months ago with a LE cert. This is set to auto renew at 2 months, however has not been renewed since it was created. I wonder if it's been sat there retrying for all that time? Blocking any other renewals and now blocking me creating a new one?

I have disabled auto renew for all domains now and will wait 24 hrs and see if I can create a new LE cert.

I'm wondering why the fix hasn't prevent this. I keep the server up to date.

I cant find any logs! Where could I check to see what requests are being made and how frequently etc?

Thanks

Are you using the LE tool or Certbot? I'm still getting the error on brand new subdomains. Haven't switched to certbot yet.

I'm just using the default virtualmin Let's Encrypt interface. I'd not heard of cerbot. Maybe worth a shot, but I'd like to stick to using the virtualmin built in functionality if possible.

Virtualmin uses the official letsencrypt tool and can also use Certbot. I'm not sure how VM detects which one to use. I'd like to switch to Cerbot since that seems to be the preferred method to request a cert. Not sure if I have to reconfigure Virtualmin after removing the one tool and reinstalling the other.

Webmin will use letsencrypt if installed, followed by certbot, followed by it's own internal implementation.

Since I've not installed certbot, I guess its safe to say I'm using lets encrypt?

Does this help to answer my issue/questions in #23? Especially where to enable/find any kind of logs to help diagnose?

I'm still getting the error while tying to create a new cert even though auto renewals have now been disabled for nearly 24hs.. maybe its still too soon.

I turned mine off for a few days, even though I never had a cert to begin with, and that seemed to fix the issue. I haven't installed Certbot yet, but I will.

OK yeah its been over 24hrs now and I have been able to generate a LE cert for the new site. Yay! :)

I will re-enable the auto renewal now for the other sites and keep an eye on things.

Still would really like to know where the logs are? I would like to monitor any requests and responses made during certificate creation/renewal.

There aren't any logs specific to Let's Encrypt background refreshes.

Oh that'll be why I cant find them! Maybe it would be a good idea to have some logs? Or perhaps an email or notification of some kind if it fails? Since the consequences can be quite drastic. In this case, once the cert had expired it seems virtualmin started serving up the next site with a valid cert. (how come it didn't serve the expired cert?) Of course the browser threw a warning because of an invalid domain, bet even if you ignore the warning and continue, then yo are presented with completely the wrong site! Should I start a new issue to address this? I saw some issues about the wrong site being served for https on domains that have no SSL enabled, but I think that is slightly different.

There is a notification via email if the renewal fails ... but nothing if it succeeds (by default).

Also, the upcoming 5.99 release of Virtualmin will display all domains with expired or nearly expired certs on the System Information page.

That's awesome. We have over 70 domains on our Virtualmin instance, so seeing all those domains with expiring certs in one place will be very handy!

Where does the error email go? I must have not configured the address correctly.

The new feature sounds really good, I look forward to its release :)

It should go to the contact email address for the domain, set on the Edit Virtual Server page.

How do I go about switching from the letsencrypt plugin that Virtualmin Pro installed to Certbot? I'm nervous that I'll break our existing LE issued certs.

The reason I ask is because I created a brand new domain and immediately got a too many requests error even though the subdomain is brand new. Something else I gleaned was that the renew for LE was already set to every two months, whereas in the past I had to set that manually. Is Virtualmin Pro issuing a request for a cert upon domain creation?

Requesting a certificate for openshift.apps.northwestern.edu, www.openshift.apps.northwestern.edu from Let's Encrypt .. .. request failed : Failed to request certificate :

Parsing account key... Parsing CSR... Registering account... Already registered! Verifying openshift.apps.northwestern.edu... Traceback (most recent call last): File "/usr/share/webmin/webmin/acme_tiny.py", line 235, in main(sys.argv[1:]) File "/usr/share/webmin/webmin/acme_tiny.py", line 231, in main signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, args.dns_hook, args.cleanup_hook, log=LOGGER, CA=args.ca) File "/usr/share/webmin/webmin/acme_tiny.py", line 122, in get_crt raise ValueError("Error requesting challenges: {0} {1}".format(code, result)) ValueError: Error requesting challenges: 429 { "type": "urn:acme:error:rateLimited", "detail": "Error creating new authz :: too many currently pending authorizations", "status": 429 }

That message means that you've had too many failed cert requests recently - switching to certbot won't fix it. You should wait 24 hours, and try again.

But the domain is literally brand new. Additionally, we have a request exemption with Let's Encrypt for the entire *.northwestern.edu domain. Good for thousands of requests per month, and currently we're one of the only groups using LE.

Aren't the Let's Encrypt limits applied on a per-account basis though, not per-domain?

When you add virtual servers, the Let's Encrypt SSL certificate does not work for the new virtual servers because the new virtual server is not in the list of valid domains. If you were to add the ability to revoke a cert on the Manage SSL -> Let's Encrypt page that could be a work around. Since you can not do a renew now, all the servers are without a valid cert in 90 days?

DavidKuykendall - as a work-around, you can go to the Manage SSL Certificate page for the master domain, and re-request the Let's Encrypt cert. Then it will include all the new alias domains.

JamieCameron - I made a mistake in not using the right terms. The error should properly be stated as: If you create a virtual server with ssl cert from Let's Encrypt, every alias domain you add to it after this cert will not work with the cert, and you can not generate a new Let's Encrypt cert; renewing does not work either.

JamieCameron - your comment "DavidKuykendall - as a work-around, you can go to the Manage SSL Certificate page for the master domain, and re-request the Let's Encrypt cert. Then it will include all the new alias domains."

I thought Let's Encrypt was erroring out, but the following leads me to think it is Virtualmin.

Virtualmin errors out on the generation of the new Let's Encrypt cert. Might be because it already has one for the first domain in the list OR it might be because it does not have one for the remaining alias domains in the list. I do wish I had saved the long error spew.

I found a work-around that DID work in that I did a self-signed cert, and then a new Let's Encrypt cert. All alias domain websites now have a valid cert.

I sounds like there are a couple of bugs here. First, when adding a new alias domain, ideally the SSL cert would be re-generated from Let's Encrypt to include the alias.

Second, even if you do this manually, it is failing. I'd be interested to see the full error message that happens in this case.

FYI, in the next release adding an alias to a domain with SSL enabled will trigger re-generation of the Let's Encrypt cert.

Fixed Pending next release. (Correct?)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.