Let's Encrypt DNS challenge method fails when using slave autoconfiguration

So returning to the original issue for a bit, I added the wait parameter to /etc/webmin/webmin/config (it was actually already there but set to 10 so I increased it to 60). This solved the problem in the sense that Virtualmin still failed to renew certificates, but then a little while later it retried and it renewed successfully. So Virtualmin e-mails me now with a failure message and then a little while later tries again automatically and the renewal is successful.

Is there a way to make Virtualmin renew the certificate successfully the first time/try, without failing first?

As an example of what I am seeing now, even after changing the Let's Encrypt DNS wait time in /etc/webmin/webmin/config from 10 seconds (the existing value) to 60 seconds (the new value), I am still getting errors like this when Virtualmin goes to renew a certificate, several days after making the change and restarting Webmin:

An error occurred requesting a new certificate for sicilyspizzaeaston.com, www.sicilyspizzaeaston.com from Let's Encrypt :

DNS-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: dns-01 challenge for sicilyspizzaeaston.com dns-01 challenge for www.sicilyspizzaeaston.com Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Waiting for verification... Challenge failed for domain sicilyspizzaeaston.com Challenge failed for domain www.sicilyspizzaeaston.com dns-01 challenge for sicilyspizzaeaston.com dns-01 challenge for www.sicilyspizzaeaston.com Cleaning up challenges Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server:

Domain: sicilyspizzaeaston.com Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.sicilyspizzaeaston.com

Domain: www.sicilyspizzaeaston.com Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.sicilyspizzaeaston.com

I have confirmed that the domain is using Virtualmin's name servers (the primary and the slaves I configured through Virtualmin) and that all DNS servers are functioning properly and accepting requests.

Yes. I control all four name servers (one master and three slaves), and all slave servers were set up using the DNS Slave Auto-Configuration guide in the Virtualmin docs. I would assume BIND is set up as described on the master system but I haven't changed any settings other than following Virtualmin's official slave DNS configuration guide. I did however do a DNS test where I added a record to a zone on the master (Virtualmin > Server Configuration > DNS Records) and then did a DNS lookup on one of the slaves and the record was present on that slave. I don't remember how long I waited in between adding the record and performing the test though (this was a few weeks ago when trying to resolve a different issue).

Hmm...I ran the command 'cat /var/log/messages | grep "sicilyspizzaeaston.com"' on the master and the slave servers after re-attempting to request the Let's Encrypt certificate for the domain but no results were returned on any of the four servers.

No, we unfortunately don't have a firewall in place right now because when I set up this new cluster of systems I realized FirewallD is not supported on the Linode-provided kernel they are currently running.

I just added a test DNS record to the domain on Virtualmin and less than 3 minutes later ran the following commands, trying to retrieve the A record I just added across all four of my name servers (the master and three slaves, with ns1 being the master):

lmerrill@AirshockMBP ~ % host -t a test-1.sicilyspizzaeaston.com ns1.jemediacorp.com Using domain server: Name: ns1.jemediacorp.com Address: 45.79.158.84#53 Aliases:

test-1.sicilyspizzaeaston.com has address 45.79.158.84 lmerrill@AirshockMBP ~ % host -t a test-1.sicilyspizzaeaston.com ns2.jemediacorp.com Using domain server: Name: ns2.jemediacorp.com Address: 45.56.99.222#53 Aliases:

test-1.sicilyspizzaeaston.com has address 45.79.158.84 lmerrill@AirshockMBP ~ % host -t a test-1.sicilyspizzaeaston.com ns3.jemediacorp.com Using domain server: Name: ns3.jemediacorp.com Address: 45.79.131.13#53 Aliases:

test-1.sicilyspizzaeaston.com has address 45.79.158.84 lmerrill@AirshockMBP ~ % host -t a test-1.sicilyspizzaeaston.com ns4.jemediacorp.com Using domain server: Name: ns4.jemediacorp.com Address: 66.228.34.214#53 Aliases:

test-1.sicilyspizzaeaston.com has address 45.79.158.84

Edit: All commands were ran on my local MacBook Pro connected to my apartment's wireless network. They were not ran on any server.

The test was successful. I added the DNS record test-2.sicilyspizzaeaston.com at 9:30 AM EST this morning and then immediately went into Terminal.app on my Mac and ran the host command with all 4 name servers as I did yesterday. The command also ran at 9:30 AM, just after I added the record, and here are the results:

Using domain server: Name: ns1.jemediacorp.com Address: 45.79.158.84#53 Aliases:

test-2.sicilyspizzaeaston.com has address 45.56.99.222 Using domain server: Name: ns2.jemediacorp.com Address: 45.56.99.222#53 Aliases:

test-2.sicilyspizzaeaston.com has address 45.56.99.222 Using domain server: Name: ns3.jemediacorp.com Address: 45.79.131.13#53 Aliases:

test-2.sicilyspizzaeaston.com has address 45.56.99.222 Using domain server: Name: ns4.jemediacorp.com Address: 66.228.34.214#53 Aliases:

test-2.sicilyspizzaeaston.com has address 45.56.99.222

I'm not quite sure how I would monitor that since those records get added and removed quickly by Virtualmin correct? What would you suggest to be the best way to be able to spot the record before it gets removed when the request fails?

OK, so as time goes by I'm learning more and more about this issue. Virtualmin tried a few times over the course of today to renew a certificate for another domain that we host and control the DNS for, but failed every time with the same errors I've been seeing. However I just got an e-mail from Virtualmin saying a certificate for that domain was successfully requested and installed from Let's Encrypt. Same domain that was failing earlier, and we've made no changes to our Virtualmin server today. So I'm not sure why it would be failing one minute but succeeding the next, it's very odd.

That makes sense. So you're saying if Let's Encrypt queries the slaves instead of the master it will fail? How can we resolve that issue? I mean we know that when I add a DNS record manually, I can look it up on all the slave servers, as demonstrated in our test a few days ago.

I already used that setting, I set it in /etc/webmin/webmin/config as instructed (see my above comment) and I am still having issues. Virtualmin attempted to renew a certificate for one of our domains this evening and of course the DNS challenge method failed, but searching the logs for the acme-challenge DNS record produced no results on either the master or any of the slaves.

Any update from the Virtualmin guys on this? We are starting to launch new Websites for some of our new clients and are having problems issuing SSL certificates for those sites because of this ongoing problem. Even with the Let's Encrypt DNS wait time set to 60 in /etc/webmin/webmin/config certificate issuance only succeeds every once in a while, probably when the master DNS server is queried by the Let's Encrypt client, and even though manually adding a DNS record in Virtualmin causes it to propagate to the slaves immediately, the same cannot be said concretely for when it's done automatically via Let's Encrypt.

I was doing some testing more testing this evening and discovered something very interesting! The issue appears to be down to the fact that Virtualmin only copies newly-added DNS records to slave servers if they are added manually from within the GUI, not automatically via Let's Encrypt. I tested this by starting a certificate renewal request, then using the 60 seconds wait time to go into Virtualmin > Server Configuration > DNS Records, clicking on the _acme-challenge TXT record that had just been created, then hitting Save. If done before the 60 seconds expired, the record would be copied to the slave servers in time for the renewal request to complete successfully.

So in other words, when requesting a certificate renewal, Virtualmin does not copy the generated DNS record to the slave servers at all, but when adding a DNS record manually via the GUI it does.

Jamie, I can still provide you with login information to my system if you'd like, but unless my system is configured in some super weird way, this may be a Virtualmin-specific issue after all; does the Let's Encrypt renewal routine skip calling whatever code that copies DNS records to slave servers?

Any update on this?

Hi Jamie,

You're certainly welcome to access my system to take a look at this potential bug. I just logged into Virtualmin and it says that remote logins are already activated, so I must have turned them on for an earlier issue or something. The IP address of the system is 45.79.158.84. Let me know if you need any other info to log in. Thanks!

teamliftfitness.com (and the associated www.teamliftfitness.com)

Thanks for your prompt repair of this issue, Jamie. Do you mind if I ask what option in Webmin's BIND module was configured incorrectly? What did you set the correct config value to?

Hi Jamie, it seems every time we think the issue is fixed something else pops up LOL. After you said last night that you found and fixed the Virtualmin bug that was causing this issue, Virtualmin tried to auto-renew one of our certificates this morning (for a different domain than the one I gave you yesterday) and DNS0based validation failed once again with pretty much the same errors as before. Remote logins to my system are still active if you want to log back in and take a look at things again. Here's the output from Virtualmin:

An error occurred requesting a new certificate for bendersdaylightdonuts.com, www.bendersdaylightdonuts.com from Let's Encrypt : DNS-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: dns-01 challenge for www.bendersdaylightdonuts.com Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Waiting for verification... Challenge failed for domain www.bendersdaylightdonuts.com dns-01 challenge for www.bendersdaylightdonuts.com Cleaning up challenges Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server:

Domain: www.bendersdaylightdonuts.com Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.bendersdaylightdonuts.com

Jamie,

I am continuing to get failed certificate renewal e-mails from Virtualmin as if the bug you fixed had come back or something. Is there something I should check on my system? I posted an example of the e-mail in an earlier reply from the other day. It almost seems like the records are not being copied again because every few hours Virtualmin will succeed in renewing a certificate for a domain that failed before, making me think that Let's Encrypt is hitting the primary DNS server during those times and the slave servers at all other times.