Undefined subroutine &main::restart_zone

Submitted by JEMEDIACORP on Mon, 10/21/2019 - 09:01 Pro Licensee

After updating to Virtualmin 6.08, we're receiving the following error when attempting to generate a Let's Encrypt certificate (either while creating a new virtual server or, when that fails, going in and trying to generate it again for the same virtual server). Renewing existing certificates does not seem to be affected at this time.

Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl manual-auth-hook command "/etc/webmin/webmin/letsencrypt-dns.pl" returned error code 1 Error output from manual-auth-hook command letsencrypt-dns.pl: Undefined subroutine &main::restart_zone called at /usr/libexec/webmin/webmin/letsencrypt-dns.pl line 47.

Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl manual-auth-hook command "/etc/webmin/webmin/letsencrypt-dns.pl" returned error code 1 Error output from manual-auth-hook command letsencrypt-dns.pl: Undefined subroutine &main::restart_zone called at /usr/libexec/webmin/webmin/letsencrypt-dns.pl line 47.

Status: 
Active

Comments

Submitted by JEMEDIACORP on Mon, 10/21/2019 - 09:02 Pro Licensee

Title: Undefined subroutine &main:;restart_zone » Undefined subroutine &main::restart_zone

Submitted by andreychek on Mon, 10/21/2019 - 09:20

Assigned: Unassigned »

Howdy -- thanks for your report!

I'm passing this to Jamie for comment.

Submitted by JEMEDIACORP on Mon, 10/21/2019 - 21:55 Pro Licensee

Hi Jamie, and thanks for your prompt response to this issue. I applied the two files from the commit you linked in your post (letsencrypt-cleanup.pl and letsencrypt-dns.pl) to /usr/libexec/webmin/webmin on my system and restarted Webmin, but now get the following error when I try to create a new virtual server with SSL and Let's Encrypt enabled:

Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl manual-auth-hook command "/etc/webmin/webmin/letsencrypt-dns.pl" returned error code 2 Error output from manual-auth-hook command letsencrypt-dns.pl: Failed to run /usr/libexec/webmin/webmin/letsencrypt-dns.pl : No such file or directory at /etc/webmin/webmin/letsencrypt-dns.pl line 12.

Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl manual-auth-hook command "/etc/webmin/webmin/letsencrypt-dns.pl" returned error code 2 Error output from manual-auth-hook command letsencrypt-dns.pl: Failed to run /usr/libexec/webmin/webmin/letsencrypt-dns.pl : No such file or directory at /etc/webmin/webmin/letsencrypt-dns.pl line 12.

Note that if I then go into Server Configuration > SSL Certificate > Let's Encrypt on that same virtual server, the generation process works flawlessly. I have confirmed that both /usr/libexec/webmin/webmin/letsencrypt-cleanup.pl and /usr/libexec/webmin/webmin/letsencrypt-dns.pl do exist, have the new code in them from the Github commit, and have permissions set to 755.

Submitted by JamieCameron on Tue, 10/22/2019 - 01:06

Make sure that /usr/libexec/webmin/webmin/letsencrypt-dns.pl and the other file start with #!/usr/bin/perl

Submitted by JEMEDIACORP on Tue, 10/22/2019 - 01:38 Pro Licensee

They actually both (letsencrypt-cleanup.pl and letsencrypt-dns.pl) start with:

!/usr/local/bin/perl

But /usr/local/bin/perl does not exist on my system. Instead, it looks like it's found in /usr/bin/perl (according to the "which" command). Could this be the problem? I didn't change any lines in the file, I just copied and pasted the patched files from the Github commit.

Submitted by andreychek on Tue, 10/22/2019 - 10:08

Understood, that's what Jamie means above... you'd need to change the path to point to the Perl binary in those files.

It's part of our build process to update the Perl path for the OS in question, but if the file is being manually pulled from git like the case here, the path would just need to be updated at the top of those files.

Submitted by JEMEDIACORP on Tue, 10/22/2019 - 11:16 Pro Licensee

I changed the path at the top of the two files from /usr/local/bin/perl to /usr/bin/perl (the correct one for my system) and that solved that particular problem but not the issue as a whole. I am no longer getting specific errors from running the commands as I was yesterday, but the Let's Encrypt certificate is still not issuing when a virtual server is created. If I go back into Virtualmin after the server is created, go to Server Configuration > SSL Certificate > Let's Encrypt and try to generate it that way, it works just fine. But before Virtualmin 6.08, a Let's Encrypt certificate for a domain would be issued successfully during the virtual server creation process.

Here is the full log of what happens as Virtualmin tries to request an SSL certificate during virtual server creation (this is a top-levle server even though it looks like a subserver):

Requesting a certificate for vm-test86.jemcdev.com, www.vm-test86.jemcdev.com from Let's Encrypt .. .. request failed : Web-based validation failed :

Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: http-01 challenge for vm-test86.jemcdev.com http-01 challenge for www.vm-test86.jemcdev.com Using the webroot path /home/vm-test86/site for all unmatched domains. Waiting for verification... Challenge failed for domain vm-test86.jemcdev.com Challenge failed for domain www.vm-test86.jemcdev.com http-01 challenge for vm-test86.jemcdev.com http-01 challenge for www.vm-test86.jemcdev.com Cleaning up challenges Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server:

Domain: vm-test86.jemcdev.com Type: unauthorized Detail: Invalid response from https://vm-test86.jemcdev.com/.well-known/acme-challenge/EbJIOBRQxcngHk_... [45.79.193.237]: "\n\n404 Not Found\n\n

Not Found

\n

<

p"

Domain: www.vm-test86.jemcdev.com Type: unauthorized Detail: Invalid response from https://www.vm-test86.jemcdev.com/.well-known/acme-challenge/dYKDbR4j_eq... [45.79.193.237]: "\n\n404 Not Found\n\n

Not Found

\n<p"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. DNS-based validation failed : Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: dns-01 challenge for vm-test86.jemcdev.com dns-01 challenge for www.vm-test86.jemcdev.com Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl Waiting for verification... Challenge failed for domain vm-test86.jemcdev.com Challenge failed for domain www.vm-test86.jemcdev.com dns-01 challenge for vm-test86.jemcdev.com dns-01 challenge for www.vm-test86.jemcdev.com Cleaning up challenges Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server:

Domain: vm-test86.jemcdev.com Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.vm-test86.jemcdev.com

Domain: www.vm-test86.jemcdev.com Type: dns Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.vm-test86.jemcdev.com

Submitted by Jfro on Tue, 10/22/2019 - 11:33

Could be this not 100% ready while   Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: dns-01 challenge for vm-test86.jemcdev.com dns-01 challenge

https://www.virtualmin.com/comment/818433#comment-818433

only pointing out. ;)

Submitted by JEMEDIACORP on Tue, 10/22/2019 - 11:34 Pro Licensee

Maybe but the DNS was ready by the time Virtualmin requested the certificate, because it creates the DNS zone (and restarts the DNS server) before it gets to the certificate part. Also, this worked flawlessly before Virtualmin 6.08 came out last week.

Submitted by JEMEDIACORP on Wed, 10/23/2019 - 09:09 Pro Licensee

Oh, thanks for spotting that, I didn't even know they had moved from v1 to v2. I just find it odd this works for renewing existing certificates on existing virtual servers but not for creating new ones when a virtual server is created; after some testing I realized I have to wait 2 minutes or more after the completion of the virtual server creation process before I can generate a certificate without getting an error. Before Virtualmin 6.08 this process happened automatically during virtual server creation with no errors.

Submitted by Jfro on Wed, 10/23/2019 - 09:58

don't know if the pre-check option for the LE script can help with that. ( could be helpfull to prevent the limit part) Sorry also only pointing this out no knowledge from scripts.

Only that v1 for new certs could have problems, renew fasing out at LE for v1 is later but soon, for new LE certs is v1 already EOL and fased out partly.

Handy to look to https://letsencrypt.status.io/

https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/2 scroll down here for more

Submitted by JEMEDIACORP on Thu, 10/24/2019 - 13:04 Pro Licensee

Any update on this from the Virtualmin team? I am still unable to generate SSL certificates automatically at virtual server creation time, but if I wait anywhere between 1 to 10 minutes (and restart HTTPD which I never had to do before now to get certificates to be generated at creation time), Virtualmin will generate them without problems. This doesn't always work though, sometimes even when I wait a few minutes and run the generate-letsencrypt-cert command from the CLI it still doesn't work. I can provide login access to my server if this would be easier to troubleshoot that way.

Submitted by andreychek on Thu, 10/24/2019 - 13:13

I asked Jamie for his input on this one, we'll see what he has to say!

Submitted by JEMEDIACORP on Thu, 10/24/2019 - 14:04 Pro Licensee

Thanks. Let me know if you need further details, login access for additional troubleshooting, etc.

Submitted by JamieCameron on Thu, 10/24/2019 - 21:38

Do you have the official certbot let's encrypt client installed? That should resolve the issue of incompatibility with the built-in let's encrypt client.

Submitted by JEMEDIACORP on Thu, 10/24/2019 - 21:43 Pro Licensee

Yeah, I think it's already installed. Doing rpm -qa | grep "certbot" on my system returns this:

certbot-0.39.0-1.el7.noarch python2-certbot-0.39.0-1.el7.noarch

Submitted by JamieCameron on Fri, 10/25/2019 - 20:34

Ok .. any way we can get login access to this system?

Submitted by JEMEDIACORP on Fri, 10/25/2019 - 20:41 Pro Licensee

Sure. I've enabled remote access in Virtualmin that will remain active until Sunday, october 27. Virtualmin stated an e-mail with details has been sent to remote@virtualmin.com.

Submitted by JamieCameron on Sat, 10/26/2019 - 21:54

Thanks! What's the IP of your system?

Submitted by JEMEDIACORP on Sat, 10/26/2019 - 21:57 Pro Licensee

45.79.193.237

Submitted by JEMEDIACORP on Mon, 10/28/2019 - 12:43 Pro Licensee

Jamie,

Were you able to gain remote access to my system before the keys expired yesterday evening? If not I can re-enable the remote access through Virtualmin.

Submitted by JEMEDIACORP on Thu, 10/31/2019 - 10:58 Pro Licensee

Happy Halloween! Any update on this issue from the Virtualmin guys?

Submitted by JamieCameron on Sun, 11/03/2019 - 13:42

Actually I'm still unable to SSH into your system - it doesn't accept my SSH key.

Submitted by JEMEDIACORP on Sun, 11/03/2019 - 13:47 Pro Licensee

Sorry about that Jamie. I've re-enabled Virtualmin remote logins, this time indefinitely, to give you another opportunity to log into the system. Also I had to change the SSH port to 2222 (so I could run a Git-based SSH server on 22) so please use that port when connecting.

Submitted by JEMEDIACORP on Sun, 11/10/2019 - 12:47 Pro Licensee

Any update on this? I get the same error now on a brand-new Virtualmin 6.08 installation I set up on a new server (it's GPL not Pro though for now).

Submitted by trendzetter on Fri, 11/29/2019 - 03:48

I had the exact error as mentioned above:

Undefined subroutine &main::restart_zone called at /usr/share/webmin/webmin/letsencrypt-dns.pl line 47

I like to confirm that after applying the changes in https://github.com/webmin/webmin/commit/771be1a754fafa02abb5d5670f3ba4a6... to the file in /usr/share/webmin/webmin/letsencrypt-dns.pl (adding bind8:: on line 47 before restart_zone) the issue with certbot renew is fully resolved on my host.

Thanks Jamie for the wonderful software that is Webmin!

Submitted by NeptuneUK on Sat, 12/14/2019 - 11:44

Hi,

Just wanted to say thanks for the solution offered here (namely the issue regarding LE and DNS validation of wildcard certs):

"Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl manual-auth-hook command "/etc/webmin/webmin/letsencrypt-dns.pl" returned error code 1 Error output from manual-auth-hook command letsencrypt-dns.pl: Undefined subroutine &main::restart_zone called at /usr/libexec/webmin/webmin/letsencrypt-dns.pl line 47."

The solution presented in comment #3 along with #5 worked for me.

Keep up the great work on Virtualmin!!

Submitted by rubenz on Tue, 12/24/2019 - 08:25

Hey guys, I desperately need this fixed.

While the fix above (comment #3) worked for regular domains, the ones that have listed domain names still fail. I cannot update them from both from web interface and from command line manually running the certbot.

I have a setup like this:

Request certificate for: Domain names listed here:

DNS verification outputs this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for mail.example.com
Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Waiting for verification...
Challenge failed for domain mail.example.com
dns-01 challenge for mail.example.com
Cleaning up challenges
Running manual-cleanup-hook command: /etc/webmin/webmin/letsencrypt-cleanup.pl
Some challenges have failed.
IMPORTANT NOTES:
- The following errors were reported by the server:

   Domain: mail.example.com
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up TXT for
   _acme-challenge.mail.example.com


And web based verification gives this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.example.com
Using the webroot path /home/example/public_html for all unmatched domains.
Waiting for verification...
Challenge failed for domain mail.example.com
http-01 challenge for mail.example.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
- The following errors were reported by the server:

   Domain: mail.example.com
   Type:   unauthorized
   Detail: Invalid response from
   http://mail.example.com/.well-known/acme-challenge/zu6MLeRGKriFkWSC_nsRM23VpYbxlapwBcxaCO822kc
   [213.133.111.28]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx/1.16.1</ce"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.


Basically, this happens for any listed subdomains (I actually have several besides mail.example.com).

Do you guys have a clue how to overcome this? Its been 2 months already I have invalid certificates - hoping it'll get fixed without me diving into the issue and trying to manually fix it/request a certificate.

No server configurations were changed, and it worked like a charm before.

My system is:

  • CentOS Linux 7.7.1908
  • Webmin: 1.932
  • Virtualmin: 6.08
  • Usermin: 1.780

Thanks!

Submitted by JamieCameron on Tue, 12/24/2019 - 13:04

What is the actual domain name that this fails for? I'd like to check their reachability from the outside..

Also, does it still fail if you don't include mail. in the list of domains to request a cert for?

Submitted by rubenz on Wed, 12/25/2019 - 17:59

Hey Jamie! Thank you for getting back to me.

I currently have 2 domains with this issue. Other domains with default settings, or without mail.* are fine.

A little bit more details:

  • If I have only domain.com and www.domain.com and remove mail.domain.com from listed domains, it works.
  • Wildcard doesn't work.

Removing mail. and other unmatched domains always solves the issue. For one of servers there are quite a lot of them, e.g. dev., lab., primary. etc. Part of those subdomains existed historically, but currently they are not set up, I just included them to be valid with the main certificate, and some of those subdomains is actually in use (mail.). It doesn't matter, the error shows up if any of those subdomains isn't set up as servers in virtualmin. Having A records in DNS doesn't help.

These are actual domains for you to check (I'll edit this comment and remove actual domains in few days, please don't quote domain names, just refer them by a number):

  1. domain1.com (issue exists, has several unmatched subdomains)
  2. domain2.com (issue exists, has only mail.*
  3. domain3.com (OK) // no mail.*, default settings
  4. domain4.com (OK) // no mail.*, "Domain names listed here" is chosen.

Submitted by skelgaard on Wed, 12/25/2019 - 13:58 Pro Licensee

after a lot of debugging, i have gotten mine to work... this error only happens on dns validation. how i got past mine, was to do

You can increase the wait time for DNS replication by adding the line letsencrypt_dns_wait=60 to /etc/webmin/webmin/config

i then when to my domain and did a ssl request. while this was running, i did a manually restart of the dns server and the request worked and renewed

Submitted by rubenz on Wed, 12/25/2019 - 15:14

skelgaard

Thanks for sharing your experience. Still doesn't work for me, but a line about dropped connection gets added to the log:

Running manual-auth-hook command: /etc/webmin/webmin/letsencrypt-dns.pl
Waiting for verification...
Resetting dropped connection: acme-v02.api.letsencrypt.org
Challenge failed for domain example.com

I have a wrong .txt domain for main example.com and still NXDOMAIN's for the rest. When do you restart the named manually? How long do you wait to restart after initiating the renew process from web interface?

Submitted by skelgaard on Wed, 12/25/2019 - 16:48 Pro Licensee

and i did cat /var/lib/bind/domain.dk.hosts to see when the record was there, before i did the restart

Submitted by rubenz on Wed, 12/25/2019 - 18:22

Wow, this did the trick!

skelgaard, thank you very much for the hints! My configuration slightly differs, I'll sum up step by step what I did for other people if they face this issue as well.

KUDOS TO skelgaard!

I have Centos 7 with BIND 9.11.4-P2-RedHat-9.11.4-9.P2.el7

  • First, apply the patch from #3 comment
  • Increase DNS timeout for you to get a little more time for manual restart. In the file /etc/webmin/webmin/config set letsencrypt_dns_wait=60
  • request the certificate from web or commandline
  • in my case hosts file was located here: /var/named/domain.com.hosts. Open that file and constantly monitor.
  • Each time when there is an acme challenge entry is added, manually restart bind (service named restart)

When it starts deleting entries, stop restarting the bind. At this point most likely you've passed the verification if you didn't get banned previously by letsencrypt due to multiple auto-renew failures. If you get an error message, that hints that you may be banned, just disable auto-renew and wait a day.

P.S. Jamie, I'm removing my actual domains from my previous comment, could you please take a look on the fix above? Additionally we can check the HTTP renew as well. It always fails for me.

Thanks, guys!

Submitted by skelgaard on Wed, 12/25/2019 - 18:14 Pro Licensee

i think he is already fixing it... i can see a new version is roling out, but not on the changelog or download link yet...

# apt changelog webmin
E: Failed to fetch changelog:/webmin.changelog  Changelog unavailable for webmin=1.940

Submitted by rubenz on Wed, 12/25/2019 - 18:23

Wow, that sounds awesome!

I'm seriously considering to buy virtualmin just to show a support for the team even if GPL fully covers my needs.

Thanks, guys!