Let's Encrypt certificate renewal failed!

Submitted by craigh on Wed, 01/08/2020 - 21:51 Pro Licensee
  • Virtualmin: 6.08 Pro
  • Webmin: 1.940

I and users are getting this error email from automated Let's Encrypt renewals:

An error occurred requesting a new certificate for example.com, www.example.com from Let's Encrypt : Web-based validation failed : <pre>sh: certonly: command not found
</pre>

I seen a few tickets open (and closed) about LE errors, and I thought one would address this, but none do and none have that specific "command not found" error.

This started about 1 January. The server has only been updated via "yum", so I don't know why this has suddenly started.

Any idea?

Craig

Status: 
Active

Comments

Submitted by andreychek on Wed, 01/08/2020 - 22:11

Howdy -- thanks for contacting us!

Try installing certbot with this command:

yum install certbot

After doing that, are you able to renew your SSL Certificates?

Submitted by craigh on Wed, 01/08/2020 - 22:28 Pro Licensee

Well, it's in the repo, but why would it suddenly not be available? That's more interesting to me at this point.

Ilia's picture
Submitted by Ilia on Thu, 01/09/2020 - 07:10

Well, it's in the repo, but why would it suddenly not be available? That's more interesting to me at this point.

It's not installed automatically, because, it's not marked as a hard-dependency. New installs (when installed with install.sh script) will have it installed automatically. Upcoming 1.941 will work the old way as well.

Submitted by craigh on Thu, 01/09/2020 - 11:16 Pro Licensee

Thanks, but you lost me there. As I said, this was installed and working up until about 1 January, and now suddenly it been removed. Virtualmin on this server was indeed installed with install.sh, so your message makes no sense.

Ilia's picture
Submitted by Ilia on Thu, 01/09/2020 - 13:27

..and now suddenly it been removed.

What have been removed? In case you are talking about Let's Encrypt support, without certbot command - then yes, 1.940 requires certbot command to be installed, in order to work properly. However, upcoming 1.941 will also be able to fall-back to only using ACME script but this time with support of new v2 API.

Submitted by craigh on Thu, 01/09/2020 - 20:47 Pro Licensee

I'm confused why you, and not Jamie, are addressing this issue. I'm trying to find a reason for the sudden "breaking" of Virtualmin and you're trying to tell me I don't have an issue. Automatic LE certificate renewals were working on 31 December, and were broken on 1 January (or thereabouts) without any action by me. Virtualmin is managing this server. I'm trying to find answers and you're telling a paying customer that I don't need answers and are in fact blocking me from an avenue to find those answers.

Please have Jamie address this ticket. You're not helping.

Submitted by andreychek on Thu, 01/09/2020 - 21:58

Craigh, Ilia is part of our team, and is here to help. He's very good at what he does.

Ilia is very involved with Webmin and Virtualmin development, and will be taking a more active role in our support tracker here as well.

He did answer your question.

If you don't understand something, that's completely okay, just (politely) ask for some clarification.

Regarding certbot, it's all a bit of a long story, but in short --

Webmin recently began using certbot for Let's Encrypt certificate renewals.

What Ilia said is that for new installs using the install.sh script, it will be automatically installing certbot.

But as you saw, at the moment there is a problem where certbot does need to be manually installed.

And to help resolve that, the next Webmin version is adding in a new built-in client to handle the case where certbot isn't available (so in the future, people won't run into the issue you saw).

The solution to your issue for now though is to just install the certbot command, and that will resolve the issue you're experiencing.

Submitted by craigh on Thu, 01/09/2020 - 23:09 Pro Licensee

Eric, that is the explanation that was asked for, and that I have been looking for since my initial post. Thank-you.

I understand who Ilia is, but I have previously only really noticed him weigh in on Authentic theme issues. I don't doubt his qualifications to weigh in on other issues (especially now that you have vouched for those explicitly), however I would like to point out that my initial reply to him was more than polite and gave him the chance to clarify what he meant, as I didn't understand it and the connection to the problem. Only when he continued to stonewall me (and basically just do what I was told) did I point out that he was being unhelpful and combative, which you can characterise as "impolite" if you want.

If you want more understanding from your customers then I suggest you create an "about" page where you make this information clear to us. Otherwise we're just guessing at who does what. Thanks.

Now that I understand the issue and what has gone wrong with Virtualmin/Webmin, I am willing to carry out the manual installation you suggested (without explanation) in your initial response. Assuming it works I'll be back to report than and close this ticket.

aVitomin's picture
Submitted by aVitomin on Fri, 01/10/2020 - 11:25

"Upcoming 1.941 will work the old way as well."

Any ETA about 941 release ?

Submitted by andreychek on Fri, 01/10/2020 - 15:10

We hope to release it soon, though I unfortunately don't have a more specific ETA.

We know folks are seeing issues with things as they are now, so we want to avoid any unnecessary delays.

Ilia's picture
Submitted by Ilia on Wed, 01/15/2020 - 06:53

Hi,

Webmin 1.941 is out to devel repos, which will go to Virtualmin repos shortly.

You could install it right now to get Let's Encrypt back on track.

Sorry for inconvenience.

Submitted by craigh on Wed, 01/15/2020 - 07:31 Pro Licensee

Hi Ilia, is that in answer to my question?

Submitted by craigh on Wed, 01/15/2020 - 09:09 Pro Licensee

Well, then I again don't understand your answer. I am politely asking for clarification.

Ilia's picture
Submitted by Ilia on Wed, 01/15/2020 - 12:25

Well, then I again don't understand your answer. I am politely asking for clarification.

Sorry for delay. I had an initial assumption, that installing latest Webmin 1.941 would solve all issues with Let's Encrypt for you, and you wouldn't want to take care about logs any more. This is why I recommended you and the other user on this ticket to go and grab it, instead of continuing with digging the logs.

Is there a log somewhere that I can parse to check on the automated renewal process?

Yes, if you still would like to check what's happening upon renewal process, you could take a look at:

/var/log/letsencrypt/letsencrypt.log

By saying this, I assume, that you were using Webmin 1.940 with certbot package (as CentOS 7 has it on default repos).

Submitted by craigh on Wed, 01/15/2020 - 18:53 Pro Licensee

Thanks. I appreciate your concern, but looking at log files is part of my job description and something I do almost every day. The problem is that I don't even know yet if I still have an issue after installing certbot manually as instructed, so I wanted to check the logs or get command-line output showing the expiry dates of all Let's Encrypt certificates on the server. On the positive side the LE tab at Virtualmin -> DOMAIN -> Server Configuration -> SSL Certificate -> Let's Encrypt started working again after I installed certbot, so it's almost certain that everything is working properly; it would just give me peace of mind to know that the automated renewal process is definitely working as well.

The complicating factor is that I do have one user who is still getting error emails (apparently every hour), but that's because one of his aliased domains has expired and he's still awaiting its redemption. Another user is not (I don't think) getting error emails, but according to Virtualmin the last attempted renewal for his certificate was on 12 January, which is odd considering the other user is getting error emails every hour. Neither user's certificate has actually expired yet, but they will within the next month.

Yes, I am using Webmin 1.940 and certbot installed through yum. Manually re-installing 1.941 wouldn't directly address my issue any more than manually installing certbot would/has, hence why I was looking for log (which I did find, thanks) or command-line output. I'll just wait patiently until the certificates I know of have been renewed automatically before I report success.

Submitted by craigh on Thu, 01/16/2020 - 01:44 Pro Licensee

I can confirm that automated renewals are once again working. Thanks.

I'd like to suggest that at Virtualmin -> DOMAIN -> Server Configuration -> SSL Certificate -> Let's Encrypt, where there is currently an error stating the following ...

Last failed renewal 12/01/2020 02:34:23
Renewal failed due to Web-based validation failed

... that there be another annotation stating when the next renewal will be attempted.

Ilia's picture
Submitted by Ilia on Thu, 01/16/2020 - 04:24

Assigned: Unassigned »

I'd like to suggest that at Virtualmin -> DOMAIN -> Server Configuration -> SSL Certificate -> Let's Encrypt, where there is currently an error stating the following ...

I will pass this to Jamie for comment.