Submitted by techdudehosting on Fri, 02/24/2017 - 22:39
Team,
When attempting to provision a virtual server using WHMCS it reaches a timeout. I have enabled rpc calls for the root and my whmcs user account but it isn't working. I have also made sure the hostname and ip address has the virtualmin port and if I click login it will log me into virtualmin from WHMCS. Any thoughts? This is a fresh install with the latest Virtualmin and Webmin packages.
Status:
Closed (fixed)
Comments
Submitted by techdudehosting on Fri, 02/24/2017 - 22:40 Comment #1
Submitted by andreychek on Sat, 02/25/2017 - 08:07 Comment #2
Howdy -- hmm, is there any chance you could share the exact error that you're receiving?
Also, is there a firewall running on your server? If so, you may want to temporarily disable it, just to ensure that it's not interfering.
Submitted by hescominsoon on Sat, 02/25/2017 - 08:09 Pro Licensee Comment #3
Is whmcs running on the same server as virtualmin? if so the firewall isn't the issue. If you have a separate machine for whmcs then you need to see what ports are needed..then you can create a rule that allows that traffic from the whmcs box to your virt box at that port is allowed.
Submitted by techdudehosting on Sat, 02/25/2017 - 10:19 Comment #4
Separate boxes. Firewall allows 10000 on the Virtualmin host. Might not be open on the WHMCS host. Here's the error: Curl Error: 28 - Connection timed out after 60001 milliseconds
Submitted by hescominsoon on Sat, 02/25/2017 - 10:20 Pro Licensee Comment #5
yeppers. make sure the firewalls on both sides allow traffic from the other machine.
Submitted by techdudehosting on Sat, 02/25/2017 - 10:28 Comment #6
I'm checking with the other host about the port now.
Submitted by andreychek on Sat, 02/25/2017 - 11:36 Comment #7
I'd definitely recommend disabling the firewall. The RPC uses ports other than 10000, and it will rule some other things out as well.
However, if it's using RPC you'd want to open ports 10001 - 10010.
Submitted by techdudehosting on Sat, 02/25/2017 - 12:17 Comment #8
At the time only 10000 can be opened. If the need arises then I can apply pressure. However I don't plan to be at this host much longer.
Submitted by hescominsoon on Sat, 02/25/2017 - 12:25 Pro Licensee Comment #9
Disabling the firewall is not a fix.
you need a host that allows you to control your firewalls. I am with one that allows you to setup your own firewalls at the machines....checkout reliablehostingservices.net. Tell them William Warren sent you. You can then setup firewalls on each machine and have full control over the traffic.
Submitted by techdudehosting on Sat, 02/25/2017 - 12:34 Comment #10
Firewall hasn't been disabled. Had host open 10000 on their side. It's where whmcs is and where I put my cPanel clients.
Submitted by endri on Thu, 03/16/2017 - 10:04 Comment #11
Hello Guys,
I am facing the same issue. Both installations are on the same VPS, clean installation. I get no errors in the logs.
In WHMCS when I click Create (to create a new service), the Working... symbol keeps going and never ends. The Virtual Server is actually created successfully on Virtualmin, but WHMCS does not know thus not deploying the information (sending e-mails) nor populating the client account with the new product. I hope someone can give some clue, is very annoying issue.
Best regards to all.
Submitted by JamieCameron on Fri, 03/17/2017 - 00:01 Comment #12
Have you raised this issue with WHMCS?
Submitted by endri on Fri, 03/17/2017 - 02:20 Comment #13
Yeah I did, no reply yet. If I link an other server (same Debian/Virtualmin environment) it works with no problem, I tried last night.
I compare things in the hope to find the difference but nothing is different from the settings point of view, made everything rigorously identical (plan and template). I have enabled PHP Logs but nothing in the logs. It is very annoying. If someone has some clue please help. I will update you if I receive any answer from WHMCS support.
Submitted by endri on Fri, 03/17/2017 - 02:25 Comment #14
To give more information; If the option Accept and Create the server immediately is selected, it will drop an Internal Server Error 500. And no logs at all are recorded.
Submitted by andreychek on Fri, 03/17/2017 - 08:45 Comment #15
The 500 error should definitely produce an error log somewhere. You may want to review the logs in /var/log/virtualmin and /var/log/apache/error_log to see which Apache logs are generating an error at the time that happens.
However, we'd definitely recommend reaching out to the WHMCS folks, as they'd better understand what's going on there.
We're happy to help them out in any way we can though.
Submitted by endri on Fri, 03/17/2017 - 14:47 Comment #16
Hi andreycheck,
Incredibly there is no error generated at all. I got crazy over this. From WHMCS support I received a reply to check permissions and owners, did everything. The fact that WHMCS works if connected to an other Virtualmin server means that there might be no issue with the WHMCS installation. Because I could not solve this, I am reinstalling the VPS hopping that I did some mistake somewhere (which I doubt) previously. I will update you just for information if this will solve the issue. Regards, Endri.
Submitted by endri on Sat, 03/18/2017 - 17:50 Comment #17
Hi everybody,
I reinstalled everything and restored the WHMCS database. Now it works fine. But I am concerned about this issue, I didn't find what was causing it. If it happens again it would be a mess on a populated system...
I have a similar issue CentOS 7 VPS, latest versions of everything.
When I do create package command from WHMCS, I also get the Working...symbol going on forever and never ends.
As with yourself the Virtual Server is created successfully on Virtualmin.
New Hosting Account Information is not sent, same as you.
The client's account is however populated with the new product. So it's "working".
However, overuse is not calculated properly. It appears Overuse is calculated on the Default Plan which has unlimited traffic, as opposed to my "Basic" plan which has a 1 GB limit.
I want to know if it's possible to debug the Remote API / CGI interface? Is there a log file?
Just isolated this problem. Installed a fresh WHMCS. Connect from fresh WHMCS to problematic Virtualmin server, same hung issue. Connect to Virtualmin on new server, working 100%. So I conclude Virtualmin on server A is broken. I highly suspect SSL or something. The annoying this is the remote API does not log much detail at all.
It's like it's stuck for 60 seconds, but yet it does complete the work. Looked at remote.cgi but I don't really want to troubleshoot that deep.
A real lifesaver here would be extremely verbose logging for the remote API but I can't find any setting.
After much testing I have concluded this happens because Virtualmin is very slow when returning results on a small server for list-domains and multiline. I have documented my findings on the WHMCS Feature Requests website https://requests.whmcs.com/topic/virtualmin-provisioning-module-curl-req... but it would be useful if someone from Virtualmin can tell us why doing the remote CGI commands list-domains and multiline often takes more than a minute to complete.
Submitted by JamieCameron on Mon, 04/01/2019 - 22:15 Comment #21
How long does it take if you run
virtualmin list-domains --multiline
from the command line?Hi @JamieCameron,
Thanks for replying. If I run a test script, such as below, then it completes in anywhere from 30 seconds to about 2 minutes. FYI the server spec is: 4 GB RAM, 5 x not busy websites 20 x email addresses, of which some are very busy.
`<?php $result = shell_exec("wget -O - --quiet --http-user=root --http-passwd=password 'https://server.host.com:10000/virtual-server/remote.cgi?program=list-dom...'");
echo $result; `
However, when it runs as part of the WHMCS Cron, I believe it takes much longer. From observation the daily CRON on a server causes quite a lot of load. For example, emails might be sent for overdue invoices, domain renewals, etc. On the server itself, the CRON only finishes 1% of the time in under 60 seconds. We also removed CLAM from a couple of busy mailboxes reducing the time to completion quite a lot.
When I examined the issue with top and the command line I concluded it can't be a server timeout. On the command line, even if it takes very long, the server never times out. It would wait up to 3 minutes to complete. In top I can see
list-domains.cgi
happily running in the background. But I believe WHMCS is using CURL with a hard-coded timeout because the message in the WHMCS module log is:"Curl Error: 28 - Operation timed out after 60000 milliseconds with 59185 bytes received"
I notified WHMCS about another issue where domain usage data is out of sync with the Virtualmin module, and they informed me the Virtualmin module is "encrypted" so I think there might be some kind of deadlock. You know the kind where WHMCS says it's the module provider's problem, and the module provider says it's WHMCS's problem.
Submitted by adamjedgar on Sun, 04/21/2019 - 16:32 Pro Licensee Comment #23
Ahaaaaa finally I place where thus is being discussed.
I run Blesta instead of whmcs, and you know what...recently I started getting this exact same error with Blesta!
It sends the request, the services are provisioned on virtualmin, however, Blesta times out and does not record the services are provisioned on only new accounts where thus error occurs. On all existing accounts prior to thus error, things seem to work fine.
Blesta developers reckon it's something to do with webserver restarting during the procedure and that it should not happen if their software is on a separate machine. Perhaps this is also relevant here?
Mine also won't even work if run from an automated cron instead of instant manual activation.
What confuses me, mine was working for about 1 month and all of a sudden, this happens to me too.
Logs don't appear to show anything.
Hi @adamjedgar,
I have a ton of experience troubleshooting this problem and will try and help you here or we can take this offline if you wish.
Here are a couple of things you can use in the isolation process:
First of all, yes possibly making the billing system separate from what is being provisioned will help and should be considered. It's a better practice as well. But understand there is a difference between "restart Apache" and "reload Apache". In theory when you add a new domain you are reloading which is pretty quick and shouldn't be disruptive, versus restart which will kill all running websites in it's tracks and restarts them. I know long ago Plesk and WHMCS had issues if you provisioned on the same server so isolating this same server situation will be a good idea.
Second of all, do consider performance. This is really easy if you have SSH access. Just run 'top' and provision. The full command to test is something like:
`<?php $result = shell_exec("wget -O - --quiet --http-user=root --http-passwd=password 'https://server.domain.com:10000/virtual-server/remote.cgi?program=list-d...'");
echo $result;`
Note in the above
multiline
which is the extended listing command that takes longer than without it. From my tests even on a small 2vCPU server with not even 10 sites this command can take anywhere from 30 seconds to 2 minutes. That is a very long time and lots of billing systems will run out of steam. If your billing system is using WGET you might even run into a 1 minute timeout which seems to be a default on some systems and has to be changed in the code, not on the server. WHMCS was so kind to send me a new Virtualmin module with a three minute timeout which took care of that part of my problem.What is the error message? Take a timer and see if it's a minute or an exact interval every time. If it's an exact interval, say 30 seconds, or 60 seconds, or whatever, it's most likely a timeout.
Perhaps your server got busier in the last month? Perhaps many more mailboxes? Spam and virus filtering can bog your server down badly but is easily identifiable with top.
Yep. That also threw me for hours / days. I think the fact of the matter is that on the server side "nothing is wrong". It's just slow. So the billing system times out. Only if something actually goes wrong on the server side an error will be reported. For now, it's just a slowwww operation.
Top is your friend, my friend.
warm regards, Eugene
https://vander.host
Submitted by adamjedgar on Mon, 05/13/2019 - 06:09 Pro Licensee Comment #25
Hi Eugene, I have begun moving across to WHMCS because of this error...my feeling is that having a much larger user base will see a resolution with WHMCS in preference to Blesta.
Having said that, there are no error logs...that is the problem. I actually gave the Blesta developers full access to my Virtualmin system and they are convinced the problem is coming from Webmin itself.
Essentially what i believe they are saying is this...Blesta is using the Webmin API in order to provision services. The process is quite fast usually only takes about 12 -15 seconds to complete if I am just creating an empty Virtual server (so timeout shouldnt be the issue...I increased that in php and it made no difference anyway)
The issue as far as i can tell is that Blesta sends the request via Webmin API, Webmin correctly provisions services (ie a virtual server), however at this point webmin is not then sending that information back to Blesta. This is where the problem lies and according to Blesta devs, it is because webmin is reloading at the point of povisioning of services thus closing off communication to Blesta until it reloads...of course this loss of connection ends the transaction and Blesta does not update.
Blesta devs are convinced that if one is running on a separate system, this would not be a problem...I cannot remember why exactly. The question i have is, initially this was working for about 3 weeks and then all of a sudden it stopped working???
Submitted by hescominsoon on Wed, 06/05/2019 - 06:57 Pro Licensee Comment #26
webmin/virt does a soft restart anytime you do anything. It has been that way forever. Short of a virt total rewrite the blesta devs need their program to try an auto-reconnect several times until virt awakens and then checks the status.
What is that? Same Apache reload (configuration) versus Apache restart (restart services from scratch)?
Connect to do what? Provision? Update usage statistics? Servers typically aren't "connected". The client send a message to the Virtualmin server, the Virtualmin server responds, and the client acts. It's more synchronous versus real-time connection oriented.
Sorry about all the questions, but which status? Is the current website active status?
Ideally the Blesta devs comes on the forum and gives their analysis. Although this post is about WHMCS so not too sure if they'll feel comfortable posting here.
But to show you what I meant by client->server->response->client:
I attached to any server.
I run the script shell_exec("wget...program=list-domains&multiline");
This call a CGI program on the Virtualmin host to obtain lots of detail about every single domain.
The server sends that data back to the billing system (the client).
The billing system interprets the data.
I just did it:
[root@ns2 ~]# date Wed Jun 5 16:15:40 SAST 2019 [root@ns2 ~]# php list-domains-multiple.php aa-dns-parking ID: 15502379642252 File: /etc/webmin/virtual-server/domains/15502379642252 ... Exit status: 0 [root@ns2 ~]# date Wed Jun 5 16:15:58 SAST 2019
The entire process took about 18 seconds to analyse 20 domains.
No actual "connection" was made, but rather a back and forth. And it works perfectly every time.
Submitted by adamjedgar on Wed, 06/05/2019 - 15:32 Pro Licensee Comment #28
Submitted by eugenevdm.host on Sun, 03/31/2019 - 18:33 Comment #20
After much testing I have concluded this happens because Virtualmin is very slow when returning results on a small server for list-domains and multiline. I have documented my findings on the WHMCS Feature Requests website https://requests.whmcs.com/topic/virtualmin-provisioning-module-curl-req... but it would be useful if someone from Virtualmin can tell us why doing the remote CGI commands list-domains and multiline often takes more than a minute to complete.
Now that is interesting, because isnt the whmcs module hard coded to stop trying well before that? (WHMCS is hard coded i thought for something about 60 seconds) Blesta devs told me their software should provision in less than 20 usually.
I dont seem to have any problems running scripts from the command shell...its when they are run from outside the shell using another application that trouble starts...ie Blesta and WHMCS (to differing degrees)
Blesta provisions but has no idea that process has complete (because for some reason the response is not being sent to it, which the Blesta devs say is because webmin is either reloading or restarting apache before the response is actually sent thus Blesta loses connection (usually if on same server apparently, although for me that made no difference)
WHMCS requests just seem to dissappear into thin air, although WHMCS thinks something has actually happened (because most of the time no there are no errors sent to WHMCS...and this is why im suspicious of some kind of firewall/security issue)
i have had this working before on Virtualmin (last year) on google cloud compute. Now I am using a Vultr server, i cant seem to get it working. I cant imagine that has anything to do with this (particularly since Digital Ocean VPS is also experiencing similar problem), but i suppose that is at least worth mentioning.
BTW, I notice that the return message you reported just now is using the "Package ID" and not the virtualmin "Plan Name". That is also another quirk i noticed last week...it works better when using the Virtualmin Plan "ID" in WHMCS.
Submitted by adamjedgar on Wed, 06/05/2019 - 18:38 Pro Licensee Comment #29
OK here is a partial solution that is now working for me on the server i was having problems with.
Check all of your server details in whmcs are correct whmcs>setup>product/services>server
i did not use any names for anything...all virtualmin templates and plan details are inputted into WHMCS>setup>product&services/add new product using the Virtualmin ID numbers (open each Virtualmin template and also each Virtualmin plan in a web browser and you will see the ID in URL...copy this into whmcs).
If i used Virtualmin names in WHMCS it failed for me but not if i use Virtualmin ID's for both templates and packages/account plans
I am not sure if this actually has anything to do with WHMCS provisioning, this is just what i did
Ok now for the Quirks in WHMCS...
For example,
In WHMCS admin area under clients>view/search clients>client name>
there are buttons directly below the first section called create/suspend/unsuspend/terminate/change package etc...
If one clicks on one of these buttons, it appears that whmcs is provisioning...i left it for 10 minutes and that little round wheel kept right on spinning and spinning. What i did not realise is that if i then scroll down to bottom of page and then click save...voila it provisions.
Now for the strange bit...if you look in my attached image link, you will notice that the service has provisioned, however in WHMCS it is still set to "Pending"!
I have setup the virtualmin module such that no product is automatically provisioned unless i accept the order from administration area in whmcs. So in light of this setting, I would have thought that WHMCS should not provision anything in virtualmin unless i accept the order from WHMCS>Orders>List pending Orders...and yet, we can clearly see goannadomains.com.au in the virtual server list!
Anyway, I have now made significant progress on my issue and essentially it is now provisioning...what worries me is that will this work if click create, dont walk away for 10 minutes and then come back and scroll down and hit save in WHMCS?
Clearly, if the administrator clicks on any of these buttons, it is over riding the pending setting and intiating the provisioning process immediately. I imagine that one should instead simply scroll down and click save?
Quirks...blasted Quirks!!!
here is Image link (registration date in image is the day i first created user in whmcs...it is not day of provisioning)
https://drive.google.com/open?id=1DhK1lsA4eDLD8g4W06V9KTDpyc3IHuhl
Submitted by adamjedgar on Wed, 06/05/2019 - 18:47 Pro Licensee Comment #30
Also, here is a later image showing a little more information...
https://drive.google.com/file/d/1S3beOdX740xabENSk5qKpjuL3wQOjf2X/view?u...
Submitted by redrum2 on Thu, 06/06/2019 - 00:39 Comment #31
So i was also having this issue, for me the ssl is the problem.. with ssl disabled every where, i and auto setup virtualmin servers from whcms automatically... As soon as i tried enabling ssl everywhere it stopped working.
I couldnt tell you exactly where to disable the options because there are a few too many. Disabled all the ssl temporary solved my problem.
@adamjedgar you are describing exactly our symptoms. However, the problems went away. Perhaps contact Nathan L. at WHMCS, quote myself Eugene van der Merwe Ticket ID: OCQ-028907. In this ticket he says:
"I have gone ahead and have put together a version of virtualmin.php with the cURL timeout set to 180 for you, and have had it encoded by our development team."
I know, the issue isn't a long wait time, but rather an indefinite wait time. But nevertheless, after having received this module my provisioning problems went away. Knowing how complex software is, it could also be that they added a new flag to ignore SSL errors meaning that possible @redrum2 synopsis correlates. Just give it a bash, and let us know. I can see there's a lot of folk who would like to have the WHMCS module working perfectly and all this chatter really is helping.
Submitted by adamjedgar on Thu, 06/06/2019 - 07:00 Pro Licensee Comment #33
@redrum....yes i had wondered about SSL. I was starting to believe that my problems...if SSL related might have been because of the way in which i assigned the webserver hostname...
domain = adam host = server1
so host.fqdn = server1.adam.com
WHMCS = sub server and is a subdomain = whmcs.server1.adam.com
I got so muddled up with the above configuration that i think i ended up assigning ssl certificates to everything individually. I was convinced that this is why mine might not have been working. However, whmcs tech guys were not worried by that.
@eugenevdm.host... yes the timeout did seem like a likely candidate, especially once i found out that whmcs timeout is hardcoded into the module. To me that seems like a crazy idea because not all systems are built with V8 engines in them.
WHMCS were quite certain that perhaps at least part of my problem was the small server i am running. The issue i have with this, if most servers are loaded to pretty much maximum, how does a small server loaded to the same percentage as a large server make any difference to the equation? (doesnt make sense to my logic...ie a small car with a small engine can accelerate just as quickly as a large car with a big engine).
Also, if my server was not up to the task, why on earth are both VestaCP and Centos Web panel systems provisioning perfectly from my Virtualmin WHMCS installation? (they are remote, virtualmin is localhost, surely the localhost should be faster and less problematic?)
What i think is needed is someone to sit down and write a decent wiki on how to setup Virtualmin properly with WHMCS. It is definately not as simple as for CWP or VestaCP. I have no problem with that, look at what one can do in Virtualmin compared with other Control Panels...its highly complex but incredibly capable, and yet surprisingly stable for all its complexity.
Submitted by hescominsoon on Thu, 06/06/2019 - 13:00 Pro Licensee Comment #34
WHMCS is notorious for saying it's the other end...that's the biggest reason i quit using them.
Submitted by hescominsoon on Thu, 06/06/2019 - 13:03 Pro Licensee Comment #35
I had issues with WHMCS as well and it turns out that WHMCS likes to stomp everywhere on the system...even when it shouldn't. This is back from 2016 but i wonder if WHMCS is still trying to go rogue and this is why when you run WHMCS on the same server it's barfing. I would try running WHMCS on it's own server and then have it reach over to the virt server: https://etc-md.com/2016/06/09/etc-maryland-enhanced-wordpress-hosting-pr...
@adamjedgar
Actually it's already been done, by WHMCS themselves: https://docs.whmcs.com/Virtualmin_Pro
Fact is possibly WHMCS have neglected the Virtualmin plugin. The documentation above for example refers to Virtualmin Pro but we all know that Virtualmin GPL also works with WHMCS.
Virtualmin and WHMCS is dead easy. The complexity comes in when something goes wrong. Then there are multiple parties involved, e.g. Virtualmin, WHMCS, your OS, your firewall, etc. Then stuff becomes complex. But believe you, I've installed and worked with Plesk, Virtualmin, cPanel, and ISPConfig. I worked with all of those for years with WHMCS. Virtualmin is by far the best because it's stock. Stuff just works, until, of course, they don't. But then you can just google as Virtualmin is as close to the real with without proprietary crap than you will ever get.
Submitted by redrum2 on Sun, 06/09/2019 - 09:46 Comment #37
I can get whmcs and virtualmin to work flawlessly without ssl enabled anywhere. This is 100% SSL related and it caused because once it is enabled the domain creations loses connections and whmcs restarts the domain-creation script when it reconnects.
Submitted by adamjedgar on Sun, 06/09/2019 - 18:06 Pro Licensee Comment #38
Recently I did this:
Go to Vultr.com Register an account and setup a brand new VPS instance Install Virtualmin gpl from Virtualmin.com
Then install WHMCS using Virtualmin built in Script
So everything is defaults...no changes except hostname is set for the VPS and firewall ports at Vultr are opened. In virtualmin I did not touch the firewall other than use the default virtualmin port 10000 (which is also coded into whmcs in 2 places domain.com:10000 and ipaddress:10000 (WHMCS have told me directly that if done this way WHMCS uses port 10000 to communicate with Virtualmin.
It doesn't work! Provisioning packages in WHMCS failedl
This is because WHMCS has a few little quirks in it that regular users would automatically fix without even realising it. New users on the other hand, who are not familiar with the VM+WHMCS combination will have problems (as is already seen regularly on this forum)
I believe one reason why CWP works better is because of the API in CWP vs Virtualmin. I am not saying the CWP API is better...I suspect Virtualmin is the more secure of the two systems...and this is quite likely why things are not quite as easy to setup in Virtualmin.
I would prefer to have trouble because a system is too secure rather than the other way around. Many here would be aware of what happened over at Vestacp last year when one of their own repo servers got hacked and then everyone who downloaded the corrupted update got a virus and was then hacked by bots...the result was the big service providers taking down compromised (and also non compromised) vestacp servers for upwards of 1 month before it got brought under control! (apparently many hundreds, indeed even thousands of systems, were compromised)
Second, I found using template and package names is problematic...it works much better with ID numbers that are only found when looking at template and plan names in the browser (I copy these into whmcs). This is not in the WHMCS documentation for Virtualmin module as far as I recall. However it is in the Blesta module (hence my reason for even trying this fix)
As I have said on other posts, one of the big flaws in WHMCS is ioncube. We cant look into the code for problem solving anything (its all encrypted). If one cant see the code, one cant know the philosophy behind the application. One is then reliant on logs.
Trouble is, in my case (and a number of others I have also seen recently) non communication errors are most often not actually recording anything in the logs! WHMCS say this is impossible...I don't care about impossible.
In any case, my problem is resolved by inputting ID numbers instead of names into WHMCS for Virtualmin Templates and Plans
Actually now that I think of it..the mention of SSL makes me wonder, could it also be a permalink issue? (I hadn't thought of this...is that even relevant?)
Submitted by sbb2019 on Fri, 10/23/2020 - 07:46 Comment #39
I just had this problem today and the issue I'm facing is that Virtualmin is taking forever to chroot jail new domain Unix users. So once I set that to no in the template, whmcs now provisions very fast.
Also did a check on VPS's SSD performance as it could be the main culprit. dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync = 180 MB/s Very slow speed, the provider is probably using a f** tape drive LOL.