Submitted by pcfreak30 on Sun, 02/23/2014 - 19:44
Is there not a good way to put some of the processing for API's like create-system in the background? It seems like it does everything upfront which causes an API call to take a very long time, especially if for instance an image needs to be copied over.
This results in WHMCS handing for a very long time when "Accept Order" is pressed due to it waiting on an API response.
I just think some of this should be looked at. Solusvm from experience seems to use some closed-source c daemon program that handles these tasks and uses an admin log on the panel if there are setup errors.
Thanks.
Status:
Closed (fixed)
Comments
Submitted by JamieCameron on Sun, 02/23/2014 - 20:44 Comment #1
I suppose we could have asynchronous APIs that have to be polled for completion, but that would be harder for the average caller to use. On the other handle, WHMCS could have made the call asynchronous on it's size so that the provisioning UI doesn't stall.
Jamie,
Just a thought on the topic...
Why not add a "PayPal IPN-like" method where the call is sent to the background and a "return URL" is specified where Virtualmin would send the response to when complete...
Perhaps to keep things flexible have a flag which determines whether the API is called in the foreground for sends the request to a background task.
-Peter
Submitted by pcfreak30 on Sun, 02/23/2014 - 22:26 Comment #3
Not sure how WHMCS would do that. I had to write a custom provision module and use php file_get_contents with a custom timeout setting for API calls as CURL was giving me fits.
Issue there is whmcs needs to know if it was successful or not, but thats only known after a call is completed via the json status key. create-system for example can hang for 10 minutes depending on the actions that are needed to be taken.
pcfreak30,
Sorry, I don't use WHMCS as I have built my own custom scripts for interacting with the API. Sorry if my suggestion won't work for needs, it was just something I thought up at the time. Cheers!
-Peter
Submitted by pcfreak30 on Sun, 02/23/2014 - 23:22 Comment #5
My comments were not towards you tpnsolutions, but to JamieCameron. Seems you can't nest comments in the tracker.
Hmm, is this nested???
*** apparently not ***
-Peter
Submitted by pcfreak30 on Mon, 02/24/2014 - 07:56 Comment #7
I just used a stopwatch to count how long an API call took. It was around 7 minutes that WHMCS hanged on creation. If a timeout occurs somehow it will not be good for automation from a payment callback or such.
Submitted by JamieCameron on Mon, 02/24/2014 - 19:13 Comment #8
I think this could be implemented without too much work in Cloudmin .. I'll look into it.
That said, WHMCS would have to be changed to actually make use of the async API, and change its UI to report success or failure in an appropriate way.
Submitted by pcfreak30 on Mon, 02/24/2014 - 19:19 Comment #9
WHMCS is closed source, so a queue could be implemented to work around its limitations with some sort of polling. For now though I can't really proceed on integration until I can get a reliable response time on the API. In some instances I have oddly gotten hangs resulting in php memory maxing.
Submitted by pcfreak30 on Thu, 02/27/2014 - 18:00 Comment #10
So here is how I see SolusVM working. When the create api command is ran, it does a few preliminary checks. Thinks like IP space, node space, if the node is up that would be direct errors.
From there it hands the processing off to a background program and returns successful. Instead of hanging the api for the whole creation, its backgrounded and issues go to a system log.
the api call doesn't take more then 10 seconds, while cloudmins method can hang for 7-10 min.
Submitted by JamieCameron on Thu, 02/27/2014 - 19:49 Comment #11
That would be another option - the API call could return success as soon as basic checks are done, but complete the process in the background.
Submitted by pcfreak30 on Thu, 02/27/2014 - 19:53 Comment #12
That would be best. I think it should be a switch to enable that behavior since the same script would be ran on CLI. What you can do is just append the switch option in remote.cgi for any api call and refactor commands to accept it or just ignore it.
Submitted by JamieCameron on Sat, 03/01/2014 - 16:25 Comment #13
Ok, the next Cloudmin release will support a
backgroundflag that can be used to put long-running operations like creating and cloning a system into the background (once initial checks are complete).Submitted by pcfreak30 on Sat, 03/01/2014 - 16:29 Comment #14
Would it be possible to get a patch for that when implemented?
Submitted by JamieCameron on Sat, 03/01/2014 - 17:12 Comment #15
Sure .. but you'd also need changes in WHMCS to use it, assuming that's where you are calling the API from.
Submitted by pcfreak30 on Sat, 03/01/2014 - 17:15 Comment #16
I know. I was writing a custom module anyways.
Submitted by JamieCameron on Sat, 03/01/2014 - 17:56 Comment #17
Ok - so other than VM creation, are there any other long-running tasks you'd like the ability to background? The API also supports this for cloning and moving VMs.
Submitted by pcfreak30 on Sat, 03/01/2014 - 18:17 Comment #18
WHMCS does create, terminate, change package/upgrade/downgrade (we enter manual settings for each product and dont use CM plans), change password, start up, shutdown, reboot.
For all of those it waits on the API. So if any of them have a chance at running for a long period they should support being backgrounded.
Also have you implemented the api by IP yet as I assume the changes will modify the api .pl files too of which we have had to modify ourselves for that. Trying to present a clash of modifications.
Submitted by JamieCameron on Sun, 03/02/2014 - 14:57 Comment #19
Ok, most of those are pretty quick - assuming the VM startup hasn't hung, they should take less than a minute. Only operations that involve a lot of disk IO are slow.
I have implemented API lookup by IP already, so I can send you a fix that includes that feature. Remind me which Linux distro your Cloudmin master is on?
Submitted by pcfreak30 on Sun, 03/02/2014 - 16:29 Comment #20
CentOS 6.
Submitted by JamieCameron on Sun, 03/02/2014 - 17:32 Comment #21
Ok, I have emailed you an updated Cloudmin RPM package.
Submitted by pcfreak30 on Sun, 03/02/2014 - 18:38 Comment #22
Also does the rpm include the past changes I have brought up? Is it a copy off git or a mix of things?
Submitted by JamieCameron on Sun, 03/02/2014 - 18:51 Comment #23
It includes all the changes for tickets that I have marked as fixed up to this point.
Submitted by pcfreak30 on Sun, 03/02/2014 - 19:30 Comment #24
Could you add the background to the delete-system command? For some reason the call always hangs over remote API even though it completes.
Thanks.
Submitted by JamieCameron on Sun, 03/02/2014 - 23:57 Comment #25
Deletion should be pretty quick already - how long does it take if you delete a VM from the command line?
Submitted by pcfreak30 on Mon, 03/03/2014 - 10:50 Comment #26
It doesn't take long, its just the web call hangs indefinitely and I have no idea why. Thats the only command that consistency acts like that.
Submitted by JamieCameron on Mon, 03/03/2014 - 13:04 Comment #27
Does the VM being deleted have a system owner account associated with it?
Submitted by pcfreak30 on Mon, 03/03/2014 - 15:02 Comment #28
Yes it does but it is the only VM associated to the user and the user is deleted in a second API call if no other VM's exist with it.
Submitted by JamieCameron on Mon, 03/03/2014 - 20:58 Comment #29
The only way I could see the API command hanging is if Webmin gets restarted as part of the VM deletion process (which you should be able to see in /var/webmin/miniserv.error).
Submitted by Issues on Mon, 03/17/2014 - 22:01 Comment #30
Automatically closed -- issue fixed for 2 weeks with no activity.