Hi,
The following features would make the Cluster Software Packages much more useful:
1- remove packages GUI option, that accepts multiple package names 2- update packages GUI option, that accepts multiple package names and does NOT install a package if a previous version was not present 3- install packages GUI option (already exists) modified to accept multiple package names
Note: feature 2 makes it possible to push "yum update" commands via Virtualmin over a superset of critical packages across a group of servers, without ever installing anything as a colateral effect. That superset can be seen as the most general "attack surface" across the group of servers even though not all servers have the complete surface actually present.
Whether this is a feature from Virtualmin GPL or Pro is another story, but I think this is a way to add a lot of value to the cluster feature, since update management is one of the most important and time consuming tasks.
Comments
Submitted by JamieCameron on Mon, 07/23/2012 - 00:05 Comment #1
You might want to look into our "Cloudmin" product - it can do a lot of this already, such as managing package updates across multiple machines. If you are only managing real systems and not VMs, the "Physical Systems" version may be what you need : http://www.virtualmin.com/catalog/cloudmin/physical
Submitted by ghomem on Mon, 07/23/2012 - 16:57 Comment #2
Cloudmin for physical systems seems to be very, very interesting.
How does it compare to the Webmin Clustering features? Do you have a detailed featureset somewhere? Is there a demo?
I'm interested in the features I described above plus something like "Cluster Shell Commands"
Submitted by JamieCameron on Mon, 07/23/2012 - 23:59 Comment #3
Check out the docs at http://www.virtualmin.com/documentation/cloudmin
We don't have a demo unfortunately. Cloudmin includes a "cluster shell commands" like feature..
Submitted by ghomem on Sun, 11/24/2013 - 12:58 Comment #4
Since we are now customers of Cloudmin as well I decided to have another look at this. The run command feature from cloudmin seems to work well.
After selecting a set of hosts the "update packages" feature brings us to a prompt which says "Install packages". That means it will install the packages in case they are not installed isn't it?
There should be an "install packages" feature that installs them if they are not installed and does nothing if they are already installed.
The "update packages" feature should do update the packages if they are installed and do nothing if they are NOT installed.
That is, the update feature should never install anything that was not previously installed and the install feature should never update anything that was already installed.
This describes a method for not breaking what is working.
Could you comment?
Submitted by JamieCameron on Sun, 11/24/2013 - 14:10 Comment #5
Currently, you can either select to update packages on multiple systems (if they are installed), or install new packages.
The latter can be done by selecting one or more systems from the "Cloudmin Managed Systems" page, clicking "Update Packages" and scrolling down to the bottom of the page. There you'll find a form for entering package names to install on the chosen systems. The only catch is that this will upgrade those packages if they are already installed, as it uses a command like
yum install
orapt-get install
which works in the same way.Submitted by ghomem on Sun, 11/24/2013 - 17:46 Comment #6
Jamie,
There is no former and latter. There is only one option on "Cloudmin Managed Systems" which is "Update packages" and shows an "Install packages" box, that receives a list of packages.
What does it do? Does it call apt-get install and yum install depending on the distribution?
Anyway, I think you are mistaken regarding the correct use of apt-get and yum.
Please read this carefully:
1)
To install packages you should do:
for p in $LIST; do check_installed $p || install_package $p done
The install_package function will then call "apt-get install" or "yum install" depending on the distribution.
Why is this necessary? Because an "install" procedure from an automation product should NEVER update a package. That is the function of an "update" procedure. We don't want to break working systems with an accidental update.
2)
To update packages you should
for p in $LIST; do check_installed $p && update_package $p done
The update_package function can call "apt-get install" and "yum install" / "yum update". If you only worked with Centos/RHEL the check_installed function wouldn't be necessary since the "yum update" command does NOT install packages - it only updates packages if they are present.
Why is this needed? Because an update procedure on an automation product should NEVER install anything that wasn't installed before.
No you will tell me that apt and yum should have thought of it themselves. I will say that it is the role of a sophisticated automation product to have this kind of business logic working so that customers can deal with a reduced probability of breaking production systems.
Before we were using Cloudmin we tried to make you guys see this on this ticket
http://virtualmin.com/node/22836#comment-103556
But you told me to use Cloudmin :-) Now we are back to bugging you.
Can you comments?
Submitted by JamieCameron on Mon, 11/25/2013 - 04:31 Comment #7
That's a good point - in the next Cloudmin release, it won't update existing packages when you use that new package install form.
So do you have systems that have packages that need updating, but are not shown in Cloudmin as having updates available?
Submitted by ghomem on Mon, 11/25/2013 - 05:16 Comment #8
Jamie,
I'm am a bit confused. Please look at the screenshot.
All the systems have packages that need updates.
If I press "Update packages" there is only the "Install packages" option available.
Should I see another one?
If the "Update packages" button leads to two different options then it should be called "Manage packages" or something more general.
Submitted by JamieCameron on Mon, 11/25/2013 - 13:59 Comment #9
Currently, Cloudmin can only fetch available package updates from systems with Webmin installed.
Submitted by ghomem on Mon, 11/25/2013 - 16:24 Comment #10
Understood.
If the server is not webmin enabled you could as well make the update text box available, just as the install box is available. That's where my comment about the update function not installing anything, only updating, comes from.
What I've said about the button label still makes sense. Saying "Update packages" is quite misleading.
Submitted by JamieCameron on Tue, 11/26/2013 - 07:19 Comment #11
Good idea, I'll do that in the next release.
Submitted by ghomem on Tue, 11/26/2013 - 08:32 Comment #12
Thanks!! Please bear with us, our suggestions will help you making Cloudmin better.
Jamie, please note that we are waiting for a PDF from you in order to submit Cloudmin to a public tender:
http://virtualmin.com/node/28268
This is urgent.
Submitted by ghomem on Wed, 01/08/2014 - 14:46 Comment #13
Hi,
What is the status of the suggestion on #11?
Another question: are you thinking of implementing groups on Cloudmin, so that managing thousands of machines is possible? (ex: customer one prodution, customer one pre-production, customer two production, customer two pre-predoctiyon,...)
One more, that connects to the previous one: we will need to manage Suse machines with Cloudmin and since the packages are not the same as of RHEL/CentOS probably it will be dangerous to do this without grouping.
Can you comment?
(We submitted Cloudmin to the tender, after we got your PDF - waiting for the results - see comment #12)
Submitted by JamieCameron on Wed, 01/08/2014 - 16:43 Comment #14
A new version of Cloudmin hasn't been released yet with the fix, but will be hopefully next week.
Regarding groups, these are already supported in Cloudmin - on the Edit Virtual System page you can put each system into one or more groups.
Submitted by ghomem on Wed, 01/08/2014 - 19:19 Comment #15
Regarding groups:
I see now that one can create a group, just as you suggested. We can have the group info appear on the cloudmin list and sort by group clicking on the header. But then one can't select all the machines of a certain group, right? If the group has, say, 30 machines do we need to click 30 times? I tried clicking on the first plus shift+click and it didn't select them all.
Submitted by ghomem on Wed, 01/08/2014 - 19:33 Comment #16
Still in this context:
I tried to register a generic centos host and a webmin enabled (virtualmin pro) centos host in the Webmin Servers Index from a Cloudmin Pro installation. I expected them to appear in the Cloudmin hosts list but none appeared.
Can Cloudmin Pro also manage external servers in terms of command execution and packages?
Submitted by ghomem on Wed, 01/08/2014 - 19:37 Comment #17
Duh... it is not on Webmin Servers Index...but on Add Physical System. Please ignore this question.
Submitted by JamieCameron on Wed, 01/08/2014 - 23:49 Comment #18
On the "List Managed Systems" page, you should be able to use the search form to show VMs in a group, click "Select all", and then perform some operation on them.
Submitted by ghomem on Thu, 01/09/2014 - 05:39 Comment #19
I don't find a search form there. Please see the screenshot on comment #8 - there is no search form.
There is a search form on the left frame just below the logout button but that doesn't search groups, searches cloudmin modules.
Submitted by JamieCameron on Thu, 01/09/2014 - 16:09 Comment #20
The search from only appears if you have more than 20 systems (by default).
Submitted by ghomem on Wed, 01/29/2014 - 07:03 Comment #21
I tried to add a Physical system that runs Virtualmin Pro to an existing Cloudmin Pro install and it fails saying ports 10001 to 10010 are firewalled. In fact they are: the server only exposes port 10000.
When I add that same server via Webmin Servers Index it works using port 10000. I can login to the servers Webmin interface without problems.
Can you help?
Questions:
Note: all our servers have -j DROP by default for everything in INPUT, OUTPUT and FORWARD targets, we open ports on a necessity basis.
Another question:
When I finally manage to connect, from Cloudmin Pro, to remote physical systems running Webmin or Virtualmin I will want to issue package updates for groups of systems. Suppose I select a group of N systems to update packages for. What will be shown on the list?
1- packages that need update, that are common to all the N systems?
or
2- packages that need update that are installad in at least on system?
Will you ensure, also in this case, that packages that were not installed before won't be installed on any of the systems?
Cheers Gustavo
Submitted by andreychek on Wed, 01/29/2014 - 09:32 Comment #22
Jamie may need to comment regarding your package questions.
In regards to firewalling though -- communication for certain actions when accessing remote Webmin systems does require the use of Webmin RPC, which uses ports 10001 - 10010.
My suggestion would be to start by opening that -- that should be all you need, though if it doesn't work we can take a deeper look at your output rules.
Submitted by ghomem on Wed, 01/29/2014 - 10:27 Comment #23
Andrey,
Thanks we will do that.
Can you tell me if SLES machines, runing Webmin can be managed using Cloudmin Pro (on RHEL / Centos) in termos of package updates? SLES uses RPM and zypper.
Submitted by ghomem on Wed, 01/29/2014 - 12:03 Comment #24
I just added a Virtualmin Pro Centos system after I opened ports 10001 - 10000.
But once I add it its status is "Alive" instead of Webmin.
I can't run commands or upload files: ERROR: SSH logins are not available
And the graphical list of packages does not appear.
Do I need SSH login? But I don't have SSH root logins enabled. What is the Webmin connection on port 10000 for, then?
Submitted by ghomem on Wed, 01/29/2014 - 12:21 Comment #25
After adding SSH keys to login as root the status is "Virtualmin" instead of "Alive" and the packages GUI already appears.
Another thing:
If I try to register the system using only SSH the status instead of being SSH is "Webmin down" (see screenshot). Why does it say "Webmin down" if I didn't request the webmin connection?
Submitted by JamieCameron on Wed, 01/29/2014 - 23:02 Comment #26
It will say "Webmin down" if it tried to connect to Webmin but could not.
You can prevent Cloudmin from trying to login at all by going to Change Passwords -> Change login used by Cloudmin and selecting "Do not attempt Webmin login"
Submitted by ghomem on Thu, 01/30/2014 - 19:40 Comment #27
Jamie,
That solves it, but there is no option to not connect to webmin when we add the machine. Please look at the screenshot - I chose "Webmin not installed yet" and got status "Webmin down". Which one should I choose?
Can you take a quick look at the last part of comment 21, regarding package updates using the GUI?
Thanks for your support.
Submitted by JamieCameron on Fri, 01/31/2014 - 12:00 Comment #28
Thanks for pointing out the issue with the form for adding a system - I will fix that in the next release.
Regarding package updates, if you select multiple systems it will show packages that need updating on at least one of those. And it will never install new packages as part of the update process (unless forced to by YUM or APT).
Submitted by ghomem on Fri, 01/31/2014 - 19:51 Comment #29
Jamie,
Thanks for the answer. I will test the next release.
Regarding the package update - the behaviour you describe is good. I worried about that you mean by "forced by YUM or APT".
If package A is installed on System-1 and needs update and package B is installed on System-2 and needs update, the list will show A and B.
Then if the operator selects A and B what happens next?
If you run
apt-get install -y A B
on both systems apt will install package B on system A and package A on system B. Is this what you mean by force by APT? Or is the logic I mentioned on comment 6 already in place to prevent this?
I asked this in the context of the "install packages box" but it applies to the graphical view as well.
YUM update doesn't have this problem - the problem is that APT doesn't have an "update" command (its "update" commant updates the repo information).
Submitted by JamieCameron on Fri, 01/31/2014 - 18:48 Comment #30
So in the example you gave, the packages will only be upgraded on each system - never installed. Cloudmin doesn't just run "apt-get install A B" on each system.
By "forced by YUM or APT", what I meant was if a new version of package A adds a dependency on C, that will get installed by YUM or APT.
Submitted by ghomem on Fri, 01/31/2014 - 19:54 Comment #31
Jamie,
Those are excellent news. I feel quite confident to use expand our use of Cloudmin to package management from now on.
Regards Gustavo
Submitted by ghomem on Wed, 02/26/2014 - 15:06 Comment #32
An FYI on the "webmin down" issue:
On the Cloudmin host, in case the OUTPUT policy is DROP:
iptables -A OUTPUT -p tcp -m tcp --dport 10000:10010 -d MANAGEDNETWORK -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
On the guest or physical machines:
iptables -A INPUT -p tcp -m tcp --dport 10000 -j ACCEPT # Virtualmin
iptables -A INPUT -p tcp -m tcp -s CLOUDMINHOST --dport 10000:10010 -j ACCEPT # Cloudmin
It's a bit confusing that the web interface needs only port 10000 whereas the remote management protocol requires ports 10000:10010. Anyway, this seems to work - feel free to add to the docs.
Submitted by ghomem on Wed, 07/16/2014 - 14:41 Comment #33
Moving this issue to Cloudmin, since we are now Cloudmin customers.
Submitted by JamieCameron on Wed, 07/16/2014 - 21:33 Comment #34
Are there still any outstanding problems here?
Submitted by ghomem on Thu, 07/17/2014 - 04:21 Comment #35
Please see comments 10, 11 e 28 and confirm if fixes were indeed made available. thanks
Submitted by JamieCameron on Thu, 07/17/2014 - 14:45 Comment #36
Yes, those fixes are done.
Submitted by ghomem on Fri, 07/18/2014 - 06:15 Comment #37
Thanks Jamie. I will test this and come back with feedback.
Submitted by ghomem on Fri, 07/18/2014 - 06:31 Comment #38
Oustanding problems on fully updated Cloudmin Pro (7.5)
"None of the selected systems have Webmin or Virtualmin installed"
This is a regression as previously at least the install option was available
Please see comments 8-11