Cluster Software Packages whishlist

Hi,

The following features would make the Cluster Software Packages much more useful:

1- remove packages GUI option, that accepts multiple package names 2- update packages GUI option, that accepts multiple package names and does NOT install a package if a previous version was not present 3- install packages GUI option (already exists) modified to accept multiple package names

Note: feature 2 makes it possible to push "yum update" commands via Virtualmin over a superset of critical packages across a group of servers, without ever installing anything as a colateral effect. That superset can be seen as the most general "attack surface" across the group of servers even though not all servers have the complete surface actually present.

Whether this is a feature from Virtualmin GPL or Pro is another story, but I think this is a way to add a lot of value to the cluster feature, since update management is one of the most important and time consuming tasks.

Status: 
Active

Comments

You might want to look into our "Cloudmin" product - it can do a lot of this already, such as managing package updates across multiple machines. If you are only managing real systems and not VMs, the "Physical Systems" version may be what you need : http://www.virtualmin.com/catalog/cloudmin/physical

Cloudmin for physical systems seems to be very, very interesting.

How does it compare to the Webmin Clustering features? Do you have a detailed featureset somewhere? Is there a demo?

I'm interested in the features I described above plus something like "Cluster Shell Commands"

Since we are now customers of Cloudmin as well I decided to have another look at this. The run command feature from cloudmin seems to work well.

After selecting a set of hosts the "update packages" feature brings us to a prompt which says "Install packages". That means it will install the packages in case they are not installed isn't it?

There should be an "install packages" feature that installs them if they are not installed and does nothing if they are already installed.

The "update packages" feature should do update the packages if they are installed and do nothing if they are NOT installed.

That is, the update feature should never install anything that was not previously installed and the install feature should never update anything that was already installed.

This describes a method for not breaking what is working.

Could you comment?

Currently, you can either select to update packages on multiple systems (if they are installed), or install new packages.

The latter can be done by selecting one or more systems from the "Cloudmin Managed Systems" page, clicking "Update Packages" and scrolling down to the bottom of the page. There you'll find a form for entering package names to install on the chosen systems. The only catch is that this will upgrade those packages if they are already installed, as it uses a command like yum install or apt-get install which works in the same way.

Jamie,

There is no former and latter. There is only one option on "Cloudmin Managed Systems" which is "Update packages" and shows an "Install packages" box, that receives a list of packages.

What does it do? Does it call apt-get install and yum install depending on the distribution?

Anyway, I think you are mistaken regarding the correct use of apt-get and yum.

Please read this carefully:

1)

To install packages you should do:

for p in $LIST; do check_installed $p || install_package $p done

The install_package function will then call "apt-get install" or "yum install" depending on the distribution.

Why is this necessary? Because an "install" procedure from an automation product should NEVER update a package. That is the function of an "update" procedure. We don't want to break working systems with an accidental update.

2)

To update packages you should

for p in $LIST; do check_installed $p && update_package $p done

The update_package function can call "apt-get install" and "yum install" / "yum update". If you only worked with Centos/RHEL the check_installed function wouldn't be necessary since the "yum update" command does NOT install packages - it only updates packages if they are present.

Why is this needed? Because an update procedure on an automation product should NEVER install anything that wasn't installed before.

No you will tell me that apt and yum should have thought of it themselves. I will say that it is the role of a sophisticated automation product to have this kind of business logic working so that customers can deal with a reduced probability of breaking production systems.

Before we were using Cloudmin we tried to make you guys see this on this ticket

http://virtualmin.com/node/22836#comment-103556

But you told me to use Cloudmin :-) Now we are back to bugging you.

Can you comments?

That's a good point - in the next Cloudmin release, it won't update existing packages when you use that new package install form.

So do you have systems that have packages that need updating, but are not shown in Cloudmin as having updates available?

Jamie,

I'm am a bit confused. Please look at the screenshot.

All the systems have packages that need updates.

If I press "Update packages" there is only the "Install packages" option available.

Should I see another one?

If the "Update packages" button leads to two different options then it should be called "Manage packages" or something more general.

Currently, Cloudmin can only fetch available package updates from systems with Webmin installed.

Understood.

If the server is not webmin enabled you could as well make the update text box available, just as the install box is available. That's where my comment about the update function not installing anything, only updating, comes from.

What I've said about the button label still makes sense. Saying "Update packages" is quite misleading.

Good idea, I'll do that in the next release.

Thanks!! Please bear with us, our suggestions will help you making Cloudmin better.

Jamie, please note that we are waiting for a PDF from you in order to submit Cloudmin to a public tender:

http://virtualmin.com/node/28268

This is urgent.

Hi,

What is the status of the suggestion on #11?

Another question: are you thinking of implementing groups on Cloudmin, so that managing thousands of machines is possible? (ex: customer one prodution, customer one pre-production, customer two production, customer two pre-predoctiyon,...)

One more, that connects to the previous one: we will need to manage Suse machines with Cloudmin and since the packages are not the same as of RHEL/CentOS probably it will be dangerous to do this without grouping.

Can you comment?

(We submitted Cloudmin to the tender, after we got your PDF - waiting for the results - see comment #12)

A new version of Cloudmin hasn't been released yet with the fix, but will be hopefully next week.

Regarding groups, these are already supported in Cloudmin - on the Edit Virtual System page you can put each system into one or more groups.

Regarding groups:

I see now that one can create a group, just as you suggested. We can have the group info appear on the cloudmin list and sort by group clicking on the header. But then one can't select all the machines of a certain group, right? If the group has, say, 30 machines do we need to click 30 times? I tried clicking on the first plus shift+click and it didn't select them all.

Still in this context:

I tried to register a generic centos host and a webmin enabled (virtualmin pro) centos host in the Webmin Servers Index from a Cloudmin Pro installation. I expected them to appear in the Cloudmin hosts list but none appeared.

Can Cloudmin Pro also manage external servers in terms of command execution and packages?

Duh... it is not on Webmin Servers Index...but on Add Physical System. Please ignore this question.

On the "List Managed Systems" page, you should be able to use the search form to show VMs in a group, click "Select all", and then perform some operation on them.

I don't find a search form there. Please see the screenshot on comment #8 - there is no search form.

There is a search form on the left frame just below the logout button but that doesn't search groups, searches cloudmin modules.

The search from only appears if you have more than 20 systems (by default).

I tried to add a Physical system that runs Virtualmin Pro to an existing Cloudmin Pro install and it fails saying ports 10001 to 10010 are firewalled. In fact they are: the server only exposes port 10000.

When I add that same server via Webmin Servers Index it works using port 10000. I can login to the servers Webmin interface without problems.

Can you help?

Questions:

  • which ports are needed for INPUT on the remote system?
  • anything need for OUTPUT (connection initiated from the remote system)

Note: all our servers have -j DROP by default for everything in INPUT, OUTPUT and FORWARD targets, we open ports on a necessity basis.

Another question:

When I finally manage to connect, from Cloudmin Pro, to remote physical systems running Webmin or Virtualmin I will want to issue package updates for groups of systems. Suppose I select a group of N systems to update packages for. What will be shown on the list?

1- packages that need update, that are common to all the N systems?

or

2- packages that need update that are installad in at least on system?

Will you ensure, also in this case, that packages that were not installed before won't be installed on any of the systems?

Cheers Gustavo

Jamie may need to comment regarding your package questions.

In regards to firewalling though -- communication for certain actions when accessing remote Webmin systems does require the use of Webmin RPC, which uses ports 10001 - 10010.

My suggestion would be to start by opening that -- that should be all you need, though if it doesn't work we can take a deeper look at your output rules.

Andrey,

Thanks we will do that.

Can you tell me if SLES machines, runing Webmin can be managed using Cloudmin Pro (on RHEL / Centos) in termos of package updates? SLES uses RPM and zypper.

I just added a Virtualmin Pro Centos system after I opened ports 10001 - 10000.

But once I add it its status is "Alive" instead of Webmin.

I can't run commands or upload files: ERROR: SSH logins are not available

And the graphical list of packages does not appear.

Do I need SSH login? But I don't have SSH root logins enabled. What is the Webmin connection on port 10000 for, then?

After adding SSH keys to login as root the status is "Virtualmin" instead of "Alive" and the packages GUI already appears.

Another thing:

If I try to register the system using only SSH the status instead of being SSH is "Webmin down" (see screenshot). Why does it say "Webmin down" if I didn't request the webmin connection?

It will say "Webmin down" if it tried to connect to Webmin but could not.

You can prevent Cloudmin from trying to login at all by going to Change Passwords -> Change login used by Cloudmin and selecting "Do not attempt Webmin login"

Jamie,

That solves it, but there is no option to not connect to webmin when we add the machine. Please look at the screenshot - I chose "Webmin not installed yet" and got status "Webmin down". Which one should I choose?

Can you take a quick look at the last part of comment 21, regarding package updates using the GUI?

Thanks for your support.

Thanks for pointing out the issue with the form for adding a system - I will fix that in the next release.

Regarding package updates, if you select multiple systems it will show packages that need updating on at least one of those. And it will never install new packages as part of the update process (unless forced to by YUM or APT).

Jamie,

Thanks for the answer. I will test the next release.

Regarding the package update - the behaviour you describe is good. I worried about that you mean by "forced by YUM or APT".

If package A is installed on System-1 and needs update and package B is installed on System-2 and needs update, the list will show A and B.

Then if the operator selects A and B what happens next?

If you run

apt-get install -y A B

on both systems apt will install package B on system A and package A on system B. Is this what you mean by force by APT? Or is the logic I mentioned on comment 6 already in place to prevent this?

I asked this in the context of the "install packages box" but it applies to the graphical view as well.

YUM update doesn't have this problem - the problem is that APT doesn't have an "update" command (its "update" commant updates the repo information).

So in the example you gave, the packages will only be upgraded on each system - never installed. Cloudmin doesn't just run "apt-get install A B" on each system.

By "forced by YUM or APT", what I meant was if a new version of package A adds a dependency on C, that will get installed by YUM or APT.

Jamie,

Those are excellent news. I feel quite confident to use expand our use of Cloudmin to package management from now on.

Regards Gustavo

An FYI on the "webmin down" issue:

On the Cloudmin host, in case the OUTPUT policy is DROP:

iptables -A OUTPUT -p tcp -m tcp --dport 10000:10010 -d MANAGEDNETWORK -j ACCEPT

iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

On the guest or physical machines:

iptables -A INPUT -p tcp -m tcp --dport 10000 -j ACCEPT # Virtualmin

iptables -A INPUT -p tcp -m tcp -s CLOUDMINHOST --dport 10000:10010 -j ACCEPT # Cloudmin

It's a bit confusing that the web interface needs only port 10000 whereas the remote management protocol requires ports 10000:10010. Anyway, this seems to work - feel free to add to the docs.

Moving this issue to Cloudmin, since we are now Cloudmin customers.

Are there still any outstanding problems here?

Please see comments 10, 11 e 28 and confirm if fixes were indeed made available. thanks

Yes, those fixes are done.

Thanks Jamie. I will test this and come back with feedback.

Oustanding problems on fully updated Cloudmin Pro (7.5)

  • the button with label "update packages" should have label "manage packages" because it takes you yo a place where you can both update and install packages
  • machines with status "SSH" can't be managed at all now:

"None of the selected systems have Webmin or Virtualmin installed"

This is a regression as previously at least the install option was available

  • no Help function to describe the behaviour of this feature

Please see comments 8-11