Centralized Control Panel - Decentralized Services

I would like to see Virtualmin be able to easily maintain each "service" on separate machines while keeping the control panel itself centralized.

Ex.

Server #1 - Virtualmin

Server #2 - HTTP/HTTPS/FTP

Server #3 - SMTP

Server #4 and Server #5 - DNS

Server #6 - IMAP/POP

The idea behind this type of deployment, would be to make the "Virtualmin" server do the management, and the other servers to run individual services, along with a simple "agent" which allows the "Virtualmin" server to control it remotely. (probably via SSH or an HTTP-API)

The overall goal is to allow for seamless and easy cluster management in a way that allows the "Virtualmin" server to distribute various tasks to different physical systems and/or VPS systems.

Ultimately, one would be able to deploy a "single" server setup where all "services" run from the same box (like the current VIrtualmin) or divide the tasks amongst any number of boxes accordingly, while keeping one central management point.

While I hate to refer to this; SWSoft is known for designing products this way (both their own and acquired products accordingly) The most notable product while being design for "Windows" is HELM.

http://www.parallels.com/products/helm/advantages/

If you were to adopt this type of system design, you could easily adjust your licensing which could generate more income for you guys.

I do not believe this idea conflicts with Cloudmin, but feel free to correct me if I'm wrong. In fact, I believe it would actually compliment Cloudmin, where Cloudmin could be used to manage the creation of cluster nodes to be used in the manner noted above.

Anyways, that's my two cents! (as usual)

Seasons Greetings!

-Peter

Status: 
Active

Comments

That is a good suggestion, and I work towards this when I get the chance .. but supporting separate systems for web hosting, email and so on is pretty complex to implement.

Virtualmin does support putting your MySQL and PostgreSQL databases on a separate system though, and you can run remote spamd and clamd servers for spam and virus scanning. Also, you could effectively make DNS remote by using Virtualmin's slave DNS feature, and not including the master system in NS records.

tpnsolutions's picture
Submitted by tpnsolutions on Tue, 12/29/2009 - 15:58

In theory, all you'd need to do is create a "base-line" install with or without the GUI frontend of Webmin/Virtualmin for each system which manages all the core system features like "users, quotas, etc".

Then instead of having each system take on the load of managing all web-facing services, you'd simply allow someone to install (perhaps using the VM installer) the services they want that node to handle.

Finally, in your "master" virtualmin installation (the management interface), you'd tell the system which server or group of servers are handling what task.

Personally for me, it doesn't have to be overly complex. It just means that the management server needs to keep track of which server to poll for each service based on domain or account ownership. Since the agents could be setup to "push" data back to the management server, this could effectively take away some of the challenges of the management server having to figure out where everything is.

Just another two cents on the topic ;-)

BTW, Webmin/Virtualmin already in my opinion has most of the key components in place, it's just a matter of allowing one installation to act as a master over the others, and remove or disable features not needed on certain nodes so that load is managed, and tasks are allocated.

-Peter

hear! hear!

that would meet my needs nicely, too. I could have my demanding site manager in a domU sandbox and the nicely behaved ones in the main domain.

D

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 12/13/2012 - 00:42

An illustration of the perfect cluster (at least for my needs)

Virtualmin Manager

Web Nodes
- Apache
- PHP
- Perl/CGI
- Ruby
- Python
- etc...

MySQL Nodes

Mail Nodes
- Webmail
- POP/IMAP
- SMTP
- Spam Filter
- Virus Filtering
- etc...

Storage Nodes
- FTP
- SSH
- Storage
- Backups
- etc.

*** Naturally there could be further separation than listed above, but that's just to give you an idea. ***

The idea behind this design is to have Virtualmin at the top of the architecture, handle all management (Manager) of accounts through a central control panel.

The rest of the cluster would be designed a way which allows for scaling to almost unlimited capacity by grouping nodes to handle a smaller amount of tasks (ie: Web, Mail, MySQL, etc)

If you like this kind of design, please post your comments here so we can rally enough interest in this design, and work together to iron out the best deployment strategy.

-Peter

tpnsolutions's picture
Submitted by tpnsolutions on Sat, 01/02/2010 - 23:40

It should be noted as stated in a former comment, that each node would likely have a thin layer of Virtualmin Agent (as I like to refer to it as) which handles the communication between nodes, and the "Manager". This would also handle tasks like user/group management, quotas, and other basic node operations.

Optionally, there could be a GUI for each node allowing a system administrator to do maintenance on a specific node if required.

*** These are all just ideas at the moment ***

-Peter

We have done some work towards this with Cloudmin Services - it can be used to offload MySQL and DNS hosting, and spam and virus filtering. Let me know if you are interested in alpha-testing it.

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 12/13/2012 - 00:30

Jamie,

I've considered over the past few weeks, putting "Cloudmin for Physical Systems" into our network design. Would this version work with what you're mentioning here?

We utilize VPS's which then run Virtualmin at the moment. This allows us to provision machines in multiple geographical regions. Our VPS infrastructure was in place long before Cloudmin matured, and has worked for us well. However, I believe using the non-VPS version mentioned above could become useful for managing our cluster of nodes.

*** On another note, I noticed in the API documentation that it seems the Cloudmin API is a bit limited in terms of managing services and creating domains across multiple machines, in comparison to Virtualmin's API (obvious Virtualmin has been around longer and is more mature).

Our current cluster makes use of a custom built database, and a set of shell run scripts that run to execute the Virtualmin API for a select service, on a group of nodes. (ex. web, dns, and mysql *** mail is currently outsourced to Google Apps, though this may be changing soon given Google's recent announcement on the topic). ***

-Peter

These cloudmin features sound nice but it's not webhosting where I need it for, just all kind of Linux servers.

I have a Business Cluster that I would like to manage this way, so also Samba and so on.

@tpnsolutions - Cloudmin for physical systems could be useful for providing a single console for all your Virtualmin installs, as it aggregates the domains from each system into a single list. The API for domain management is simpler than Virtualmin, as we expect that most domain operations be performed on the system managing the domain. Cloudmin acts more like a directory service for domains ..

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 12/13/2012 - 22:36

Jamie,

Just a thought to consider, perhaps to make Cloudmin that much more powerful as a sort of "console" would be to allow Cloudmin to issue commands to each of the individual Virtualmin installs.

That is, say I want to setup:

"hosting" on server #1

"mysql databases" on server #2

"dns" on server #3 and server #4

"email" on server #5

You might be able to issue from Cloudmin something like:

cloudmin create-domain --domain mydomain.com --hosting "s1" --mysql "s2" --dns-master "s3" --dns-slave "s4" --email "s5"

Which would cause a user to be created on each of the servers, along with the appropriate services enabled.

This ofcourse is a pretty rough example, but I'm sure you can see what I'm talking about.

What would make such a feature even more interesting would be to later offer the ability to choose which node a domain/service is added to based on "resources" and "groups".

"groups" would be say "hosting" (which would include all your hosting servers), "mysql" (which would include all your mysql servers).

"resources" would be say "cpu", "memory", "disk space", etc...

This added functionality would be nifty, cause then Cloudmin would be sort of the "manager" of all these servers where as Virtualmin would then handle the actually tasks on the individual nodes.

Further, say I setup a cluster of "hosting", "mysql", etc nodes in my "Fremont, CA" data center, I could then make sure that an account is created on all nodes that belong to that data center... (ex. "hosting-fremont", "mysql-fremont", etc)

Just adding to the brainstorming as always :-)

Feel free to run with any of these ideas, or any variation of them...

-Peter

The tricky part here is that some of those services interact with each other. For example, the web and mail services share the same home directory and users, so they have to be somehow shared between the two machines. This can be done (via NFS and LDAP), but the setup is complex.

Other services like MySQL and DNS are self-contained and so can be offloaded - we support this already with our Cloudmin Services product.

tpnsolutions's picture
Submitted by tpnsolutions on Thu, 12/13/2012 - 23:00

Jamie,

I hear what you're saying, however in our custom design which is powered by Virtualmin, we just create the same user on multiple nodes. Then enable each service individually. This simplifies a bit of what you're saying because each node operates technically, independently. No NFS or LDAP required.

Maybe this type of design (that we're using) isn't always consider optimal, it is however in some ways more simply and why not make things simple if you can :-)

*** NFS, and LDAP offer as you noted complexity, so I try not to do this type of stuff if possible ***

If one node dies, it won't take down the rest because they're running independently. It technically means that Cloudmin is just secretly talking to Virtualmin's individual API in theory... Just making issuing a command to a number of nodes a bit easier than doing them all manually one by one.

The other thing is, for services that don't require shell access (mail, mysql, etc) you simply don't enable it for that user on that node :-)

*** you could go one step further by not adding unneeded directories to the home of each user on each node. don't need a maildir on a "hosting-only" node for instance, or a public_html directory on a "mail-only" node. these directories and related settings could be added based on what that node is suppose to handle. ***

-Peter

Guys, why are you only talking about Cloudmin ? this is webhosting based AGAIN.

I would like to suggest that things are managed from webmin instead of cloudmin, the part under must be teh manager, nog the part on top.

I mentioned Cloudmin because it isn't just for virtual machine hosting - it can also manage multiple Virtualmin instances on regular machines. If we were to implement something like this, it would make use of Cloudmin as it already has the ability to manage multiple Virtualmin installs..