Submitted by sfatula on Sat, 12/11/2010 - 23:27
I can see where using virtualmin command line program I can change template settings and other settings. What I don;t see, perhaps I missed it, is how I set System Settings -> Virtual Configuration type settings, all those screens.
I want to script setup of new machines without editing files directly.
Status:
Active
Comments
Submitted by JamieCameron on Sun, 12/12/2010 - 00:26 Comment #1
There is an API command called
modify-template
that can do this .. but the parameters it takes are rather obscure, as it refers to template variables using their internal codes.Which global settings do you want to change exactly?
Submitted by sfatula on Sun, 12/12/2010 - 00:29 Comment #2
But I don't want to modify a template, I want to modify System Settings -> Virtualmin Configuration, all of those screens, you name it. Any number of settings from those screens, NOT templates. So, simple example, on the Actions upon server and user creation, I want to change the from address for email sent by virtualmin. But that's one example, lots of those screens I have settings I want to automate changing by my installer when I load a new machine. Right now, it takes me a day to set all that up.
Submitted by JamieCameron on Sun, 12/12/2010 - 02:35 Comment #3
Unfortunately there is no API for the settings on those pages.
However, many of them set make changes to /etc/webmin/virtual-server/config . If you work out what lines get added to that file by the actions you take in the UI, you could write a script to add the same lines on a new system..
Submitted by sfatula on Sun, 12/12/2010 - 03:29 Comment #4
Then consider this my request for an enhancement!
I am not so sure its that simple. For example, the very first setting, "Also update outgoing addresses for mailboxes", I am suspecting that might change other files as well, perhaps postfix for example. So, in some cases, it may be difficult to determine what all Virtualmin does when those settings are changed. But, what else can I do. It's the way it is.
Enhancement?
Submitted by JamieCameron on Sun, 12/12/2010 - 15:56 Comment #5
What I would recommend instead is configuring a system the way you want it, then taking a backup of all Virtualmin settings with a command like :
virtualmin backup-domain --all-virtualmin --dest /tmp/virtualmin-config.tar.gz
then copy that
virtualmin-config.tar.gz
file to the new system and running :virtualmin restore-domain --all-virtualmin --source /tmp/virtualmin-config.tar.gz
Submitted by sfatula on Sun, 12/12/2010 - 16:05 Comment #6
Yeah, I thought of that, but a lot of settings still have to be changed. There's a lot of settings where the domain name of the host is in there, etc.
It MIGHT work to just restore the settings I can't change from the API though, the Virtual Configuration settings, not sure, have to go through them and see what might be the fastest way.
What I WANT to be able to do is load up a new machine with all the correct settings in an hour.
Submitted by JamieCameron on Sun, 12/12/2010 - 17:43 Comment #7
Most settings are host-independent, so you could restore a backup and then use the API to change the host-specific settings.
Another approach if you are using virtual machines and Cloudmin is to create a "template" VM which you then image, and create new VMs from. But this isn't much use if you are only using real systems..
Submitted by sfatula on Tue, 12/14/2010 - 00:29 Comment #8
What if I do not want all virtualmin settings backed up and restored? Which backup setting stores off the System Settings -> Virtualmin Configuration settings? I don't see any that make me think they might do this.
Submitted by JamieCameron on Tue, 12/14/2010 - 16:58 Comment #9
The checkbox labelled "Module configuration" (or "Virtualmin configuration" in future releases).
Submitted by sfatula on Mon, 12/27/2010 - 22:03 Comment #10
So, I followed your virtualmin command line backup and restore, and, no webmin settings werre changed on the restored machine. There are tons of webmin changes I want to carry over, ports, category assignments, logging, authentication, you name it, none came over. Other thoughts?
virtualmin restore-domain --all-virtualmin --source /tmp/virtualmin-config.tar.gz Checking for missing features .. .. all features in backup are supported
Starting restore.. Extracting backup archive file .. .. done
Restoring Virtualmin settings (Module configuration, Server templates and plans, Reseller accounts, New mailbox email messages, Custom fields, links and shells, Third-party script installers, Third-party content styles, Scheduled Virtualmin backups, FTP directory restrictions) .. .. done
Restore completed successfully.
Submitted by JamieCameron on Mon, 12/27/2010 - 23:58 Comment #11
Those webmin-specific settings are not considered to be part of virtualmin, and so are not included in a virtualmin configuration backup. For those, you would be better off using the module at Webmin -> Webmin Configuration -> Backup Configuration Files. You can select which modules' settings to create a backup of - for example, to migrate Webmin port(s) and authentication options, you would backup the Webmin Configuration module.
Submitted by sfatula on Tue, 12/28/2010 - 09:06 Comment #12
Ok, so, I figured you'd say that. SO, to correctly backup a machine for the purposes of restoring it, one has to use BOTH virtualmin AND webmin backups. WHICH webmin modules to back up could be tricky since some may overlap. For example, virtualmin likely backs up DNS settings for the domains it hosts, BUT, may not back up the DNS settings (general) or module settings. Same for postfix, etc.
SO, does anyone have the perfect and correct way to backup up a server so that there is nothing to do once restored? (Not speaking of anything outside of webmin or virtualmin). And what to restore first (My guess would be webmin backup)? .
Secondly, I will document all the virtualmin things that do not properly restore in another ticket. One I mentioned previously was postgrey, it turns it off since it is not loaded by default on a new install (I guess, or it may also be off since postfix is not entirely restored either). ANyway, I'll document those.
For my purposes here, I'll mostly concentrate ob webmin backup setting since when setting up a new machine, there aren't really any domains so most all settings I want to keep are within webmin.
Submitted by sfatula on Tue, 12/28/2010 - 09:27 Comment #13
(See previous comment) I should add about the postgrey that even if you loaded postgrey during the restore, the other issue is all the manual entries that may have been added to the standard postgrey install for whitelisting. Those should be retained also.
Submitted by JamieCameron on Wed, 12/29/2010 - 02:07 Comment #14
Unfortunately Virtualmin's backups are mostly domain-based, and so don't include all global settings like the BIND config, Postfix ports and so on. This allows them to be restored on a different operating system or Linux distribution, but means that some global settings will not be included as you mentioned..
In your case, is the destination system the same Linux distribution and version? If so, you might want to consider doing a lower-level backup of the whole system, at System -> Filesystem Backup.
Submitted by sfatula on Wed, 12/29/2010 - 08:34 Comment #15
Wow, I thought the purpose of the virtualmin backup (which I have seen people comment in the forums as well) is to be able to reload the machine after installing virtualmin? This is not the case?
There are DAYS worth of settings to be made if a system cratered, most all of which are virtualmin/webmin related. I am not speaking for the stuff outside of those products, just what they use.
Are you telling me that postgrey is not restored and I lose all those settings? SInce I have set up postfix and domains to use sender canonical maps, are you saying those SHOULDN"T be restored? (they are not). etc. etc.
Those are all things set up from within virtualmin and webmin. Your whole page here:
http://www.virtualmin.com/documentation/system/backup-and-restore
bascially states is does what I thought it should do. For example:
"This allows you to restore the state of an entire virtual server (including all databases, users and aliases), without effecting other parts of the system."
So, canonical maps are not the state of that virtual server? They sure are!
Postgrey is not a virtualmin global setting as referred to in the global settings section? SInce it is set within virtualmin? I sure would think so!
There are a whole lot of settings I believe should be backed up by Virtualmin that are set there, and webmin that are set there.
So, I guess this thread has been about two different problems -
Restoring a system, and it sounds like I MUST do file system backup combined with virtualmin backup for this.
Moving a virtualmin/webmin system (without ANY domains) to a new system to set it up as it's own new host. That is the part that seems to be troublesome. I guess I will experiment with webmin backup configuration files to see how much I can set up without scripting. Same for the virtualmin. Without these, thus far, I have more than 1,000 lines of scripts and I am not even remotely close to be done, estimate 10,000 lines would be needed. That's way too much work to use a machine as a seed to set up another machine, basically, identically but with no domains. Troublesome spots are of course host names stored in various places, ips stored in various places.
Submitted by sfatula on Wed, 12/29/2010 - 20:51 Comment #16
So, assume a server with a root filesystem, boot (of course) and a /home filesystem (which happens to include mysql data).
Would a good backup plan include:
In the end, restoring the root filesystem from backup SHOULD restore all VMIN/Webmin settings. So, aklk that is left is user data, which SHOULD come from the virtual server backups. I see no reason to filesystem backup /home, correct?
Submitted by JamieCameron on Wed, 12/29/2010 - 22:44 Comment #17
Yes, if you were to backup everything except /home using the Filesystem Backup module and then use Virtualmin backups for domains, that would be sufficient to move everything to a new system.
I agree that Virtualmin's global backup feature isn't doing a good job when you've customized a lot of things outside of it's regular UI. I am working on improving this ... I'd be interested to know what kinds of settings you have customized and would like to include in a backup.
Submitted by sfatula on Fri, 12/31/2010 - 16:32 Comment #18
As mentioned, it's not just things out of the regular UI, it's lots of things in the regular UI.
The thought occurred to me, if I could get a 1 domain VIrtualmin PRO license somehow, I could use this on a $3 VPS plan I have and that would be my seed. Since it would have no domains (well, itself), I should be able to copy over the virtualmin config by just using the Virtualmin/Webmin directories and not have to worry about all those domains, etc. Yes, there is still some scripting to do, but, not much!
Would that be possible?
Submitted by sfatula on Fri, 12/31/2010 - 17:45 Comment #19
If that is not possible, perhaps you can allow my one license to be used on 3 machines? The third would be my cheapie VPS with only 1 domain to be used as a seed system. It a really really really cheap VPS and could never beused for hosting (virpus,com). $3 per month!
Submitted by JamieCameron on Sat, 01/01/2011 - 16:10 Comment #20
Sure, you can use it on three systems as long as only one is actually used for hosting..
I am working on including the missed settings (like postgrey and DKIM) in Virtualmin backups.
Submitted by sfatula on Sat, 01/01/2011 - 16:21 Comment #21
In some previous ticket, you had indicated that a license can be used on one server AND another server of our choosing, typically or backup. Is this not the case? i.e., can only one really host anything?
If that is the case, then, any chance of a < 10 user license? Or, do you simply not care about single user hosting?
The second server must be a backup, hot spare, or development server, and not for production use. Every production server needs a license.
The 10 domain license is currently our smallest license...not because it shouldn't be used for a single site, but because that's the cheapest license we can afford to support (we just about break even on Virtualmin 10 licenses, because they have the highest support load on average, and would lose money on them if we charged much less; we have three salaries to pay here, and are getting close to needing to add a fourth), and there isn't going to be significant difference in cost between supporting someone with one domain and ten domains (we have a baseline cost for support that is almost identical no matter how many domains the user is licensed for, and the smaller products actually generate more support load). It's an unfortunate paradox of our current licensing model that our lowest cost products are also the most expensive to support. I don't know how to work around that problem, short of not supporting lower cost products, or limiting the amount of support available...which I'm simply not comfortable doing. cPanel solves it by not offering any smaller licenses (they only offer an unlimited license), and Plesk solves it by charging for support on an incident basis. But, it's a problem that has to be solved somehow if we're going to offer anything cheaper than $139.
Hope this helps explain where we're coming from on this. We try to be really generous in our pricing and licensing terms; and I believe we're the most liberal and most cost-effective option for a full-featured control panel (cPanel and Plesk do not permit backup or hot spare servers or development servers, for example). But, we do have to pay our salaries. Millions of downloads doesn't put food on the table or keep our rent paid. ;-)
Submitted by sfatula on Sat, 01/01/2011 - 17:41 Comment #23
Oh, I totally understand all that, except, for one minor detail.
If one already owns a license, as I do, its very highly unlikely an additional 1 user license would add ANY additional support costs.
True?
That's probably true. We just don't have any way to sell you anything else right now.
We have the limitations of our shopping cart and license manager code (which is hairy in the way it interacts with the shopping cart; quite horrifying actually, and every time I touch it something breaks). We can't possibly have a license to match every single circumstance. In that case, we'd have as many license types as we have customers and as many price points. We'd need a full-time person just to manage the product list. ;-)
I am currently working on making discounts work in our shopping cart, and I'm also working on being able to offer a smaller 5 domain license to resellers (who generate the fewest support queries of all, since they tend to have dozens of installs and their experience level with hosting on Linux is very high to start with; our biggest customers product the lowest number of support queries per license, by far). Once that's in place, we can hook up existing customers that need them with smaller licenses for their VPS needs (this is the most common reason people ask for a smaller/cheaper license, is for use on a VPS), but also for people in your situation where you just have one high-load site that needs its own server.
In the meantime, since it's just one domain, go ahead and use your current license on that other server. We'll just trust you to buy one of those low-cost secondary licenses when they arrive in a couple of weeks. (Remind me to set you up with the reseller role in a couple of weeks so you'll be able to see that license.)
Submitted by sfatula on Sun, 01/02/2011 - 16:12 Comment #25
Here's another simple one. I go into webmin and DELETE modules I know will never be used. Now, I go to a new machine, install it, and, restore the backup. The modules will come back, right?
Again, my example here is to try and take a PERFECT configuration from an existing machine, and, make a new machine with no domains in it. (VIrtualmin AND webmin).
I thought maybe I could install the products, and, reload webmin but it appears I'd have all these extra modules in it. Still can't find an easy way to do this.
Perhaps my best way is to rsync the webmin and virtualmin directories. This doesn't handle all the servers, etc. like MySQL, but, don't care I think.
Couldn't I install virtualmin/webmin by simply using the rsync then changing the server key, etc.? Nah, that probably doesn't work either, can think of problems with it.
My best bet MAY still be to install virtualmin on new machine, reload webmin configuration from webmin configuration back, patch up settings as needed, and, then reload virtualmin configuration with no domains. Can you think of a better way? I am wanting to be able to startup a new hosting server as easily as possible and right now, I could easily spend 3-4 DAYS typing in webmin and virtualmin settings.
So, do you delete every single file on your system that you don't use regularly? Isn't it overkill to delete a bunch of Webmin modules? (Which also interferes with your ability to do integrity checking on the packages you have installed; so rpm -Va will spew out hundreds or thousands of changes, even though everything is legit; it's usually a bad idea to delete files out from under the package manager.)
Webmin modules are tiny. You can't be saving more than a couple MB (probably less) by deleting a bunch of modules. If you simply disable them for the root user (and they'd only be visible to Virtualmin users if you setup Virtualmin to grant them that privilege), you'd only need to copy one file over (webmin.acl). I can't think of any reason to delete modules, except in extremely space-limited deployments (embedded uses, for example), which is what the minimal Webmin package is for. Hiding the modules is the preferred method for removing clutter (and Webmin will automatically tuck in unused modules into a hidden modules category for you, much like Windows will clean up the desktop of unused files for you automatically).
But, you're making it sound way more complicated than any Virtualmin migration I've ever seen. ;-)
Seriously, in my experience even a pretty complicated migration takes an afternoon, plus a day or two of testing and making sure the DNS gets synced up right (which requires some tricks for database backed sites to make sure the user doesn't get the old site; but that requires application level cooperation that Virtualmin can't do anything about).
How many machines are you deploying? I thought we were talking about one server? Why would you spend days automating something you're only doing once? There are, of course, lots of ways to setup new servers on a large scale in exactly the way you want to do it (I've used kickstart, as well as a couple of different disk mirroring systems, in the past, for this). The Virtualmin install is scriptable, and on a fresh system, it would be entirely safe to copy the config directories over from your "template" system or disk image.
It sounds like you're conflating two different problems here:
Migrating one server
Setting up multiple servers in a specific way
The former requires some human involvement, because the original server didn't come from a template, and can't be used as a template for setting up other servers because it has a bunch of existing websites and customizations on it.
So, which problem are we trying to solve here, and let's get that one solved so we can wrap up this ticket. (If you want to make new specific tickets with feature requests to make migrations easier, that's cool, too.) ;-)
Submitted by sfatula on Sun, 01/02/2011 - 17:59 Comment #27
I've been sitting here for 6 hours now, and, I haven't gotten past the webmin section of the webmin program for configuring a new system. I am speaking of setting up a new system. IDEALLY, the same as some existing system without any domains, mail, dns, mysql database, you name it. The systems are remote, and, I have no ability to mount dvds, etc.
Currently, I am editing all the usermin options, and configuring each of those as far as what the user can or cannot do. It's far from trivial. I don't like the defaults in many cases.
As far as deleting modules, the option is there, there is possibly a performance penalty for loading the left hand menus when the modules exist based on another ticket. So, why not delete them?
How many machines? Hopefully, dozens, we'll see.
I want to set up a server, every so often, in a specific manner. That's the problem I am trying to solve. And I want to take an existing server which I know to have our "perfect" configuration and somehow use it to create the new server (empty of domains). Since virtualmin/webmin versions and options change, having a specific pre-defined setup doesn't cut it for me.And, perhaps the original perfect setup was not so perfect after all.
So, there's module configurations (including their default global settings typically done with "module config" within a specific module, there's all the webmin configuration, usermin configuration, apache configuration changes, postfix configuration changes (tons), I duplicate some modules like MySQL since the server will also serve as replication client, so, changes there, etc. etc. etc. I could go on but no point.
I simply want to find the best way to take an existing "ideal" configured server, and make a new configured server with the same settings except no data or domains. Obviously, some scripting is needed to clean up host name, ips, license #, etc. and that's not an issue, I will gladly do that. But configuring from scratch via a script that was editing webmin config files and such directly is clearly a LONG LONG path.
Again, not speaking of ANYTHING outside of webmin/virtualmin, that has to be handled obviously. Just any and all things that it's possible to change from within those products.
My goal is to be able to deploy a new empty configured server in say an hour. Not days and days of effort, which seems needless since the configuration has previously been done on another machine.
As far as deleting modules, the option is there, there is possibly a performance penalty for loading the left hand menus when the modules exist based on another ticket. So, why not delete them?
That was referring to Virtualmin plugin modules (modules with virtualmin- in the name, and only available on Virtualmin systems), not Webmin modules. The Webmin module menu is not subject to the kind of complexity that the Virtualmin menu is subject to, and is never particularly slow, even on very slow systems (as far as I've ever seen). Webmin knows what modules are available to a specific user based on one file (the acl file for that user), while Virtualmin generates the menu based on a huge variety of files and configuration options--the former is basically static per user, while the Virtualmin menu is highly dynamic based on what domain you're looking at and the user's privileges. As an aside, Jamie has made quite a few changes to make the Virtualmin menu faster; that problem is a problem for better code, not more manual labor on the part of users.
And, I've already mentioned one reason not to delete them: The package manager won't know about the deletion and will report it as a corrupted install, when you verify packages.
Another reason is that there are a number of circumstances where the module can come back when deleted (such as when upgrading the package).
Deleting modules is simply not a productive use of your time. If you're seeing performance problems with the Webmin menu, then that'd be something we'd want to look at specifically. There should be no notable performance benefit to deleting Webmin modules, in any part of your system.
I want to set up a server, every so often, in a specific manner. That's the problem I am trying to solve.
Webmin and Virtualmin are not provisioning tools. Webmin has a UI for dump and tar in the Filesystem Backup module, which can be used to do filesystem level backups and restores, but that's not provisioning of the sort you're after. There is a provision product based on Webmin called LinMin, but we don't have anything to do with that (Jamie wrote the initial version for them under contract many years ago, but it is independently developed now, as far as I know). I've never seen it or used it, so I have no idea what it can do. There are many server provisioning products on the market. I don't know anything about them, though.
But, we're not in the server provisioning business, and make no claims to be. ;-)
When I have done server provisioning in the past, I used either kickstart (which is standard in RHEL and CentOS) with a PXE boot network install (which you can't do with servers on a network you don't control), or rsync or ssh to copy in the portions of my template system that I wanted, or a combination of those techniques.
Since virtualmin/webmin versions and options change, having a specific pre-defined setup doesn't cut it for me.
Everything on your system is going to have new versions happening all the time. It's not specific to Virtualmin/Webmin. Just run "yum update" at the end of your setup script or manual process. New versions of everything will get pulled down, and all will be well. That's why package managers are awesome. You no longer have to think about whether you have the newest version...just tell the package manager to get you all the newest versions, and it does it for you.
If you want to talk about the Webmin configuration problem, and you only want to copy over the Webmin/Usermin/Virtualmin configuration, you can just copy the /etc/webmin and /etc/usermin directories over. That's where all of their configuration goes. Webmin, Virtualmin and Usermin are not mysterious about where their configuration goes. It's all in /etc/webmin and /etc/usermin.
But, it sounds like you want the whole system to match exactly, which is way outside the scope of anything Webmin/Virtualmin tries to tackle (unless you have the ability to make use of the Filesystem Backup module to backup and restore the filesystem completely on the new system...which probably realistically requires you to have console access and probably a bootable CD to start from, since replacing a root/boot filesystem on a running remote system is hairy/scary stuff). We're not building server provisioning or server mirroring software here. We just manage your servers; we don't deploy them.
This ticket has gone all over the place. I'm having a hard time keeping up with what it's even about. ;-)
In short:
We don't do provisioning or server replication. If you're deploying dozens of servers, you'll want to get familiar with some tools that are for that purpose (or write your own scripts to do it, which is what I've usually done).
Webmin and Usermin and Virtualmin configuration is very, very, very easy, to replicate to new servers. It's all in /etc/webmin and /etc/usermin. As long as the system has all the same packages (including optional Webmin and Virtualmin modules), everything will Just Work (there are some things you'd want to be aware of doing this, such as explicitly set IP addresses rather than letting Virtualmin auto-detect them, which would require a bit of sed after you copied the files over).
Submitted by sfatula on Sun, 01/02/2011 - 19:35 Comment #29
I won't continue the part about deleting modules, you have your opinion, I have mine. I agree to disagree.
It is not sufficient to simple copy over the etc dirs. What about all the postfix options I set from within webmin? They won't be there. What about the MySQL changes I made from within webmin? They won't be there. etc. Firewall, mysql, etc. etc.
So, I was more wanting to know what ideas you might have to do this. I am not the least concerned with sed and changing ips, simple. It's all the things webmin does behind the scenes to manage OTHER packages that it the issue.
So, it sound like you have no idea how this might be done. That's ok. I'll have to find the best way then, was just more hoping someone had done it. I am not trying to copy entire servers, images, bare metal, etc, I want the installs to run like they aways do. I just want to change the configuration instead of having two screens up, old system, and new, and, making them match screen by screen.
Sorry for the confusion. TIcket closed unless you have other thoughts.
Submitted by sfatula on Tue, 02/15/2011 - 14:57 Comment #30
"In the meantime, since it's just one domain, go ahead and use your current license on that other server. We'll just trust you to buy one of those low-cost secondary licenses when they arrive in a couple of weeks. (Remind me to set you up with the reseller role in a couple of weeks so you'll be able to see that license.)"
Is this available yet?