extended support for custom fields

Related issue: http://virtualmin.com/node/12245

not quite, but I think it may provide for a good jumpstart. Basically, I need a way to access individual fields and look up the value using a corresponding key. So this would need to be accessible via something like this:

On the other hand, someone is probably going to point out that these values are unlikely to change, so either these lookups would need to be cached somehow, or completely prevented - so I guess it would be better to expose these to environment variables available to the httpd/apache process via some optional flag for custom fields.

So I think on the webmin side of things, one would merely need to extend the apache vhost section by adding something along the lines of this:

See: http://httpd.apache.org/docs/current/mod/mod_env.html#setenv

In turn, on the web app side, we could be using apache_getenv() to retrieve the webmin-configured value.

yeah, basically along those lines - the primary idea is that I want the webmin/virtualmin vhost configuration to be fully manageable, even after installing a script - i.e. when migrating vhosts for example.

Actually, this is related to the other suggestion I made, i.e. about supporting FTP/SFTP-based install scripts.

Some background: From a maintenance standpoint, the two most common tasks involving virtualmin here (and probably elsewhere) are either consolidating existing websites (i.e. turning them into vmin vhosts) or scaling things up at some point, i.e. migrating virtualmin vhosts to other servers, possibly even to dedicated servers at some point (personally, I tend to use webmin anyway - but that cannot be said of all the people I am dealing with, and it's not a feasible assumption to make).

Also, I know quite a few people who are using virtualmin's "clone server" feature to develop web applications from scratch, i.e. they have set up various apps like wordpress/drupal/phpBB/mediawiki locally and added all the usual things that they typically require (config files, plugins, themes, styles etc) and then use a handful of such virtualmin vhosts as "templates" for starting new projects by cloning them whenever they start a new project.

And then, it is also not uncommon having to move things to the client's server at some point, which may not be local, and which may not run webmin/Linux.

The good thing about services like simplescripts is that these can be also easily used to do a simple form of unit/regression testing by deploying a local web app to other server environments, including non-Linux/Unix based environments like Windows IIS.

Thus, virtualmin's current way of "patching" those config files directly makes the migration part tricky, because once the script is installed, webmin no longer has any concept of dealing with the app afterwards, no matter if it's editing configuration properties or if it's migrating the website to a new server.

What I am suggesting involves adding another layer of indirection and instead of patching those files by adding hardcoded config values, using environment variables to dynamically look up config values, so that webmin is fully aware of these config values and can actually manage them via its custom fields, even after completing the initial installation/deployment.

In summary, I am looking for a way to support "remote installation" and "remote migration" without being necessarily specific to webmin (web apps using config values via ENV VARS can also be easily used elsewhere) .

And based on looking into the docs, and based on working with virtualmin/webmin, it actually seems like an excellent platform to make this happen and support such use-cases for "power users", i.e. seamless website migration between servers using custom fields for configuration values, using SFTP/FTP for remote deployment and using these features in combination to easily conduct regression testing, but also template-based web app development.

Sorry for all the verbosity here, I guess it's difficult to get across - but basically the idea is to get rid of all the repetitive steps when migrating a website to a new server or when deploying it to a new environment remotely.

Leveraging webmin's support for custom fields to delegate control of config values to webmin seemed like a feasible approach to me.

So, a specific use case would be: - configure a basic wordpress install locally, install various plugins/themes etc - use this as a "template" for a new wordpress website (e.g. developed for a client) - clone the server, while being fully aware of config files (install scripts) by using custom fields - finally, be able to deploy the website to a new server via remote installation over FTP/SFTP

Maybe I am missing something, but it seems that using custom fields for such script-specific config values would solve quite a few potential issues to support such use-cases.

Thanks

PS: I was hoping to look into the Virtualmin/Perl code but could only find the Webmin/Usermin github repos - where do I find the virtualmin code, or could you alternatively provide some pointers (filenames) to find the relevant code locally (specifically, 1) adding directives to Apache vhosts (SetEnv) 2) the code for the current custom fields implementation to add an option to expose them to apache and make them mandatory, and 3) the location of the "install script"-APIs to check if I can easily replace substitutions with getenv() calls instead - it's just an experiment for now ...

EDIT: Ok, I think I have found some relevant stuff in virtual-server-lib-funcs.pl - namely, the various *_custom_fields() functions, will look into list-custom.pl and modify-custom.pl next ... still have to find the vhost handling code.

EDIT2: Okay, if I am not mistaken, it seems that I need to look at the "global variables" and "custom fields" options to come up with apache vhost variables (SetEnv) that can be managed as custom fields and made mandatory, and edit_phpmode.cgi seems to be the place to check for apache vhost handling, whenever these are updated, httpd would need to be applied/restarted - does that sound about right ?

Here's a drupal-based article on using virtualmin's "clone" feature for easily replicating pre-configured templates: http://drupion.com/resources/control-panel/cloning-virtual-servers

Here's another discussion (this time on WordPress): http://www.turnkeylinux.org/forum/support/20100203/webmin-wordpress-site... http://www.virtualmin.com/node/24410#comment-109726

As you can probably tell, web developers would LOVE seeing virtualmin extended accordingly :-)

Currently, the "clone virtual server" feature also explicitly reads: "Warning - any web applications in the cloned virtual server will continue to use databases in the original virtual server until their configuration is manually changed." - so it also needs manual patching afterwards. Thus, this would be another issue that could be solved by delegating configuration of installed scripts to apache environment variables via mandatory custom fields.

Related:

http://www.virtualmin.com/node/26152

http://www.virtualmin.com/node/20242

(The point being not to do link spamming here, but to demonstrate that we could solve quite a few limitations by pursuing this further - the restrictions discussed here are not really necessary)

Perhaps a better solution would be for Virtualmin to know how to update the install directory and database information for scripts that have these in their config files when a domain is moved or migrated? That would avoid the need to perform auto-configuration tricks with environment variables ..

That would involve teaching each installer script how to update/replace the corresponding lines - which must be done already currently. But I am not sure how scalable that is.

There are so many variations of "installer" scripts, and even just a "stock" WordPress install can be easily messed up by editing a few files, up to a point where auto-update or plugin installations may no longer work.

Then again, you must be doing this already during the installation, so it should be possible to let the installer expose a function to update the corresponding entries at a later time if required - ideally, it should use some checkpointing feature, i.e. store a basic md5sum or CRC of the file after it was modified so that it can make assumptions about it not having changed - or if it has changed, it should show a big red warning saying that the file has been modified and that modifications may no longer be safe.

I am not sure if changing the substitution mechanism to add getenv() calls instead of hard-coded values qualifies as "auto-configuration tricks" though - it's not exactly brain surgery, and environment variables are commonly used for all sorts of server related tasks, including in php/httpd space, so it would actually not be that bad.

To me, it seemed at least much more straightforward than having to edit tons of files :-)

Besides, environment variables are scalable by nature - any custom-coded support for "paths" or access credentials would not be particularly flexible or future-proof - imagine some 3rd party script requiring possibly other information to be 100% migratable - environment variables are straightforward to set up and manage.

To be honest, I am not just thinking in terms of "shipped" installers/scripts here - those are probably a no-brainer, as you guys will handle the details - but unsupported/custom installers could very well benefit from some sort of API to delegate the configuration of such parameters to webmin - be it via reaching out to CLI tools, by using environment variables or by coming up with a simple helper script that can update/patch things reliably, despite possible changes to the underlying files.

Let's also think in terms of entry barriers: an environment variable based approach would be extremely straightforward, and would even allow newcomers (non-Perl programmers) to contribute new/updated scripts - because all it would involve is locating the corresponding lines and adding getenv() calls - while also specifying a profile with environment variables that are required for the corresponding app, and you'll be all set...

Here's another example why delegating the configuration of web apps to webmin/virtualmin would be a good idea and help simplify things tremendously:

http://www.virtualmin.com/documentation/system/backup-and-restore#using_... While every effort is made in Virtualmin to convert backups as necessary when restoring on a system with a different configuration to the original, in some cases this cannot be done 100% correctly. For example, if the new system uses a different base for home directories (like /virtualmin instead of /home), paths in PHP, Python or Ruby application configuration files may no longer be correct.

Here's the simplescripts reference for installing scripts: http://www.simplescripts.com/articles/script_packaging

Note how this simply uses define() to globally define config values - but using server-side environment variables for this would be more flexible, and not specific to a certain language

I have been playing with this a bit and been thinking about your comments, and based on my experiments I am not convinced that environment variables would be particularly "tricky" here.

In fact, I cannot think of any other approach that would be fairly straightforward to implement, while also consolidating/unifying the implementation (i.e. reusing existing code) and supporting many desirable features in in one go, using extended custom fields/server variables, such as:

• reliably backing up and restoring vhosts, while being fully aware of config settings for each installed script, to provide customization opportunities
• migrating vhosts to other virtualmin servers, possibly using even webmin's support for clustering/RPC
• supporting remote-deployment of existing vhosts, even to non webmin servers - without requiring any manual editing/patching
• cloning servers, including DBs
• cloning servers while also allowing them to be customized/parameterized using custom fields afterwards, affecting their configuration

By delegating the specification of configuration values back to webmin/virtualmin, it would all of a sudden even become possible to export existing servers as server templates, including installed scripts, while webmin would be fully aware of each installed script and its required configuration values - i.e. a wizard could be presented to easily update things as needed.

thanks for the update - BTW: even just the feature mentioned last, i.e. being able to turn installed scripts (wiki, wordpress,forum) back into templates for new installations would be extremely powerful and could also help lower the maintenance overhead - while also allowing other virtualmin users to contribute templates for web apps.

Being able to take any existing virtualmin vhost, clone it and customize its configuration via webmin would be a great time-saver.

For example: - clone blog.example.com - notify: "there's a wordpress blog installed on this vhost, would you like to re-run the configuration wizard or use the existing config (default) ?" - override & customize config: specify db to create/use, table prefix and access credentials

I still think that it would be quite possible for Virtualmin to know how to re-configure scripts to use a new home directory or database connection. The hard part is the DB connection, as most scripts can presumably detect their install directory based on the path to the PHP file being run.

Well, if you truly don't like the idea of using env vars or explicit shell commands for this, one could also think about extending the RPC feature so that scripts would call back into webmin to get their configuration values for things like DB access (db name, user, password, host, port etc).

so instead of patching scripts with hard-coded credentials, you would instead add lookup commands to call the webmin RPC system and get the values for each script.

Thinking about it, I actually like the idea a lot! :-)

It's just that this part of the webmin RPC would be ideally exposed via a more standard means, i.e. SOAP/JSON/REST or AJAX - all web apps can make such calls.

Understood, on the other hand my original suggestion would also do just that, i.e. turning static config info into getenv() calls ...

The only other idea that I can currently come up with, is having a webmin specific subfolder in each script tarball along the lines of "config.webmin" that can be looked up by webmin to write updated configuration info directly into the static file you're currently using - this could then also be done via RPC or webmin/virtualmin-based "wizards" to update things when a vhost is moved/migrated to a different server, or when a vhost is used as a template by "cloning" it.

not sure how well that would work though - ideally, people could manage all scripts installed to a vhost that way - i.e. if someone installs phpBB, wikimedia and wordpress, all of these config files should be managed via webmin.

PS: Can you point me to some example script installer in the git repo, so that I can take a look and see what's currently involved ?

Thank you for the pointer, will have a look - but basically, those scripts seem to always create a static configuration file, right ? Are those literally static or can these be *.php files, too ?

The trick is updating apps like phpBB and Wordpress to use it, which would mean patching their source and then keeping that patch up to date. Couldn't we directly use the existing file names instead? I mean, phpBB, WP etc already know about those files, so it would be a nobrainer. For example, in SquirrelMail config.php:197, I now have this:

Why not simply introduce a lookup call here:

I do not know enough about those install scripts - but are these primarily php scripts or also non-php code ?

My original suggestion to use environment variables here for interfacing between virtualmin and those apps would obviously be agnostic to the type of script/CGI used, as these could be looked up by any script using their equivalent of a getenv() lookup call.

On the other hand, if scripts are >= 99% php (?) one could also simply provide webmin-specific APIs to the php interpreter, e.g. by simply including a webmin-specific header that makes the corresponding lookup APIs available.

For instance, the config.default.php file for phpMyAdmin would then get a stub with APIs prepended to the top of the file, i.e. something that includes "webmin-installer.php" and exposes a handful of APIs: - get_host_and_port() - get_database_name() - get_table_prefix() - get_user() - get_password()

This would be sufficient and 100% self-contained - so there would be no need to modify ANY other files beyond config.inc.php - it being a php file, it would be processed by the app during initialization, and the corresponding lookups would be done transparently, without the app (phpBB, drupal etc) having to be aware of where those values came from.

Internally, these lookup functions could be either mapped to environment lookups via getenv() calls or to webmin RPC calls to get the corresponding configuration values.

That should be between 50-100 lines of php code that would need to be made available to each script, e.g. as part of a "phpMyAdmin.webmin" or "phpBB.webmin" directory, where the corresponding integration wrappers would reside - and whenever something is edited, updated or migrated - the corresponding values could be easily changed via webmin, without the script needing ANY patching - i.e. complete delegation to webmin, because those files would never directly contain any access credentials, just lookup functions to get the values out of webmin.

I only checked a few scripts in the virtualmin (GPL) directory, but it seems this should work - all config files seem to be *.php, i.e. are parsed/executed, so can also contain lookup calls to get the corresponding values via virtualmin. django or other non-php scripts would obviously require a corresponding shim layer to retrieve values via python, but it seems doable - even without requiring any file-level patching.

The neat thing being that this would also support multi-script installs, where each script may live in its own folder, while still supporting cloning complex sites and customizing things afterwards.

Am I missing something ? What do you think ?

typically, there's a single config.inc.php file that is generated by the drupal/wordpress or phpBB installer - so that's the only file that would need changing, which should be a no-brainer because it is automatically created during installation.

Regarding your sentiments concerning a comon config file format, I agree 100% - however, that's not going to change until people (projects & companies) team up and lay out their requirements, probably not just open source projects like phpBB, drupal or Wordpress, but also companies and services providing scripted configuration/installation and management, i.e. simplescripts, scriptacolous, installatron, but also webmin...

In other words, a multi-vendor approach would need to be taken - possibly through a 3rd party open source project that has contributors from all parties involved, especially from OSS projects like wordpress.

Arguably, keeping up with all these projects and maintaining working installers is very time-consuming and ultimately pretty expensive - so there IS something to be gained here, but we're basically asking for competitors to team up with each other, so I wouldn't hold my breath :-)

On the other hand, most standardization committees work exactly like that, i.e. with representatives from each involved party, trying to find common grounds.

I believe it would be possible, but it would be a different project - and so it must not take up too much time, so it would make sense to look into existing solutions tackling the same problem and determining if these can be adapted/customized or at least used as a foundation.

The most portable solution would be delegating things to some external web-service (LDAP-like), i.e. it would not be specific to any single OS and would work for everybody.

Supporting just php would not be very difficult, but other frameworks would need some more work.

Just to update this discussion: There's a feature request about supporting a CM (configuration management) system like puppet or chef: http://www.virtualmin.com/node/33698

Overall, that would seem to probably scale much better than anything discussed here, and installing/maintaining/updating and migrating scripts is pretty much what configuration management is all about in the first place, so it's probably pointless to reinvent the wheel by "cooking" our own CM system into webmin/virtualmin, and much better use of manpower to look for existing CM systems to could fill this gap.