more streamlined vhost migration between virtualmin instances (via clustered Webmin)

great to hear that this going to be worked on, I am sure that many people will find this really useful ! Besides, it would be great if such migrations could be based on "stages" that have preconditions, to ensure that stages are not automatically triggered without their preconditions being met.

A multi-stage migration could typically be implemented by adding a condition check as a recurring cron job, that's only deleted after the condition is true, which would in turn add a new condition check for the next stage and trigger any subsequent jobs.

For example, typically we'll only want to actually change the A record to the new server's IP if the TTL has been lowered accordingly, and if the glue records have been updated already.

So we could have a TTL precondition that runs dig to ensure that the TTL change has propagated (i.e. hourly checks), and only really enable dependant stages if that's the case, and show "red warnings" indicating that the migration cannot be recommended to proceed yet.

PS: thanks for the update, and thanks for being so responsive here!

as I mentioned in the extended custom fields issue, I believe that the migration (backup/restore) feature would benefit from being able to deal with installed scripts, even after the installation - i.e. to support migrations between multiple virtualmin instances, but also to move a whole virtualmin installation to a new server.

Even if supporting <a href="http://virtualmin.com/node/31971>migrations via mySQL replication is not deemed feasible currently, it would be good to formalize the configuration requirements for scripts, but also file system requirements such as chmod permissions.

If these details were known to webmin/virtualmin, it would become possible to largely automate the steps detailed at: http://www.virtualmin.com/documentation/system/backup-and-restore#using_...

For example, static and readonly folders/files could be safely rsync'ed while writable locations would need to be re-synced at the of the migration to ensure that the new server has all the latest data/files.

http://penguinbliss.wordpress.com/2013/10/30/mass-migration-between-virt...

A multi-stage approach could trigger the DNS/TTL changes and then register a cron job to regularly run dig for each update and check if the TTL update has propagated already (which should normally be the case within 24-48 hrs). Given that bind9/named may be running locally, and may thus overshadow other name servers, it would probably be a good to either run with +trace or use @ to specify a root name server.

I was looking through those install scripts and since I had to migrate a few vhosts recently, it occurred to me that it would be really useful if such a streamlined migration could support disabling the vhost via virtualmin, or ideally even by being aware of the "maintenance mode" supported by 99% of all web apps, and use that when doing a live migration

Related: - https://virtualmin.com/node/32008 - http://virtualmin.com/node/31977

Funnily enough we were just discussing adding this kind of feature today. I'll update this bug when it is done..

Based on your feedback here http://virtualmin.com/node/22867#comment-126833 I have come to believe that it's probably not such a good idea to implement this as a dedicated module or as a separate feature - given the powerful "custom commands" feature, this could be built on top of "custom commands" that are chained together - there are very few changes needed to make this possible, and putting a wizard-based wrapper around it would make this pretty straightforward to implement, even without being very familiar with Perl.

Basically, I am suggesting to postpone this and instead focus on extending the "custom commands" and "wizard" functionality in order to enable contributors to implement such things directly.

Given the challenges surrounding the theming updates, I believe that it would be better not to add more custom Perl code to the code base, and instead use this as an opportunity to sufficiently generalize "custom commands" and unify command handling.

Honestly, coding this from scratch would be pretty involved - but custom commands already support clustering and RPC, i.e. they can be easily run on other machines - currently, the destination server has to be specified during command creation, but once that's optionally made selectable at runtime, people could literally "live-migrate" virtualmin vhosts using a handful of custom commands presented in a simple multi-page wizard, based on the "post install wizard".

In complex code bases it usually pays off quickly to unify things like this - so please don't prioritize this, I am convinced that any "native" implementation would be inferior to leveraging custom commands for this, even if they may need to be extended a little.

Hi again,

I've been playing with this a bit lately and trying to come up with a workflow that would help minimize unnecessary downtime, while allowing the new server to be set up and configured already:

• foo.com's DNS entries are pointing at ns1.example.com, TTL set to 300 sec
• there's an A record that points to the old server (example.com being the new server)
• migrate the vhost to the new server, setting up a subdomain foo.example.com
• this is so that name-based vhosts can be reached and configured via the subdomain
• DNS for example.com is running on the new server

So, my question is this: When I edit the virtualmin vhost on the new server to change the domain from foo.example.com to foo.com - is the scripted logic smart enough to just update those existing entries or is that going to mess up my bind setup for the corresponding zone ?

Thank you

I was actually referring to using virtualmin/webmin for all steps:

1) have an existing webmin/bind master zone example.com 2) have a virtualmin vhost foo.example.com 3) using virtualmin "edit server" to rename foo.exampe.com and move it to foo.com

thanks for clarifying, so it will recognize that it doesn't need to create a new master zone for the foo.com TLD because it's already set up by bind/named, right ?

didn't work as expected, here's the corresponding error:

.. failed : Failed to start BIND : Starting named: Error in named configuration: /etc/named.conf:309: zone 'foo.com': already exists previous definition: /etc/named.conf:129 [FAILED]

and indeed, it ended up being there twice

thanks for the update - is this the feature currently to be found under "Transfer Virtual Server" ?

Just wondering, as we've been discussing a number of additional steps to make this a little more straightforward and future-proof, such as i.e. seeing this unified with the existing backup/migration tools and making the whole thing based on the existing wizard framework that sets up chained cron jobs for each step of the migration.

Would that be something worth exploring, or is this considered over-engineered ?

You mentioned at http://www.virtualmin.com/node/32025#comment-126939 and http://www.virtualmin.com/node/31971#comment-126611 that you were interested in supporting multi-stage migrations that may contain verification/validation logic before triggering subsequent steps (e.g. DNS updates, SQL backups, file transfers) - so I'd just like to know if this is still on the table - and if so, if you have any pointers/hints for people interested in implementing such functionality ?

Thanks

ok, thanks for clarifying - obviously you guys cannot be expected to implement each and every tiny feature, but how about implementing a modular "trigger & actions" system on top of cron jobs that would ultimately run a customizable check to trigger an action/custom command ?

Once something like that is in place, "we" (=end users!) could use this to easily augment virtualmin accordingly by having a handful of triggers (e.g. check DNS TTL) and actions that are presented using the existing wizard functionality.

I do think it would be worthwhile to extend virtualmin's infrastructure accordingly to allow fellow users to provide such features, without necessarily having to touch Perl code.

So, I am really only thinking in terms of building blocks, not actual end-user features - but just something analogous to the existing "custom commands" that could be chained together to model more complex workflows on top.

Honestly, looking at the functionality/code provided by webmin/virtualmin, it seems that most of these features are already in place in some way or form - apparently, those were just never really generalized to be reused on other contexts, but simply added over time without any refactoring/unification taking place (e.g. cron jobs, custom commands vs. bootup actions and templates - but also import/export and migrate vs. backup ).

This is probably a different discussion, as it's no longer specific to migrating vhosts - but obviously there's quite some overlapping functionality here that is still pretty use-case specific and unfortunately not sufficiently agnostic to the original module/plugin for which the code in question has been written.

With a little more refactoring/consistency, we'd be pretty much halfway there and could get rid of quite some redundant code, too while also making the underlying framework pretty flexible over time ?