Mass-restore failing: When saving restoring all sites at once, alias-sites try to get restored before aliased one

I'm trying to mass-move 50+ domains to a new server, which will then get the IP address of the old server, and hit a bug in mass-restore, while missing a function to do it easily:

As there is no "migration" utility for doing this painlessly in VirtualMin Pro (or did I miss something?), I last-resorted to do a "backup-almost-all-servers-TO-ONE-FILE" then to do a "restore-all-servers-from-that-one-file" strategy.

I selected backup-all-except 3 domains from a large account to keep backup-size and restore-time reasonable for all other domains.

The backup worked, but at restore, it hangs when trying to restore an alias-domain with the error below.

Cause is that the main domain didn't yet get restored, and it tries to restore the alias domain (as it was in the gz file before that one) !!!

I checked the sequence of backup: the domain alias gets backed-up at very end, which is correct. BUT: as virtualmin generates the .gz file at very end, the domains get re-sorted alphabetically in that file, instead of keeping the backup order, so that restore can happen.

So mass-restore failed as follows: blog.domain.com was alias from domain.com (name changed):

Restoring backup for virtual server blog.domain.com .. Restoring virtual server password, quota and other details .. .. done Re-creating records in DNS domain .. .. domain not found!

Re-starting DNS server .. .. done Applying web server configuration .. .. not running!

Re-loading Webmin .. .. done

.. failed! See the progress output above for the reason why.

Needless to say, with 8 gigs for 50+ servers, it is very slow, and when we finish exercising and do it for real, it will lead to about 2 hours downtime, which is a pain. Any suggestions to do it better are welcome. We can't move the disk as we are migrating Linux OS as well.

Status: 
Active

Comments

Correction: I'm now trying to continue restoring sites, and saw that the restore sequence is not alphabetical, main sites get restored first, but then sub-sites get restored in a strange order.

But to help reproduce the bug: the situation before backup was as follows:

main-server.com - subserver.com - blog.subserver.com alias

main-server.com got restored ok then blog.subserver.com alias tried to get restored BEFORE subserver.com

Also is there a "mass-delete" / "batch-delete" function to delete all virtual servers at once ? or do i need to do the 50+ one by one ?

Actually, if I restore over existing domains, will it just overwrite ?

I'd recommend using the command-line "virtualmin" command if you want to perform mass operations like this .. it can be used in a shell loop to restore or delete multiple domains easily.

Yes, that's a good point, still mudding through the migration, worked around the bug now.

However there is still a bug in here, restoring aliases before the aliased SUB-server when restoring from a SINGLE file (loaded from a SSH server...) ;)

Btw, not a bug, just a nuisance, the huge file gets loaded twice, once to see what to restore, and a second time to restore.

Yes, that is a bug .. did it actually hang though, or just skip the alias domain because the target didn't exist yet?

It stopped right away the big restore in the middle (see exact messages in my first report).

Ok .. this will be fixed in the next Virtualmin release, so that aliases are always restored last.

Cool thanks.

While at it, a little UI improvements which would have saved me reboots to stop a user error of me....:

in:

Backup Virtual Servers

Virtual servers

Servers to save: O All virtual servers O Only selected .. O All except ..

The list of servers is always clickable and servers are selectable, even with the PER DEFAULT "All virtual servers".

It would be really cool to at least disable (web-browser will grey it) and select-all in the list when "All virtual servers" is selected, including when page loads and it's selected by default.

My stupid user error:

Just selected a small site, and did FORGET ti change to "Only selected", leaving "All virtual servers" checked. The whole backup started, and as there are no "Cancel" button, i had to reboot server, to stop the backup...inelegant, but didn't want to loose an hour.

This is yet Another probably related bug:

To go around the first bug i did following using the full file:

First restored all servers and sub-servers, but unchecking aliases

Second restored all aliases, unchecking the other servers and sub-servers.

BUT:

The aliases of sub-servers DID NOT alias the right sub-server, but the first sub-server from the list of servers/subservers, not even belonging to the same server !

Fortunately we checked that and could fix them. BUT Definitely a third bug.

We also had an issue with secondary name servers updating, as when we did restore, we did it on anther address, we removed the secondary name servers to not affect continuation of operations of main server, then when putting them back, the domains got added to secondary, but NOT the A,MX and so on records.

We had to mood through the bind config files, restart bind a few times, add missing secondary servers and add the missing notify, also-notify, and allow-transfer bind records which were missing for all servers.

Looks like a fourth bug.

No hurry on this side though, we have one more server to migrate, but things are now pretty in control.

So total of 4 bugs (1 fixed) and 1 feature-request in here, all related.

I need to add for the wrong allocation of alias to subdomain:

  • it appears like aliased onto the correct sub-domain in the list on the left
  • even when I edit the server !
  • but when i go to Server configuration ---> Website options i get following error:

Error

Website options cannot be edited, as no Apache virtual host for aliasnamedomain.com port 80 was found!

Good idea on disabling that list .. I'll add this in the next release.

So for the alias issue, did you originally have a sub-server with an alias, and when it was restored the alias website no longer shows the same contents as the sub-server?

Correct. Actually it was displaying the default site, which didn't get restored right, and was a sub-server, which confused us too.

Actually, it looks like the Virtualmin entry was made correctly, but not the Apache entry, if i see the message above right. We simply restored several times from same backup file, ticking the sites as needed, but the aliases's apache-config didn't come through right, no idea why. Still left one site alias in that state, in case you wish info.

So we got to report "bug" #5 in here: - Default Site attribute (defined in, and any other Server configuration ---> Website options) is not backed up with the site, when you backup all sites.

And I need to report migration-nightmare bug #6 in here as well: - Any special settings in "Services -> DNS Domain" are NOT saved with the server. E.g. one of the domains defined a special IP address pointing to another email server (like gmail), and that setting got lost in the backup-restore process.

We are still fighting with postfix settings that we did backup and restore with webmin, and while incoming email works ok now, outgoing doesn't. No idea where to search anymore. Compared settings of old and new in all details, they are same: old accepted, new rejects as "relay not allowed".

It sounds to me like the restore process didn't actually restore some or all of the custom settings for some domains, leaving them instead with the defaults from the new system .. for example the "DNS Domain" settings.

If you re-restore a domain for which this happened, do the records get corrected? You can select just DNS settings on the restore form ..