[Solved] Migrating to new server - various problems

UPDATE

I noticed that several settings where completely not transferred correctly, despite the output. For example, ssl wasn't enabled on restored servers, wrong certs were pointed to, quotas were default instead used from the first server and many others. Despite fixing those settings, issues persist.

So I checked validate servers option and I got that there is no databases for each domains (or subdomains) which is not true... So I checked to edit those databases and got:

Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)

That's crazy and very concerning. So far automatic migration is a big failure. It suppose to be the easiest, quickest and the LEAST PROBLEMATIC way of migrating sites but it turned out to be complete crap. So it would see that the whole webmin/virtualmin backup options were also crap and none functioning properly.

The sites were transferred from ubuntu 16.04 server into vanilla ubuntu 16.04 server with freshly installed virtualmin/webmin so it all should go smoothly.

Is there a way to submit that as a bug or loads of bugs?

I tried to log in as root to mysql:

So I checked sockets: find / -type s

and I can't see any mysql.sock which worries me even more. So I did:

service mysql start Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

So I checked:

I'm not an advanced server user and rising difficulties start to add up. Do I see it correctly that apparmor blocked mysql? Why? What virtualimin did to trigger it? Shouldn't it be compatible with ubuntu server and thus handle such things automatically during restoring backups?

Last time I had a problem with connection to mysql as root, I had to reinstall the whole serwer since mysql cannot be re-installed with virtualmin - too many dependencies I mean, I was able to uninstall mysql but wasn't able to install because of virtualmin conflicts.

It turned out that also apache crashed but I was able to restore it when I removed virtual servers. Unfortunately mysql isn't staring and most probably I will have to reinstall the whole system again... and move sites manually :(.

Not sure what are you talking about. I'm new on that forum. What a detailed issue? I gave lot of details above, I may give more if needed but the thing is: there are so many of them that I don't know what is relevant so sorry if I didn't provide what you think of.

Tomorrow I will be reinstalling the system anyway and putting fresh virtualmin - again. This time I won't use aromatic backup/restore because it always damages mysql irreparably on the second server. I have no idea why, there are no mysql errors during backup/restore. But the issue is not only with mysql and many things are not identical with the old server so it seems the backup/restore tool is not reliable and just doesn't work.

As far I know they are similar. I can't tell they're identical, because on previous server system is set by provider, on second server system is installed by me (from Ubuntu 16.04 server iso) so there can be some package differences and some small setting differences. However, if backup/restore utility requires to have system really identical in every way with every small setting - this is ridiculous, it won't work most of the time. So once again what "identical" means in that context. Which settings must be identical? If such requirement is there, there must be some guidelines for it or scripts checking it. And yet, nothing like that exists so it seems, backup/restore utility is massively flawed and works only some small cases with idealized conditions and no one knows them for sure.

Before scratching the system, I have some idea to play with. Servers have different root passwords, which is normal and sane thing to have. However, mysql root is set by virtualmin to be default with root and password is encrypted within mysql db so there is no changing it manually. Maybe, MAYBE, virtualmin replaces that database and since setting "password the same as root" stays the same and it doesn't check password compatibility between tables (a big flaw in vurtualmin utility). My second guess is, myqsl unlike other processes needs this root password to launch it's process and the password must be provided by systemd. When this password doesn't match, launching mysql fails. That would make sense from security point of view (hackers process could impersonate systemd but without password it cannot access mysql and exploit it).

The fact that mysql works after restore and breaks after first reboot can suggest that I might be right or some similar protection is set. Maybe apparmor is checking that password after all?

I am just fishing... so I will play with root passwords and see if it helps. If it doesn't pan out, I'm out of ideas and my only option is to reinstall the system.

OK, here are the errors:

1. ** Adding new SSL virtual website .. .. certificate authority file is not valid : Data does not start with line -----BEGIN CERTIFICATE-----**

Really? This is automated tool, why is it doing it wrong? I don't see any fault from my side. ssl works on first server, it should work on second one but it doesn't because it fails in too many areas.

It does copy cert files correctly, so what is the problem with that error?

It doesn't set virtual servers as ssl websites (although that's the setting on original server). This is weird because I thought it just copies appache2 confs, so I am confused here, does the conf include ssl and other configs are forgotten or does it mess with apache2 confs (which it shouldn't) so they are not identical? Also, cert files seems to be copied to new server. Manually enabling ssl to a virtual server on a new server doesn't helps, ssl still produce issues.

2. Use of uninitialized value in string eq at /usr/share/webmin/virtualmin-awstats/virtual_feature.pl line 222.

I have no idea what it is about but AWstats works well after the restore and all past data are there.

3. Performing other Apache configuration .. .. configuration failed : Failed to copy /etc/php/7.0/cgi/php.ini to /home/site1/domains/test.site1.pl/etc/php7.0/php.ini : cp: cannot create regular file '/home/site1/domains/test.site1.pl/etc/php7.0/php.ini': No such file or directory

Why? This is nothing unusual, in fact having php is very basic thing on each website server. And why does it try to copy it from etc instead just restoring the ones that were originally there? Or maybe it did but it wanted to override with etc files and failed hence the error? And after checking those files are there so this is puzzling and probably has no negative effect since those are standard files and they are where they suppose to be.

4. Creating SSL certificate and private key .. .. SSL website failed! : Failed to open /home/site1/domains/test.site1.pl/ssl.cert.webmintmp.1393 : No such file or directory at ../web-lib-funcs.pl line 1433, line 1.

No idea what it is about but this is a test subdomain, it may not work so this doesn't matter really. I never checked it, because it needs refreshing anyway from the original.

That was it and yet problems are mainly with mysql. Weirdly, when I deleted virtual servers, it didn't fix mysql... yesterday. I went to sleep, check out server today and mysql seems to be working... But I can't log in to server with ssh, because users are not there and root somehow has problem with key, it asks for passphrase then asks for password which I give and it doesn't accept it so I am lost. Does restore changed root pass to for the one from the first server? I try this, nothing. Maybe it changed to the virtualmin original root password on the first server before I changed root password? But at this time I was blocked, probably by fail2ban and somehow I can't restore the connection. This is what happens when there are too many layers and I have no idea what is happening really, which ones are affected by virtualmin and which are not.

This is so time consuming, I am losing my patience. I would have reinstall system quicker.

Since mysql was working I was tempted to install virtual servers and move site manually without reinstalling whole system.

I noticed, that when deleting virtual servers, virtualmin failed to remove mysql databases and users associated with virtual server users... So I deleted them manually.

From there I created first virtual server, moved cert and needed files, fresh wordpress site works, all looks ok, ssl works now. However, it couldn't be so easy... Some new, unexpected problem showed up. I cannot successfully import database... because some tablespace exists... So I went to mysql directory of this table and found culprits - 2 files that were abandoned even when table suppose to be empty. I deleted those files and... still not out of the woods yet.

I get:

I don't get it. I do import on empty table. Directory of that database shows correctly only db.opt file. Now instead the tablespace exists, error changed into something else but with the same table.

UPDATE: Those deleted files come back... Sometimes I delete tables and they are gone and sometimes they stay. I get even info that the wp_domain_mapping already exists when previously there is nothing (no files in database directory). The weirdest thing is, THIS TABLE DOESN'T EXISTS in imported database. So why is it showing out of the thin air and blocking query? This is madness :(.

Thanks to help on #mysql irc channel I was able to resolve my problems.

Because deleting virtual servers left mess inside mysql and because I deleted manually database directories (couldn't do that by normal, proper means), I had to refresh the whole installation (clean all databases and initialize mysql anew, then renew mysql root connection) which fixed the issue. This was quite a hassle and I wouldn't done it by myself.

1. Renewing mysql

• stop mysql
• potentially do database dump (backup) if you want to restore some databases later
• delete all directories and files from /var/lib/mysql/
• issue a command as root: mysqld --initialize --user=mysql

This restores basic files and basic databases, but not webiste database.

2. Restoring root connection (Ubuntu 16.04)

• stop the mysql server.
• edit /etc/mysql/mysql.conf.d/mysqld.cnf, add "skip-grant-tables" under [mysqld]
• start mysqld
• connect with commands: "mysql -u root" and do mysql command "FLUSH PRIVILEGES;"
• if 5.7+, do mysql command: "ALTER USER 'root'@'localhost' IDENTIFIED BY 'newPassword';"
• If 5.6 or earlier, do: "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newPassword');"
• remove the added line from /etc/mysql/mysql.conf.d/mysqld.cnf
• restart mysqld.

3. Create website database and user for it, check in virtualmin if database is assigned to proper virtual server and if all credentials are correct
4. Import database for website (this time without error).

After all that, mysql works normally, I could import database and in general move site to that server. Everything works perfectly now as far I can tell.