Virtualmin Server Replication not working

Submitted by itinfra on Fri, 07/06/2018 - 08:13 Pro Licensee

Good day guys,

So i have two Virtualmin Servers, a primary and secondary and they both the /home directory shared between them via NFS and we do not have any SQL Databases yet. I'm using Cloudmin's Virtual Server Replication to replicate all other things except the home directory and Databases and scheduled it to run every hour. Unfortunately, this doesn't run. Whenever i do a manual replication (selecting Replicate Now), the page keeps loading for hours and nothing happens eventually. I've tried doing things on several occasions and i've been unable to replicate between the primary and secondary.

Thanks for your help.

Status: 
Active

Comments

Submitted by JamieCameron on Sat, 07/07/2018 - 14:21

When you try to run replication manually, does anything get logged to /var/webmin/miniserv.error ?

Submitted by itinfra on Mon, 07/09/2018 - 09:39 Pro Licensee

Nothing gets logged to /var/webmin/miniserv.error

The page just keeps loading endlessly and nothing happens afterwards.

Submitted by JamieCameron on Wed, 07/11/2018 - 00:21

To make any hidden error message more visible, try temporarily switching to the older Virtualmin Frame Theme and then re-try setting up replication.

Submitted by itinfra on Wed, 07/11/2018 - 10:01 Pro Licensee

I changed it to the theme specified and clicked Replicate Now afterwards. The screen gets stucked at "Transferring backups to destination systems .."

See the complete screen message below;

Starting replication from primary.example.com of Virtualmin settings ..

Finding source and destination systems .. .. found source primary.example.com and destination secondary.example.com Refreshing domains on source system .. .. done

Creating temporary directories .. .. done

Backing up 6 virtual servers on source system .. .. created backup of 115.61 kB

Transferring backups to destination systems ..

Submitted by JamieCameron on Wed, 07/11/2018 - 19:42

Is any thing logged to /var/log/messages or /var/log/auth.log on the remote system by sshd when the transfer happens? You should at least see an SSH connection from the source system.

Submitted by itinfra on Thu, 07/12/2018 - 08:18 Pro Licensee

Checking /var/log/auth.log, these are the logs that come in anytime i click replicate now. It establishes connection, accepts the password of the username then shows received disconnect from the IP address of the primary server;

Jul 12 13:26:35 secondary systemd-logind[1290]: New session c974 of user username. Jul 12 13:26:36 secondary sshd[16776]: Accepted password for username from **.***.***.*** port 55344 ssh2 Jul 12 13:26:37 secondary sshd[16776]: pam_unix(sshd:session): session opened for user username by (uid=0) Jul 12 13:26:37 secondary systemd-logind[1290]: New session 2175 of user username. Jul 12 13:26:37 secondary sshd[16817]: Received disconnect from **.***.***.*** port 55344:11: disconnected by user Jul 12 13:26:37 secondary sshd[16817]: Disconnected from **.***.***.*** port 55344 Jul 12 13:26:37 secondary sshd[16776]: pam_unix(sshd:session): session closed for user username Jul 12 13:26:37 secondary systemd-logind[1290]: Removed session 2175.

Submitted by JamieCameron on Sun, 07/15/2018 - 01:53

Can you check if scp-ing a large file from the source to the destination systems works OK? Because that's all that should be happening at the transfer stage.

Submitted by itinfra on Mon, 07/16/2018 - 16:03 Pro Licensee

I tried scp-ing a large file from the Primary to the Secondary and it works fine... i wonder what the problem could be.

Submitted by itinfra on Wed, 07/18/2018 - 06:42 Pro Licensee

Hello Jamie,

Still no headway with this.... still stuck at "Transferring backups to destination systems..." I looked at auth.log again during the replication process and i got this from it;

Jul 18 11:40:56 secondary server sshd[19367]: Received disconnect from "primary server IP" port 59194:11: disconnected by user

Jul 18 11:40:56 secondary server sshd[19367]: Disconnected from "primary server IP" port 59194

Jul 18 11:40:56 secondary server sshd[19326]: pam_unix(sshd:session): session closed for user username

Jul 18 11:40:56 secondary server systemd-logind[1290]: Removed session 4330.

Why does it receive a disconnect from the Primary/Source Server and closes the session while replication is still in progress? Could this be the reason for the issue?

Regards.

Submitted by JamieCameron on Fri, 07/20/2018 - 00:24

That could be part of the problem..

Is the cloudmin master the primary or secondary system in this setup, or neither? Also, is authentication done using SSL keys, or password?

Submitted by itinfra on Fri, 07/20/2018 - 09:23 Pro Licensee

The cloudmin master is a seperate system, its neither. Authentication is done using password.

Submitted by JamieCameron on Fri, 07/20/2018 - 13:38

If you SSH into the master as root and use the shell API command cloudmin transfer-file , can you copy a file from the primary to the secondary system?

Submitted by itinfra on Mon, 07/23/2018 - 10:00 Pro Licensee

I did this and tried to copy a file from the Primary to the Secondary, the screen got stuck at copying for a very long time until i did a Ctrl-C to stop it with the same events we were seeing earlier logged in auth.log. I then decided to do this in the opposite direction (from the secondary to the Primary) and it was still the same thing. I then made another attempt to copy some files again from the Primary to the Secondary and it was successfully copied.

I then tried the same thing both ways with Virtual Server Replication but its still stuck at "Transferring backups to destination systems".

Submitted by itinfra on Mon, 07/23/2018 - 11:37 Pro Licensee

Hi Jamie,

I permitted root login on the Primary and Secondary Servers and re-added them to Cloudmin with root. I then tried Replication afterwards and it now completes. Seems that issue has now been resolved.

But i noticed this; i changed the password of a Virtual Server admin on the Primary but then tried to login to the secondary with the new password after replication but i couldn't (Login failed). Isn't the password change replicated as well?

Thanks

Submitted by JamieCameron on Mon, 07/23/2018 - 15:21

Oh, so were you using a sudo-capable login before to do the transfers?

Submitted by itinfra on Tue, 07/24/2018 - 03:13 Pro Licensee

Yes i was... I thought you knew.. should have stated that earlier though. So Isn't password change also replicated?

Submitted by itinfra on Fri, 07/27/2018 - 07:50 Pro Licensee

Hi Jamie,

I'm still looking forward to your response regarding my inquiry above.....

Thanks.