OBJECTIVE>> To setup a dual-node HA cluster to provide uninterrupted and/or load-balanced hosting for KVM instances managed by Cloudmin Pro and mirrored to the second host using DRBD
HARDWARE>> Nodes: 2X Dual Xeon 16GB Internet connectivity: 2X Internet leased lines(different ISPs) load-balanced through a dual-wan router Virtualization: KVM Replication: DRBD/Pacemaker Replicated Drive: /KVM
NETWORKING: Both internet lines connected to Dual-Wan router. Both nodes; eth0s connected to router for private IP assignment and Internet access Both nodes; eth1s connected to each other for DRBD/Corosync/GFS2 communication
SOFTWARE: Both nodes; CentOS 6.5 installed Both nodes; Cloudmin Pro installed Both nodes; DRBD configured in Dual-Primary mode to replicate /KVM on a GFS2 filesystem
CLOUDMIN CONFIGURATION: node2 as Cloudmin replica of node1 node1 and node2 part of host failover group ("System Operations -> Force Failover" OPTION MISSING)
TRIAL: Creating the first Instance on node1: Test1and testing failover to secondary host: node2
Following the above setup, we consider that Cloudmin will assume the existence of the test1 image file on the local drive /KVM on node2 when it is created in node1 and switch to node2:Test1 when node1 fails.
Instead, what happens is that test1 goes down as soon as node1 is shutdown.
QUESTIONS: 1. What are we doing wrong here? 2. Is the topology and configurations correct? 3. Are we missing steps? 4. Is there anything additional to be done? 5. Is there an alternative/easier/more logical approach to achieving the objective?
Thanking you, Surjo Banerjee
Comments
Submitted by surjo on Sat, 08/09/2014 - 14:38 Pro Licensee Comment #1
Submitted by JamieCameron on Sun, 08/10/2014 - 17:26 Comment #2
Image mirroring using DRDB isn't something we have tested unfortunately - rather, the common setup is to use an NFS or iSCSI mount from a central storage system by two VM hosts. Cloudmin can then do automatic failover if one host goes down.
Submitted by surjo on Mon, 08/11/2014 - 06:07 Pro Licensee Comment #3
Thank you for your reply Jamie.
-- Ok, please give me a hint on the following scenerio then..
If host#1 is reading the image from its local storage: /kvm/test1.img and it fails, assuming failover is setup the right way, will Cloudmin Pro then switch to host#2 and look for the /kvm/test1.img on host#2's local storage? This is logical if Cloudmin is simply replicating/moving the test1 instance (except test1.img) from host#1 to host#2. And this should solve our problem.
Please help us out. We have spent more than six months on research already and according to all our trials and reviewing every piece of documentation available, Cloudmin seems to be our best chance. We have been using Webmin/Virtualmin for more than a year now, so we are quite familiar and very comfortable with your products. The launch of our healthcare application is held up because of not being able to put together a redundant hosting platform.
We feel we are very close to building this HA cluster using a combination of DRBD, KVM, GFS2 and Cloudmin Pro and I am willing to share the entire approach if it helps you in any way. At this point all the major elements are functioning as expected, just a bit of fine tuning will get us on track here.
Your guidance is highly appreciated! :-)
Awaiting your response..
Thanking you, Surjo.
Submitted by surjo on Mon, 08/11/2014 - 09:38 Pro Licensee Comment #4
Jamie,
Status: Failover not triggered automatically and cannot find a way to trigger manual failover in Cloudmin.
https://www.virtualmin.com/documentation/cloudmin/vm/failover
Ex#1 "If you want Cloudmin to automatically perform failovers, set the Failover group enabled? to automatic mode. Otherwise you can select manual mode to trigger failovers manually when you detect a host system has gone down."
My Cloudmin version has three options: Automatic and manual/Manual Only/Disabled Automatic and Manual: No migration action observed on Cloudmin after forcing node1 to fail. Test1 has been down on node1 for more than 30 minutes and hasn't migrated to node2.
Ex#2 Setting Failover to Manual: No way to force failover: System Operations -> Force Failover is not an option seen as opposed to to the documentation: "If you have set a failover group to manual mode, Cloudmin will not automatically move virtual systems off down hosts in the group. However, you can force a failover at System Operations -> Force Failover. "
Remaining settings:
SAVE
Now I am removing Cloudmin from node1 and node2 to find out if that helps.
Please advise..,
Surjo.
Submitted by andreychek on Mon, 08/11/2014 - 13:23 Comment #5
I read your most recent post after I typed the following text, and realized you attempted my first suggestion already. We may want to wait for Jamie's advice, as it sounds like you're not seeing the "Force Failover" option. However, I did want to offer what I typed up, even if some of it is no longer relevant to you --
In the scenario Jamie described, both host #1 and host #2 would be sharing the same storage, where the Virtual Machine images would be stored on a shared NFS or iSCSI mount.
The exact details of that scenario are described here:
https://www.virtualmin.com/documentation/cloudmin/vm/failover
There's an alternative that Cloudmin supports. Rather than using shared storage as described above, Cloudmin could also monitor a given server, and if it goes down, it could change the DNS records so that a second server becomes the live server.
That scenario is described here:
https://www.virtualmin.com/documentation/cloudmin/vm/roundrobin
Would either of those do what you're after? The second option is interesting in that it would also allow you to do load balancing, if you wanted.
Submitted by surjo on Mon, 08/11/2014 - 14:35 Pro Licensee Comment #6
Thanks Andrey,
I have already looked at Jamie's post that mentions those two options. The first one as you are aware, hasn't been successful for us because the Fail-over doesn't switch anything. Ideally, fail-over is something we would want to get working because of the DRBD limitations that may not handle a load balanced dual-primary very well.
We will need to spend some time with DNS RoundRobin and DRBD to work stably in Dual-Primary mode. Proxmox seems to work with DRBD if shared storage isn't available.
Could you please suggest? As far as I have read, Cloudmin is superior to Proxmox in many ways and I would like to stick with it because of the seamless integration with VM level virtualmin/webmin deployments that we have to manage.
I do believe Cloudmin would do it considering it is recognizing the local images during migration of VMs. Only that Fail-over doesn't seem to be active for some reason as I described in my previous post.
Please advise, I think we are very very close..
Regards, Surjo.
Submitted by JamieCameron on Tue, 08/12/2014 - 02:19 Comment #7
Regarding the "force failover" option, did you select the VM from the left menu, or the host system? It only appears when a VM is chosen, AND the host system for the VM is part of a failover group.
If this still doesn't help, I would be glad to login to your Cloudmin system to see what is going wrong - since your setup is a bit unusual, there may be bugs in the failover process.