Migration of Proxmox Server and switching over to Cloudmin - Hetzner Online root server

Hello Virtualmin/ cloudmin team

I want to get your concurrence on the topic of migrating to Cloudmin from Proxmox. below are some points i want to know your opinion.

  1. Which operating System to use for cloudmin: I have add quiet a bit of trouble working with hetzner servers, i am not complaing about them but root servers do have any support from those guys. at the same time i cannot afford their dedicated service as the cost of that is not feasible. The networking setting in particular are a nightmare with them, Having said that I have not been able to properly confidently configure CEntos. were as debian and ubuntu have been manage.. not perfect though. So could anyone of recomend or share your experiences with OSes for Cloudmin host.
  2. Has anyone successfully configured cloudmin on a hetnzer root server with success? anyone out there who has done this already?
  3. Will Virtualmin team be able to help/ guidance with the networking issues if i go ahead with cloudmin replacing my proxmox servers presently?
  4. The main reason i want to do this is due to the fact that we are pretty much invested with Virtualmin based servers.
  5. Just for the background, we are a small startup offering web solutions and to some extent offer webhosting on our VPS servers.

Thank you for anyone commenting with +ve answers or guidance..

Thanks, Rohit www.interstellarconsulting.com

Status: 
Closed (fixed)

Comments

Howdy -- Cloudmin is supported on CentOS, Ubuntu LTS, and Debian.

Sorry, we don't have any experience using Hetnzer. I'm not sure if any customers are using that.

Sure, if you're using Cloudmin Pro, you are welcome to ask about any issues you run into using the support tracker here, including networking issues. We're happy to help.

If you decide to use Cloudmin GPL instead, you can still get help using the Forums, which we monitor (along with lots of folks in the community!)

Can you describe your current setup, and what type of VPS's you are using with Proxmox?

I put this issue out in public so may be there are some Hetzner running folks out there..

I have 2 bare metals servers with hetzner both are having a combination of Containers and KVM VPS.
initially when i started of with them, i was planning to run them in cluster but apparently there are challenges there in terms of managing the ip address their failovers. presently i am middle of migration of all VPS to one of them and then experimenting on one of the bare metal 32GB ram, i7 3 rd gen, 3tb in software raid, etc etc.
i have also 3 seperate ips routed to this server and a /29 ip subnet.. so plenty of VPS with dedicated ips i can run.. Some of the VPS run virtualmin on which our customers websites are hosted, some are for internal company use and some are in place for testing, etc... i do have cloudmin running at my home but the networking of that is simple as i had to just configure it to use ips from router.. but with Hetzner and IP routing along with Centos, this is a daunting task.. i have spend weekend and weekends struggling with this,, proxmox is not bad and is pretty much stable but i would like to have a central cloudmin instance where i could manage all my Virtualmin based vps centrally..

Anyway i not being specific here.. but wanted to just put this out there.. and may get a discussion going,

Lastly, incase hypothetically lets say i go with centos and cloudmin .. would you be able to login to that server provided i give you access to help me sort this potential block of the configuring the networking.. ?

Understood about wanting a central Cloudmin installation for managing things.

Do note that if you had things setup the way you wanted on Proxmox, but also wanted to be able to use Cloudmin to assist in managing your various installations, you could use Cloudmin Connect. It doesn't have any of the VPS features, but it can help you manage all the servers, access all the Virtualmin control panels, and such.

As far as networking support is concerned --

We're certainly here to help.

We'd first ask you some troubleshooting questions here first, have you provide the current networking information, and so forth.

Networking issues can be tricky to troubleshoot remotely if it's causing the system to be offline.

But if we did some troubleshooting here and didn't sort out the issue, and the system is remotely accessible even with those networking issues, then sure, we could log in to take a look.

thanks for the support, Cloudmin connect i understand. but i dont want to put for that service seperately just to manage my virtualmins. i do own 2 Lifetime licenses from you guys.. the idea moving away from Proxmox is that we have extensive knowledge of one group of products in this case virtualmin / webmin / cloudmin?

Coming to my baremetal, loosing connectivity is not of a concern as we can easily recover from that. they provide excellent rescue system where i can connect to the disconnected box. mount the partitions and revert back the connection changes and we can start again.

Coming to a question though, which OS would you prefer technically and some points why.. between Centos / Debian / ubuntu. our production environment for 2 servers is ubuntu and we have no real issues with the 14.04 OS. rather stable i have to admit.

Thanks, Rohit

CentOS, Ubuntu, and Debian would all work well.

Most folks are using either CentOS or Ubuntu.

If you're most familiar with Ubuntu 14.04, I'd recommend going with that.

Ok, I have been busy last days to backup and migrate our existing customers to another baremetal based proxmox server and running VPS and virtualmin. This task is now completed, I would not like to get your concurrence to eventually move towards cloudmin as our virtualization layer. Will give some update on the situation.

Baremetal Server1 running proxmox: had 3-4 VPS servers for different purposes, main being one hosting my domain, all other have been transfered. http://www.intodns.com/interstellarconsulting.com

Baremetal Server2 running proxmox: This is hosting all our customers servers, The nameservers are NS1, NS2 on which client domains are pointing to.

Plan1 migrate: is to migrate my own domain which is on Server1, to one of the IP address on server2. ns4.interstellarconsulting.com ['144.76.93.92'] [TTL=38400] Question1: i had planned to transfer via virtualmin build in function from server 1 to 2, then simply bring down server1, i noticed that the nameservers were still resolving to the server 1 IP address.. and hence my website on server2 was not seen on the internet.. I am hoping to correct here, is this due to the TTL time 38400 and i should reduce this to 60 secs.. before the transfer?

Plan2 install: If Plan1 is executed and working as intended, i should not have any downtime.. and with this i plan to wipe server1 and install Ubuntu 14.04 with cloudmin, i know you guys recommend Centos, but with Hertzner routed IP address, Centos does play well. i have had quiet a bit of pain in this area.. Question2: Will i be ok to go with Ubuntu 14.04 and install cloudmin on it_? i am more confortable with Debain based Distro i guess.

If you you help with the above query, we will proceed and then tackle the next bit of issues.. i am allow me 20 days of time and have installed cloudmin on server at home and played with a bit.

Thanks Fingers crossed, Rohit.

Title: OS for cloudmin with Hetzner Germany provider - Replacing promox » Migration of Server and switching over to Cloudmin

Howdy -- yeah, if you're able to shorten the TTL maybe 12 to 24 hours before the transfer, that will greatly help with most DNS caching. What you shorten it to is up to you -- some choose 10 or 20 minutes, others choose a minute or two. You wouldn't want to make it too short, as it will cause access to your websites to appear slower (since more DNS queries are taking place). Setting it to maybe 10 minutes or so is common. If you aren't worried about the extra time to access your websites, and your DNS server can handle it, setting it to a minute or two is fine though.

As far as your distro is concerned -- Joe and Jamie like CentOS. I much prefer Ubuntu though, and that'll work just fine for this.

A note regarding the provider you're looking to use --

We've had a bear of a time trying to setup IP addresses for Virtual Machines using Hetzner.

I know it can be done, and I've seen folks say they've gotten it to work... but it can be pretty tricky to setup, and we personally don't know how to do it :-)

So if you haven't already, I'd highly encourage you to try setting up a Cloudmin based Virtual Machine as a test, just to make sure you're able to get all that working before you begin you're migration.

Feel free to let us know if you have any additional questions!

Yeah i have been working with debian / Ubuntu based installs on Hertzner provider and have managed to install/ configure, atleast Proxmox.

Coming to Cloudmin, i plan to test it first before i am going to make it a Production system; i will keep this thread updated with the progress. Please keep tuned.

I tried to setup cloudmin on hetzner with 5 separate IPs about a year back. I was unable to setup cloudmin GPL to work as required. But I was also able to setup proxmox using hetzner's guide for KVM. In Proxmox, I only have to request separate MAC address for each IP and then give that mac address to KVM machine while making and the IP is allocated by DHCP.

If you are able to get it working using separate IPs or subnet, kindly write about it somewhere (some kinda tutorial). Hetzner is really tough with networking.

Yes, note that it is possible to set the MAC address to use for each individual Virtual Machine in Cloudmin.

That may end up being required to get things working there.

Yeah, setting up the IP address via MAC is not the most efficient way to work with Hertzner.. it always comes down to assigning the MAC via Hertzner portal.. which is not ideal... Last yr that was my setup.. then around april this yr i did quiet a bit of digging and found a way to work with routed method described in their proxmox documentation.

This setup was tested in the production environment and my server2 is running with this config.

I plan to install Cloudmin on ubuntu, because apparently with Hertzner its easier to config network on debian based distro as compared to Centos. having said that.. i have even managed to configure Centos with routed IP with hertzner.. but it was a headache... i dont want to do that again.

I will document every thing in here for my reference later and for others too.

All that sounds great, thanks for letting us know!

Ok, we have migrated our root domain to another instance of KVM on server2 proxmox. will do some tests on server1 for my learnings.. and will start the cloudmin install over the weekend. should i begin with free KVM version or already get the license ?

If you are planning on only using one server for everything, you could get away with Cloudmin GPL.

However, upgrading from Cloudmin GPL to Cloudmin Pro isn't as seamless as it is with Virtualmin... so you might save yourself a little hassle by going for Cloudmin Pro if that's the eventual goal.

Remember that if Cloudmin Pro doesn't do what you want, just let us know within 30 days and we'll refund your order.

ok got my self a annual license.. thnx., will let you how it went.

may be a bug, the script which got assigned to me.. did not have the key in place.. it was failing to install.. i manually added the license key.. onto the script to proceed with the install, btw.. you should get paypal somehow in your billing portal.. having CCards linked, does not feel safe.

We can manually setup Paypal for folks looking to pay yearly.

I'll look into the issue with the install script and the license key. We've sometimes seen that happen if the key were to somehow get clipped when pasting the download command into a terminal. The key gets inserted because it's passed in as a URL parameter... but if that parameter gets cut off somehow, that can cause what you're seeing there.

But we'll make sure there isn't a deeper issue going on... thanks for the heads up!

Ok gentlemen, this morning cloudmin on ubuntu was installed. networking was configured using same method similar to proxmox installation, routed method. Bridge was configured for KVM, downloaded an image for ubuntu 64bit KVM. installed.. to test.. and wallah.. the KVM additional IP address is pingable from outside.. and it can also ping the host.

I will need help configuring cloudmin, as i have in the past only played with it at home. @ Andre - I can give you access, if you can help me give me some tips or help configur further.

I think this is the first time this is done on the internet, someone managed cloudmin to install on hertzner root server. i have some 2 additional ips.. which i will configure for LXC and may be one for dockers.. beep beep :).. and will slowly move towards migration our infrastructure from Proxmox.

Incase of questions, i am going to leave my email.. rohit@interstellarconsulting.com

a bit excited :)

Title: Migration of Server and switching over to Cloudmin » Migration of Proxmox Server and switching over to Cloudmin - Hertzner Online root server
Title: Migration of Proxmox Server and switching over to Cloudmin - Hertzner Online root server » Migration of Proxmox Server and switching over to Cloudmin - Hetzner Online root server

It sounds like things are coming along pretty well!

Regarding configuring Cloudmin, what is it that you'd like a hand with?

  1. LXC container setup. i have installed the packages and add LXC shows up under create system.. but there is no save/create button. Note i havent configured the network as described in the documentation for LXC.. as it might conflict with the KVM network setup on the host cloudmin server.
  2. configure DNS roundrobin. i need to read up on this one.
  3. Was planning to have redundant virtualmin systems, need help with the best possible setup on this.
  4. Central CLAM Antivirus/ Antispam, for all my virtualmin servers.
  5. Can you take access to the server, might be easier for you to recomend.. what is needed.. this is going to be a test server for atleast 15 days.. before i use it for production.

give me email address and i will send you the login details, Rohit

If you run into a problem that we can't resolve here, we can certainly log in to take a look. What I did though is mark your request as private... if you like, you can place login details here, where all three of us can see it. Then, if you run into any problems, Jamie, Joe, or I could login to take a look.

It sounds like you have a number of things you'd like to setup there though, and perhaps a good place to start would be to work through those one at a time.

I'll ask Jamie regarding whether LXC should work on a system with KVM. I thought they would, but maybe you are seeing some sort of incompability there. Note that you can have as many different host systems as you like, so if you have another server available to you, you could configure it as a separate host for use purely with LXC. At that point, both your KVM and LXC hosts could be managed from a single Cloudmin installation.

You may have seen this already, but information on setting up Roundrobin DNS is here:

https://www.virtualmin.com/documentation/cloudmin/vm/roundrobin

For redundancy, you can use either that, or this shared storage redundancy described here:

https://www.virtualmin.com/documentation/cloudmin/vm/failover

Regarding migrations and networking -- we're certainly happy to help, though we'd highly recommend migrating those in a way that would allow you to test them first, before making them live. We have less availability on the weekends than we do during the business day, but even then, we'd still recomend making sure things are working before making the system live.

Regarding central anti-spam -- it sounds like you may want Cloudmin Services:

https://www.virtualmin.com/documentation/provisioning/intro

https://www.virtualmin.com/documentation/provisioning/hosts

Does that sound like it'd do what you're after?

Okay, so that briefly goes over the issues you mentioned. Which one would you like to start with?

let me think and plan with the info you have provided.. first i want to definitely solve the LXC container issue, Meanwhile i have create a clone of root for you .. feel free to look around.. going to get some sleep and will try out some stuff tomorrow.

Link: https://cloud1.interstellarconsulting.com:10000 user:virtualmin pass:virt@123

FYI, it should be possible to run LXC and KVM instances on the same host, or even run LXC inside a KVM instance (I do this myself).

LXC inside of KVM. i haven't considered that.. what are real world benefits of this?

Ok, I just realized that.. Cloudmin KVM images cannot be snapshot-ted, my hardware config has 2 harddrives on software raid1. and i have partitions like /swap /root / boot /home on EXT 4 file system.

In the Proxmox setup i had the same setup, but i was able to take snapshot of the KVMs. how are they managing this? i am not comfortable with LVMs, as i havent really used this.. especially from recovery point of view.. i know how to mount EXT and manipulate files for examples.. if incase i screwed up the network setup.. any suggestions should i really move over to LVMs?

What error do you get when you try to snapshot a KVM instance?

Regarding LXC inside KVM, I personally use this for LXC testing so that I don't need to run a large number of physical systems.

Hi Jamie,

There is no error, just this message " Disk Snapshots test.localhost
This system does not support snapshots : Only systems using LVM for disk images can be snapshotted "

with regards to the LXC issue, I tried to create a container via command line and that works.. but i did not spend time to start it, as it was erroring out. That means the system is able to manage to create LXC containers via the terminal, but the create option via the cloudmin panel is not activated.

Actually, if you logon.. you will see what i mean. After the Advance option display on the LXC create page.. it ends abrutly.. there is no CREATE button. Need you to look into this one, the login details where posted in post 25 above.

Rohit

What error are you seeing when trying that on the command line? They may each be the same problem.

as suspected, looks like its looking for a LXC bridge

lxc-start: conf.c: instantiate_veth: 3105 failed to attach 'veth1H939I' to the bridge 'lxcbr0': No such device lxc-start: conf.c: lxc_create_network: 3388 failed to create netdev lxc-start: start.c: lxc_spawn: 856 failed to create the network lxc-start: start.c: __lxc_start: 1121 failed to spawn 'u1' lxc-start: lxc_start.c: main: 341 The container failed to start. lxc-start: lxc_start.c: main: 345 Additional information can be obtained by setting the --logfile and --log

Let me create a bridge br1 and see if it helps..

ok i added additional bridge lxcbr0 and was able to create the container manually via the terminal, was able to attach to it and seems to be working. I had to setup libvirt though, https://help.ubuntu.com/lts/serverguide/libvirt.html But still the cloudmin gui does not have the option to save or create a LXC container, after advanced options.. there is no create button under create LXC containers..

Regarding KVM snapshot, could Jamie confirm.. that without the disks using LVMs, this is not possible with cloudmin. as compared to proxmox, where without LVM, just traditional EXT partitioning.. somehow they are able to offer snapshots.. may proprietary..

That's correct, Cloudmin only supports taking snapshots of running VMs when the disk image is on LVM. We don't know of any way to snapshot a regular file under Linux.

Thanks Jamie, I will do some research to find out how proxmox manages this..

meanwhile, I am still waiting for the resolution/help with the network LXC issue @ andreychek.. all details posted previously, along with the login pass details.. please have a look.

Jamie, do you know what might be preventing the "Create System" button from showing up for LXC, although it is available for KVM?

He's able to create LXC instances from the command line using the API.

That could happen if no LXC hosts have been registered, at Host Systems -> LXC Host Systems.

Jamie, Same host which has seen registered for KVM is also registered as LXC host, actually you dont get the option of creating LXC containers until you register... so this is not the problem.

Oh, so you mean that the page appears to create an LXC container, but the Create button is missing from the bottom?

Yes.., waiting for your inputs on this one. you could ask me to install cloudmin again, it might fix the issue.. but then you get not get a better chance to fix it :)

Any Update for me ?

Out of curiosity, do you see the same issue when using the Virtualmin Framed Theme?

I'm curious if this is a theme issue, or a Cloudmin issue.

No, This is not the issue with the framework.. I already tried that intially. Why dont you login and see it yourself? we have been exchanging messages on the same topic since a week now?

Okay, so just to clarify, the "Create" button also doesn't show up when using the Virtualmin Framed Theme?

Jamie, do you have any other thoughts as to what might be causing that?

Does anything get logged to /var/webmin/miniserv.error when you open the VM creation page?

I suspect that something is causing the page to error out before the Create button is displayed.

Ok, I did check this but there was no error. what i did was went to KVM create page and pressed the create button with any settings.. and check the error log. it was there " Error: Error: Missing or invalid hostname for new system "

but there is no error log when i go into LXC Create option, Note LXC add option is just fine.

Ok, out of curosity, I tried to re-register the LXC host. it does not allow me. ""Failed to save LXC host : This system cannot be de-registered as a host as it is still in use by one or more virtual systems""

i have meanwhile restarted the host, but it has not helped.. same issue no create option. at this stage i am thinking that i might need to reinstalled LXC packages?

Any chance we could login to your Cloudmin master system remotely to see what's going on?

Ok, it's as I suspect - the page is erroring out before fully loading.

Any chance we could also get root SSH access to your system?

sure, the root pass is now changed to virt@123, thanks for looking into this.

Ok, I see the error, and have patched it on your system - can you try creating an LXC container now?

Ok Jamie, I downloaded an image for LXC and set a guest on the system. works fine. couple of questions,

1.The patch you have done will that be a bug fix? 2. Also the the networking on the guest LXC seems to be not functioning, mainly the gateway. As with the provider Hetzner, all the network config is of the type routed and so the host machine ip is the gateway for all the guest systems.. KVM guests are working fine as in they pick up the gateway as per the IP pools i have defined. any thoughts? 3. The LXC images on your repository are not the best, do you plan to update this?

  1. Yes, this will be included in the next Cloudmin release.

  2. What gateway are your LXC containers being assigned? The host system, or the next hop that the host uses?

  3. Yeah, we need to update these. Unfortunately there is no good public repository of LXC images (unlike say Docker)

What gateway are your LXC containers being assigned? A. the next hop that the host uses, for the routed method to work the gateway should be the host IP. This is working fine for the KVM, the gateway(Host ip) gets assigned and works out of the box.

Regarding LXC templates: Can you try to include turnkey images from here, this is available in proxmox by default. i think Cloudmin can somehow also reference this. https://www.turnkeylinux.org/blog/announcing-turnkey-lxc for LXC OSes, you could take them initially from here. http://download.proxmox.com/images/system/

You should be able to configure the gateway IP as part of the allocation ranges for LXC instances, on the "LXC Host Systems" page.

Hi Jamie,

I thought that i would find that option here, but i already tried this... there is no network settings available here. i am only able to provide the ip allocation range.. but the settings are missing as compared to KVM.

Jamie, I verified that as well on his system... in Host Systems -> LXC Host Systems, there isn't an option to set a gateway.

The only network related settings on that screen are "IP address allocation ranges" and "Bridge on host system for LXC interfaces".

Oh right, if you are using shared IP allocation ranges, you would need to define the range and gateway on the Host Systems -> IP Addresses -> Common IPv4 Pools page.

Hi Jamie,

I had done this already.. the default ip address as gateway is defined.. and works perfect for the KVM instances.. but for LXC instance.. it does not, for some reason its using the real gateway.. which will not work in the case of Hetzner way of route networking..

You may have a look, the login details are still the same...

Ok, I think I see the bug here now ... will update this ticket when it's fixed.

Ok Jamie, Meanwhile i wanted to also test dockers and so first i installed the docker engine. using this link, as by default the gui was error out that docker is missing..
https://docs.docker.com/engine/installation/linux/ubuntulinux/
The i was able to download couple of docker images ubuntu and tried to run.. the gui ran for some time but then it failed, i then via the command line was able to start stop those dockers.. but apparently the gui seems to be not behaving well.. i noticed that that the advanced option in the docker host also does not expand.. i might only think of docker to get activesync capable mail server.. but this is something which might interest me to also host..
Open VZ - No since i need a kernel change. LXC -- defintely as they are light and have worked pretty well in the past. KVM -- this is my main go to VPS option, secure and works. as a general comment, cloudmin needs lot more work to be able to be a go to system as compared.. the biggest qualm i feel is the gui.. for example if a task is running for a long, the browser sometimes goes into an infinite loop.. there is no status updates just dots.. this as compared to proxmox, their status updates are on the seperate window in their gui..and does not hi jack the whole browser tab.. ofcourse you can have different tabs for cloudmin/virtualmin, etc.. but i feel this is a main factor..

I am sorry if i sound a bit -ve, but i am just relaying info as a end user. I will wait for your update on the LXC network.. and then i want to install the whole system from scratch with LVM to support snapshots.. it will take some months to migrate my Production Enivronment to cloudmin.. but i will continue to test with you guys.. i am learning a lot :)

I'm not sure I'd recommend Docker for your use case - it's a very very lightweight container system that is really only useful for deploying apps. Much lighter than LXC, but with almost no isolation.

BTW, the fix to support the LXC default gateway properly has been implemented.

yeah i was only evaluating Dockers and i guess trying to learn to deploy it, may be offloading some of the functions to it in the future.. thanks for the LXC fix.. after which i will install the whole OS again, cloudmin with LVM..

When do u plan to release the next cloudmin update jamie?

I believe the last release was a few months ago, so we may be due for a new one here soon!

This week. I was hoping to complete support for Docker volumes first, but the API Docker offers for that is so limited that I may have to skip it.

Hi With the latest version 9.1 the LXC creation issues has been taken careoff.. that means out of the box i am able to deploy the LXC container. but the gateway ip address i am not able to setup and hence the container is not able to connect..

I'm glad to hear that LXC is working better, though we'd certainly like to get the networking up and running as well.

What is happening when you attempt to setup a gateway IP address? Are you receiving an error of some sort?

No, If you remember the issue was that there was no way to set the gateway for LXC networking.. similar to the KVMs. and Jamie mentioned that he will fix it in the next release.. the details in the previous posts..

Gotcha! Jamie will respond here shortly, I just wanted to make sure we had all the necessary info available for him when he gets here.

Sounds like that is the case, thanks!

The latest release (9.1) should respect the gateway you have set for your IP allocation range for new LXC containers.

Status: Active » Closed (fixed)

Howdy -- it looks like you made quite a bit of progress on all that, those are great steps!

Yeah Jamie and I had some trouble getting that to work in the past, that all looks excellent.

Does this link here perhaps work for you, it's a link to the post itself, rather than a "reply" link:

https://www.virtualmin.com/node/44163