is there a way to completely backup my linux system with all servers?

On many setups, all it takes to perform a comprehensive backup is to generate a scheduled Virtual Server backup each night, which can be setup in Backup and Restore.

That'll backup all of the data for your Virtual Servers, including email, dns, databases, and so forth.

Further, that would also generate a file named virtualmin.tar.gz, which contains your Virtualmin settings, server templates, accounts, and such.

If you are ever required to reinstall -- all you'd have to do is:

• Perform a default installation of your OS of choice

• Run the Virtualmin install.sh script to install Virtualmin

• Go into Backup and Restore -> Restore Virtual Server, and have it import all your Virtual Server backups as well as that virtualmin,tar.gz file.

There are a number of image generating tools on Linux if that's your preference. I don't have any specific recommendations, though I'll offer that if you like Acronis, there is a version of Acronis for Linux.

If it were only that simple!

There are potentially lots of settings that are not backed up. A lot of default settings in webmin can be changed, and, these are not reloaded of course. Then there are external packages of course that one might add, these are not backed up (though, not sure how to solve that one).

I wish ALL webmin and virtualmin settings would be backed up so that it is as simple as you say, but it's not. We have around a 3,000 line shell script to reload the system and configure virtualmin and webmin and external packages. By hand, it took several days.

Perhaps, for example, I change sshd settings in webmin, perhaps I change bind memory settings and turn of IPV6 name resolution, perhaps... Extra perl modules, module settings, enable worker mpm in apache, you name it.

I think some of this is unfortunately outside the scope of virtualmin. But, some of it is not. I wish for that stuff, that everything is saved.

If you end up having to restore a bunch of servers to a different machine (same os) on different ips, then a whole new crop of problems comes up!

Virtualmin's backup is intended to backup the virtual servers...that's it. No intention to backup the system itself. It assumes the underlying system will have the same software (roughly, though it will adjust for version changes and such) as the original system on which the backup was made. It would be a bug if you don't get warnings about missing capabilities on the new system, but Virtualmin wouldn't be able to fix the missing capabilities.

There are tools for backing up a whole Linux system (including a few in Webmin), either for use as a template system when deploying large numbers of nearly identical systems, or for use in disaster recovery. But, Virtualmin's backup is not intended for that purpose. Webmin does have a filesystem backup module, which can backup everything on your system...and that would be what you'd want to use, if you want to be able to replicate the entire system (including OS and everything else) in a disaster recovery type situation. That's probably not the ideal tool for replication and large scale deployment...and I'm not sure what tool would be most appropriate for such a case, as I haven't had need for such a use case (when I was building large scale data centers, we used kickstart to build identical systems, and then software and configs in revision control; there was no backup involved).

As Eric mentioned, there are full-system backup and replication tools for Linux, which may be what you want, if you want an identical copy of your server.

Our backup process is the following:

1. Full filesystem backup, excluding /home, with the Filesystem Backup module, once every month, with weekly incrementals. All of our changing data is in /home and is "owned" by various Virtual Servers in Virtualmin, so we don't need to have constantly updated filesystem backups.
2. Backup of all of our virtual servers every day.

This would allow us to recover from a catastrophic failure in a few hours (less if we had the backups already sitting waiting on a new machine, but we'd have to spin up a new server and copy the backups to it).

Go ahead and try reloading your virtual servers, (say 100 of them) with different IP addresses from your virtualmin backup. Your choices are:

1. Load to the original IPs from the backup (server moved - new addresses, can't use)
2. Load to the shared address (breaks every sites SSL).

It should load them to the configured virtual ips! Instead of having to manually change them all one by one.

I don't consider webmin module settings and webmin settings to be outside of virtualmins scope. I wish it was not. It makes the whole thing far harder to deal with. You can't rely on mostly identical hardware, same ips, same network card, etc. when you are restoring. I simply wish virtualmin backup had buttons to also backup webmin and webmin nodules settings, and, restore everything in either module whether or not the hardware has changed, the ips have changed, etc.

Other software, we have to deal with, agreed. I agree with the OP - backup is way too limited. Loading images onto new hardware in a new data center from a new provider is not always so easy, remotely.

If you are looking to clone entire systems for deployment purposes, Virtualmin's own backup / restore capabilities are not the right way to go, as they were never design for this situation. Even if they were expanded to capture more Webmin-controlled settings, it is still likely that you would miss other custom settings made manually.

Instead, I would recommend creating a template system with Virtualmin installed and configured the way you like, and all other packages updated and settings tweaked. Once this is done, you can create an image of the whole disk with something like Ghost, and then use that to create new systems.

This because even easier if you are using virtual machines, as they can be directly cloned and imaged .. but if you are installing systems for customers in multiple physical locations, VMs won't really help as you'd need a separate host system for each one.

Nope. Not at all, which is what I tried to explain in the other issue you said not to use any more since here was more relevant. But I am hijacking this one, and, don't feel its the same issue, so, would rather keep the other one.

My issue in this case is the OP is correct, a virtualmin backup does not come close to backing up an installation, which I define as everything managed by virtualmin AND webmin. You guys limit it ONLY to virtual servers, which is a small part of an installation. I do exclude external packages not managed by virtualmin or webmin as you do.

This makes it a huge task to restore a lost machine. Perhaps not for those who keep most everything as default admittedly.

If the problem you want to solve is that you want to duplicate an entire system, settings, virtual servers, installed software, for the purpose of disaster recovery, you should be using a full filesystem backup tool (possibly one that has bare metal recovery capabilities). Virtualmin virtual server backup is not for that purpose. There are several tools in the Linux world capable of backing up and restoring a whole system...Webmin happens to have modules for some of them (Filesystem Backup uses tar and dump, and there is a module for Bacula and Amanda which simply manages those tools). We never set out to make a system recovery tool, and we've never suggested Webmin or Virtualmin were for system backup and recovery, though you can use some of the tools that Webmin has a GUI for to do so.

The purpose of a Virtualmin virtual-server backup is to take one or more virtual servers from one machine and move them to another machine with similar configuration; it can smooth over many kinds of changes, like different OS and version, but it doesn't make system-wide changes, and it couldn't safely do so, because virtual server backups need to be restorable by non-root users. Virtualmin backups are a user-level feature.

Or, if you want to be able to roll out numerous almost-identical machines from bare metal, again, there are tools for that (Kickstart comes to mind as a scripted version, and there are image based tools, as well). That's not what we build here, though, and it would require at least a couple of additional developers and several years to do. Cloudmin can do it for virtual machines, to some degree, but I doubt it meets your needs, either since you don't seem to want to make a template system image and copy it (which is entirely doable for bare metal systems as well, though we don't have GUI tools for it, I've done it from the command line in the past).

So, why not make a system image that is perfect according to your needs, and then duplicate it to every new machine, making only the IP changes and whatever other config changes you want to make? If it is replicable enough that you believe Webmin or Virtualmin ought to be able to automate it, shouldn't it just be something you can copy from a template system? What is the problem with that method of duplicating a configuration?

An image is useless when hardware changes drastically in the case of disaster recovery. Please do not debate that, it is a fact, and I can come up with all sorts of examples where this fails in the real world. Cloning systems, different case, not talking about that here, the OP merely said virtualmin backup too limited, I agreed.

As I said, not wanting virtualmin to do full backup at all! Virtualmin backup is too limited, it needs to backup virtualmin, webmin modules settings, module settings. Nothing more, no full system backup, no extra packages not managed by virtualmin or webmin. JUST everything virtualmin and webmin manages and their settings.

The OP has had to pay for his restores, and I can easily see why! Of course, he does want to go beyond just virtualmin and webmin. I am not agreeing with that part. Perhaps you are arguing with him, not sure.

I just want the products that come from you guys, and the packages they manage, to be backed up with one backup, and restored with one backup. If you guys don't agree then don't do that. It's just too much work to restore a virtualmin/webmin installation, way way way too much work. I do NOT want to get into details, that can be argued ad naseum and every install is different so there is little point to it. This is the summary version.

To reload the OS is a whole other issue entirely. To clone a machine is a whole other issue entirely. Nowhere has anyone mentioned cloning. Unless you are reading elsewhere, which is a different issue. This ticket is about virtualmin backup being too limited.

You really do not see any value in one simple virtualmin/webmin/settings backup? Why in the world would you not want to make all sorts of different tasks easier for your customers? I don't understand??

I've said enough here, it is not my ticket. I'll let the OP post anything else he wishes to. I have said my piece. I did not wanto hijack his ticket, just to make it be known I mostly agree with him for what it's worth.

@sfatula: I don't really understand what your problem is.

You talk about restoring your whole server contents to a new machine in case of failure. The Virtualmin backup function is not designed or intended to do that!

Why don't you just make regular filesystem backups of your "/", like Joe suggested, with the backup software of your choice (tar, dump, bacula, Acronis, ...) in addition to backing up the virtual servers with VMin like once per day?

Also, Webmin has its own means of backing up the filesystem, and also its module configuration (look under Webmin -> Backup Configuration Files).

Restoring a backup to a different machine with a set of one hundred new IP addresses is outside the scope of any backup software. How should that software, or even Virtualmin, know what vserver to assign to which new IP address? It cannot know that. But you can restore to the previous IP addresses and use the Virtualmin scripting interface to batch-change the addresses.

And if you need to regularly restore stuff from a broken to a different machine that is SO different that the backed-up root filesystem does not work there, you should be looking into virtualizing your servers. That way it does not matter at all on which host your machines are running. I recently tested that procedure, and using VMWare, it is a simple matter of untaring the root filesystem onto a fresh HDD file and doing a grub-install. And, hell, if you want it even faster, then make backups of the HDD file that holds the root filesystem. That way you just need to restore that to the new host and are up and running again immediately.

That is correct, SSL breaks and many many other things when you restore virtualmin backup. I've posted about this in the forums as well as my own trouble ticket.

I'd love to see some work done to it... Would be happy to volunteer my time and thoughts to the design.

just subscribing to this interesting topic

PS

I personally use OpenVZ for backups and install Virtualmin in a container.

With tools like vzdump or vzdup (similar to vzdump, but uses duplicity which offers encryption and incremental backups) you can backup your container with minimum downtime. Disaster recovery is then typically only

1. OS reload in the data center
2. installation of OpenVZ (very quick and easy)
3. and a restore of the container (very easy).

There is virtually no performance penalty with OpenVZ (and with the new vswap feature in the RHEL6 based version extremely easy to configure).

KitchM - we have a fix for that AWstats issue, but it hasn't been released yet. The work-around is to remove the file /etc/awstats/awstats.example.com.conf before restoring (replace example.com with the actual problem domain name).

Could you tell me more about the issues with SSL that you encountered when restoring?

The way I recall it, is say you have 100 sites with 50 ips. ALL of them go back to ONE IP, and thus, the problem, now you have to change all of those manually somehow. It's either that choice, or, you have to restore the original ones. Often times, people are going to a backup server, different ips. Or, they are moving to a new more powerful server. Virtualmin does not ALLOCATE ips from the new pool of choices. That's not a choice. But it SHOULD be. After all, you define that in Virtualmin. Of course, that then affects bind....

The problem with these imaging type tools (in addition to my earlier comments) is if you need 100% uptime, and, you don't or can't thusly bring down MySQL at all just for a backup. I've not seen them work with these things running, generally, there are database issues on higher volume systems with many transactions per second. But that's a whole other issue. The topic here should be virtualmin backup. If all those other tools were so great, then, simply delete the backup function as it would be un-necessary. I don't think you can though.

There are so many files missed by this process. I'd still repeat my request to do a better backup that enables people to simply restore virtualmin AND include webmin settings since they are seen as one product. But I'll keep quiet again. I hassled you guys enough about backup issues. We had talked last year and someone said they were going to take a look at the backup / restore process. So', I'll wait for that.

I have to deal with this again this weekend as I have to move an install from one provider to another. So, all sorts of issues will come up.

Ok .. so it sounds like the real issue is that when restoring a backup of a domain with a private IP, a new IP doesn't get allocated on the destination system. Currently the restore process lets you select if a domain should use the common shared IP or allocate a new IP ... but this applies to all restored domains, which really isn't what you want.

I will work on fixing that ..

FYI, the next Virtualmin release will change the default mode when restoring domains from the UI to re-allocate a private IP of the domain had one originally, and use the default shared IP otherwise.

If restoring using from the command line or via the remote API, you can use the --original-ip flag to trigger this behavior.