Virtualmin backup and restore - make more complete

9 posts / 0 new
Last post
#1 Wed, 06/08/2011 - 22:32
sfatula

Virtualmin backup and restore - make more complete

I'd like to make this the small version so no one misunderstands at all! So, not posting whys, when I do, people want to come up with all sorts of reasons why not to do this, and imagine all sorts of things I am not trying to ask for. The why is VERY simple - it makes your life easier. If anyone sees the wisdom of this - please post. Having been through more than 50 virtualmin installs, various situations, I certainly do! This was being discussed on some tickets, I'd rather open it up to more people. Esp newbies.

I would like to redefine virtualmin backups (and thus restores) scope. Currently, the intent is to store off virtualmin settings, and, virtualmin virtual servers. It works ok for this.

I'd like to suggest making it somewhat better for all, esp those who have encountered numerous issues in the past, or, those who highly customize webmin settings and module settings (both). I define this as follows:

Webmin settings, - all settings done on the Webmin Configuration screens, AS WELL AS all settings done on various modules "module config" screens found in the upper left of most modules.

Module settings - All values that can be configured for each module from Webmin or Virtualmin. So, for example, all Apache settings that can possibly be set, all sshd settings, you name it.

So, I would merely want an additional option to store the webmin and module settings within the virtualmin backup, JUST AS it currently allows you to do with virtualmin settings. You don't have to click the box, but you might if you have a use for it. You don't have to run a separate webmin backup, etc.

This would enable a simple restore of a single backup to a system. Not asking for any backup outside of this. Such a simple restore simplifies the process for anyone wanting to restore their virtualmin/webmin system, or anyone trying to help someone else restore their system (as we do!). It is not a substitute for other forms of backup and not intended to be. But one place = good thing.

We simply see virtualmin and webmin as a single product. Virtualmin can't really run without webmin, they are tightly integrated. Webmin can run without virtualmin, so, webmin backup is fine for those customers. But virtualmin and webmin do need to be in sync else bad things can happen. Thus, this request.

I suppose it could be as simple as including whatever webmin backup does in the virtualmin backup file, just as it does for the other stuff. It would certainly encourage people to do a more complete backup as well, which unfortunately, we find many do not realize this. When a user has changed more than 500 settings in webmin and later, comes to us as a new client to fix things due to some disaster, it's no small task to set them back up. While we appreciate the work, really, the customer shouldn't have to do this. We believe adding the checkbox would make it clear it's a necessary thing. Many of them seem to assume the virtualmin backup also backs up webmin, which costs them in the end.

Thu, 06/09/2011 - 02:58
Locutus

You're not getting it, are you?

Webmin has its own backup feature. As I already said in reply to one of your lengthy issue posts: Take a look at Webmin -> Backup Configuration Files.

Why making such a fuss about having two options in two places that do the same thing? Simply schedule a regular backup of the config in Webmin if the other suggested options don't work for you, and you're done.

Thu, 06/09/2011 - 12:37
sfatula

Rather than make suggestions and not reading what someone posts, why not reply to specific points. I am going to assume for the moment that you really do have an interest in the whys, and, actually care and are not just posting to be troublesome as you did in the other ticket. Hopefully, clearly, you agree that I am somewhat passionate about this need. Maybe, just maybe, there is a reason.

So, first, some background.

We are consultants and system admins. We do all sorts of troubleshooting work for people with problems, and many other sorts of things. In this specific case, I am speaking of Virtualmin or Virtualmin PRO customers, who, either, were set up by themselves, or, someone else and have a problem of some sort. So, asking ME to do this or that is 100% irrelevant to what I am asking for, so, please understand this point. We do a lot of work beyond Virtualmin (it's not a huge market and not what we specifically target), but, this is the topic here. So, it is simply not relevant to say run a filesystem backup or use webmin backup. Ok? It is also the case that we also have our own systems which we DO manage, and, some clients systems which we do setup and manage. That is not the issue here, for the most part. But it still is an issue that can be made simpler.

Now, we have had the joy of restoring around 20 systems in the past year or so, not sure why so many this year compared to previous years, maybe word of mouth, who knows. 19 of those posed major problems. So, before we go anywhere, understand, 19 out of 20 immediately says to me there is a problem without knowing any other info. Would you agree with that? Now, I will admit that there are likely many people who don't have major problems as they of course would not be coming to us.

Most of these people (or whoever installed their system) ASSUMED that virtualmin backup backed up virtualmin, which included webmin. It's not a separate system, after all, they installed virtualmin NOT webmin (their point of view). Reasonable? Yes. Technically correct? No. It doesn't back up "webmin" as you may know. SOME of them assumed it backed up the system, I do not believe that can be helped except with maybe some help text. We agree on this one I am sure.

So, there's your background and why your thoughts and suggestions are meaningless to this point.

Now, what types of problems do people have. Let's go over, for example, what the Virtualmin doc says THEY do and analyze how that would work. Their "use case". Perhaps it works well for them. Curiously, they have this comment in the doc:

"Daily backups of all virtual servers on the system using Virtualmin's Backup feature. We do a full backup, including Virtualmin meta-data and Webmin stuff". Hmm, what does the "and Webmin stuff" mean? Do they really do this from virtualmin backup?

Going down the page, and obviously recognizing the problem, it says this for remote systems:

"Restore the virtual server backups from the Virtualmin backups. Test. If something is broken (because of customizations I did outside of Virtualmin to non-Virtualmin related services), install the necessary packages (assuming I installed everything from packages, as I generally do), and restore the configuration (if necessary) by selectively pulling stuff out of the filesystem dump"

Does that sound easy or trivial? Well, imagine if you will an admin trying to help someone else out who doesn't really know what they may have ever done? Not trivial. Not when it says "outside of virtualmin", this also means anything done in webmin which is technically outside of virtualmin.

So, this is in the official doc. If you have a problem with that, blame them. Does this sound like something a new user of virtualmin might understand completely and easily?

My understanding, is what they are doing is actually different than what the doc says. I believe, and may recall incorrectly, they only do the filesystem backup monthly or something like that. If that is the case, then, this poses many other problems. As you will see below.

So, now we come to the next idea, you said to merely run virtualmin backup and webmin backup. Ok, have you done this? I challenge you to restore a system using this technique. this technique will in fact work as long as both are done on the same schedule. If they are not, it may still mostly work if the system is fairly static. If the system has new servers added and created a lot, and, named record changes, who knows what else, then, this fails miserably.

Why does that fail? Well, if they are not on the same schedule, then, here's the problem. First, you make a choice, restore webmin then virtualmin via their backup modules, or, virtualmin then webmin. So, take webmin then virtualmin. Ooops, your restore of virtualmin fails. Why does it fail? Well, because it starts to get confused because things already exist that it does not to expect to exist. Given enough time, you can make this work by manually taking care of things.

So, let's try the other way, let's restore virtualmin THEN webmin! Well, that has another problem, given the assumption they were not done at the same time. Now the problem is apache changes, bind changes, you name it is not as current on the webmin configuration backup, so, now you have missing stuff.

The third way is to do what the virtualmin doc said, which is manually fix things.

Ok, so, back to the beginning. Remember, not systems we set up. How could ALL of this be avoided? Adding a simple checkbox to the virtualmin backup screen to backup not only virtualmin, but, webmin settings. The "hidden task" here is for the programmers to actually work out the details I mentioned and make sure one does not overlay the other with invalid info, data not stored twice, etc.

So, I ask you, which is easier for a non professional? Which do you think would be easier and less work to restore? Is that asking too much, to actually make it more easy to use and learn?

So, you other point was why duplicate the backup? In reality, I assume they would not be exactly the same since you have to consider what the other module does backup. Secondly, there are all sorts of stuff already in virtualmin that exist in webmin! Third - it will cause LESS problems for users of virtualmin and make the restore process easier for all. Shouldn't that be what we want?

What types of things go wrong you ask? Some examples. Perhaps the customer changed their bind to not do ipv6 name resolution since they are running on a VPS with little memory, and, also some other bind settings to reduce memory. You restore a virtualmin backup, well, guess what, the system doesn't run, out of memory. Why? Because that's "outside of virtualmin". Nevermind the changes were made mostly or entirely in webmin, which was installed by virtualmin.

Maybe the customer raised (or reduced) limits in httpd.conf via webmin. So, we restore the backup, and, complaints all over for apache problems. Why? Well, it doesn't restore the apache configuration that was changed by a user in webmin, which they didn't see themselves as installing.

Obviously, you should realize by now I have tons of examples, but more are pointless. This explains is clear detail what the issue is.

If you still disagree, that's fine, but don't say there isn't any issue. There IS, else, we would not be getting this business. Which is fine, perhaps I should not complain, my complaint is more out of wanting users to not have to pay more and more for something that I believe can be avoided, fairly easily.

Yes, I do realize this does not resolve the support ticket we both commented in as that was from a customer who wanted a perfect backup of everything on the system. I am not at all asking for that, I just want a perfect backup of virtualmin, which includes webmin! So, that users of the product can more easily understand and backup their system without endless research and planning. Perfect in reality? No, but many of the 19 problem systems we had would have been avoided. But not all of them.

Imagine, user wants to restore, they have installed the packages and virtualmin, and they simply click restore. How nice would that be?

Thu, 06/09/2011 - 13:14
andreychek

Howdy,

We do care about improving Webmin and Virtualmin, and all of you guys are skilled sysadmins -- so we value your input.

Now, as this discussion has gone on, the posts have gotten a bit longer, and I'm having trouble keeping up :-)

Here is what I'm planning on doing -- I'm planning on reviewing your posts over the past couple of days, and I'm going to turn all that into a set of simple feature requests. All of which may or may not be accepted, we can debate those points then.

However, my goal in the meantime is to make this all as short and simple as possible, and then we can work from there :-)

Thanks,

-Eric

Fri, 06/10/2011 - 03:43
Locutus

Rather than make suggestions and not reading what someone posts, why not reply to specific points. I am going to assume for the moment that you really do have an interest in the whys, and, actually care and are not just posting to be troublesome as you did in the other ticket.

Alright then, I'm outta this discussion.

Unlike Eric, Joe and Jamie, I'm not getting paid to give help here, I do this voluntarily in my free time. And I don't have THAT much time or feel the urge to read through multiple screenfuls of stuff about things that have already been said.

And mostly those are things that should be no problem to people who wish to operate a hosting service on the grounds of any hosting control panel, in this case Virtualmin. Quoting your example of "installing packages that are missing after a restore": If you can't do that, if you are not able to -- and not willing to learn how to -- identify and install some missing packages on a Linux system, you should not be operating a hosting system at all. That's my opinion. Period.

So, if my suggestions are meaningless and just "troublesome" to you, so be it. I'm going to ignore further posts or help requests from you and won't give you any such meaningless suggestions anymore.

Fri, 06/10/2011 - 09:06
andreychek

Howdy,

I'd like to make this the small version so no one misunderstands at all! So, not posting whys, when I do, people want to come up with all sorts of reasons why not to do this, and imagine all sorts of things I am not trying to ask for. [snip] I would like to redefine virtualmin backups (and thus restores) scope. Currently, the intent is to store off virtualmin settings, and, virtualmin virtual servers. It works ok for this.

I'll start with this (don't worry, I'll get to your Webmin config backup in a minute) -- over the course of the various comments that came up throughout the 3-4 threads that discussed backups this week, we saw several legitimate issues you brought up that should probably be addressed.

Unfortunately, they were buried just a bit too deep within the discussion for us to be able to address each issue that needed addressed.

One example that pops to mind is the issue with IP addresses and SSL certificates... if you find that migrating from one server to another is causing unexpected behavior in some circumstances -- we need to address that :-)

Open a bug report just for that one specific issue, and we'll talk about it.

That goes for any of the issues you brought up that can occur during migrations -- if you run into strangeness, file a request to address that strangeness.

For us to be able to help though, make sure you keep them to one topic per support request, and keep the requests as simple as possible. We get confused easily :-)

I'd like to suggest making it somewhat better for all, esp those who have encountered numerous issues in the past, or, those who highly customize webmin settings and module settings (both). I define this as follows:

Okay, now onto your two specific requests you made in this thread here --

I understand what you're asking. And I appreciate that, if people think something is being backed up, and it's not -- that can cause some trouble.

Here's the trouble -- Virtualmin's backups go to great lengths to be cross-platform and portable. Backups generated from one system should have no trouble going to any other supported distro, version, or architecture (and if you ever discover that's not the case, that's likely a bug).

Webmin config files, on the other hand, aren't that way. They're tied to the specific system you're on, and frequently could cause breakage if changing distributions or versions.

System config files are the same way -- if files from /etc are copied from one distribution to another, something is likely to break.

So, if we provided an option within the Virtualmin backups for including a Webmin and system config backup as well -- that would mean that someone who copies those files to another system expecting that it can be safely imported would be in for a big surprise when their new server stops functioning.

Our concern here is about it breaking the expected behavior for migrations.

However, it sounds like the point you're bringing up is for a typical backup in preparation for disaster recovery -- if folks who setup Virtualmin backups are expecting to see it backup their Webmin and system configuration as well, that's no good either :-)

You brought up two specific points:

Webmin settings, - all settings done on the Webmin Configuration screens, AS WELL AS all settings done on various modules "module config" screens found in the upper left of most modules.

I appreciate that users don't all know to look for this... but just to make sure we're on the same page -- do you feel that the backup available in Webmin -> Webmin -> Backup Configuration Files backs up all the Webmin settings that would be needed?

Worded another way -- if we figured out a way to get users to this screen, would that solve this particular problem?

Module settings - All values that can be configured for each module from Webmin or Virtualmin. So, for example, all Apache settings that can possibly be set, all sshd settings, you name it.

Well, remember that Webmin directly edits the config files in question. It doesn't store any metadata about what options a user has tweaked.

So if a Webmin user wants to edit the SSH config, and set "PermitRootLogin" to "No" -- Webmin directly edits /etc/ssh/sshd_config, and makes that change.

In this example, the only way to have a backup of the changed settings would be to actually backup the full /etc/ssh/sshd_config file. There's no record that the "PermitRootLogin" option was changed.

What that means is that backing up the module settings is a matter of backing up all the config files in /etc.

That's also configurable within the Webmin "Backup Configuration Files"... but I think your point here is that this option isn't helpful if users don't know to look for it.

So the same question applies to this too -- if we got users to look in the Webmin backup screen, would that solve the problem you're bringing up here?

Thanks,

-Eric

Fri, 06/10/2011 - 14:50
sfatula

Thanks for reading. Glad you understand the issues here, I was a total failure in explaining to Locutus who still seems to think I am asking for help! So, my posts were mostly directed his way, apparently, wasted as I never was able to get him to understand. But I was wanting to! I apologize to him for not explaining very well apparently as I am sure he (like I) am a good system admin. I guess I wasted his time. Nevertheless, the posts are meant for you guys, not him in the end.

Anyway, my goal is to help more novice people use the product, and not totally get hosed when something goes wrong. Rather than say let's only have seasoned expert system admins use the product, why not try and open it up to more people where possible. Certainly, at least more novice admins. If something goes wrong and they get hosed, they may just drop the product which i don't want either since I like the product and wish to see it succeed.

Yes, I have been remiss in not filing all the bug reports. What ends up happening of course when one is trying to get something up and running after some problem for someone, and, the work goes into days, then, one tends to forget what precisely was the issue so it;s get lost in the frantic pace. Sorry about that. The IP thing happened months ago, so, do not recall all the details, so, not sure if opening a ticket is a good idea at this time.

I'll gladly keep to one topic per support request. However, if someones chimes in saying it's not a problem, I will feel the need to defend the request if I feel they are wrong.

The example of one person who you can verify was in fact the one support request where the user said virtualmin backup was too limited, after he had issues of course. So, there's one example of someone who seems to think it does more than it does.

I agree with your statement of the first issue - virtualmin backups are intended to be portable across distros. Yes, distros can use different version of products, thus, their config files can be different. So, perhaps the solution here is two allow the user to pick which type of backup they want. For example, "portable backup", or, "disaster recovery" backup. One might be portable, the other not so much. I would imagine, the typical user wants the latter, rarely, the former. Right? EVERY user should want the latter. But as you say, they are for a different purpose. I am glad to hear you recognize this! So, what would be the problem with offering two types of backups?

For disaster recovery, which everyone should want, the more it can do the better. However, simply storing filesystem level is too much, that i do agree is a different issue. Some customers have TB of space, to offload that daily offsite (as you should) is simply not feasible. Other techniques are used in such cases.

The webmin backup is not sufficient as is due the the reasons I tried to explain. There is a sequence involved to it if it is a separate backup. Better to include in the vmin backup IF you think creating two types of backups is a good thing. I do obviously.

Yes, backing up the various modules config files in /etc is precisely what I am hoping can be done in the 'disaster recovery' style of backup. Yes, PermitRootLogin is a perfect example, we at least always change that to certificate only. One of tons of settings we change.

So, in summary, why not 2 backup types? The guy trying to move would almost certainly take a manual backup of a portable nature since he doesn't likely want to go back to the last scheduled backup, right?

I am not so sure webmin actually does backup all of the needed config files. Those are fairly easy to find, just look at each packages list of files it installed for the most part since they are almost always in /etc. I recall at one time we ourselves were going to use the webmin backup and found some files missing, I know I filed at least one bug report but we then went a different way.

Worst case scenario is the guy who does the daily disaster sort of backup, but system dies, and, replacement machine is different architecture or OS, though, not sure why. Seems highly unlikely? Possible? Yes. Perhaps the restore end can handle that, perhaps a checkbox to not restore such files using some appropriate wording would resolve that. After all, anything you add to the disaster recovery style of backup is in addition to the stuff it does now, so, it shouldn't HAVE to break anything.

As you know, in the current documented process, you guys show manual work involved anyway, so, for me, the backup and restore should function is the most useful and typical manner people would want. And I believe that to be the disaster recovery style of backup, not, the portable kind.

So, still thinking here, maybe there is only one style of backup, and that includes the /etc config files and the /etc/webmin config files for the modules. That isn't a whole lot of data. So, the issue is then merely a restore issue, i.e., are we trying to change architectures or os?

Comments?

Fri, 06/10/2011 - 21:12
andreychek

Yes, I have been remiss in not filing all the bug reports. What ends up happening of course when one is trying to get something up and running after some problem for someone, and, the work goes into days,

It's perfectly fine, and no need to post anything about the IP addresses now... just for future reference, plenty of those sounded like things we'd like to correct. If you run into them again, give us a shout :-)

The webmin backup is not sufficient as is due the the reasons I tried to explain. There is a sequence involved to it if it is a separate backup. Better to include in the vmin backup IF you think creating two types of backups is a good thing. I do obviously.

Well, I think I'm going to start by doing some testing with the Webmin backup features, and see what their failings are.

If they don't backup things they should -- well, we'd like to make sure that the features that current exist do what they're supposed to :-)

And then we can work from there. I'll do some testing on that next week.

Thanks!

-Eric

Sat, 06/11/2011 - 05:55
Locutus

After following this discussion (rather superficially I must admit) for a bit longer, I'd like to apologize for my previous harsh reaction -- it would indeed seem that the parts of sfatula's posts that I read conveyed an incorrect picture of what he was after.

Even though this topic is still too extensive for me to follow completely, I'd be glad to see improvements in Virtualmin stemming from it! :)

Topic locked