Waiting for the number of running backups to drop below the system limit of 3 ..

Eric, to answer your question, I'm setting up a daily (incremental), weekly (full) and monthly (full) back-up on each virtual server for all of the sub-server under that virtual server. I am not doing a server-wide back-up of all virtual servers, as I have a different system for doing server-wide back-ups outside of Virtualmin.

Jamie, I assumed there was a place to configure this, and I strongly suspect I left this setting at it's default when I set up the server originally. I'm leery of setting anything to "unlimited", but if I can set it to something high (like in the dozens or even hundreds) and scheduled back-ups will wait patiently in the queue without all trying to run at exactly the same time and crippling the server, then that would work for me. Is that what you suggest?

Craig

Eric, what you describe seems way more tedious to set up. Admittedly, it's going to be tedious to set this up for existing virtual servers, but if I'm not mistaken I seem to recall that I can include a back-up schedule in a hosting plan template. Feel free to correct me if I'm wrong, as I'm going by memory and haven't checked it for myself yet.

If I understand you correctly, you're suggesting that root set up a single server-wide back-up schedule. Sounds fine (but is what I already have happening outside of Virtualmin), except that individual users will not be able to manage their own back-ups, and I will have to be asked to step in to restore back-ups instead of individual users doing it themselves through Virtualmin. This seems to be totally against the whole point of a self-service control panel. Or am I misunderstanding you?

What I want -- to set up a basic back-up schedule when creating virtual servers and then allowing users to manage it all themselves through Virtualmin -- doesn't seem to be an edge case to me; rather it seems to most logical and basic set-up. Are you suggesting that doing it this way is going to cause performance problems on a server that is otherwise capable of handling nominal traffic to existing websites? Is the performance problem going to be because, as I said in my last post, scheduled back-ups won't "wait patiently in the queue without all trying to run at exactly the same time and crippling the server"?

Jamie, that wasn't what I said, but maybe it's a good idea. Except that what I had in mind (and I'm surprised this seems so unusual, apparently) is setting up new virtual servers with a default back-up schedule (daily, weekly and monthly) as if the user himself had done so. He can then delete or change or add to the default schedule as he wishes, as well as restore individual back-ups on his own.

The issue seems to be not in the act of setting up scheduled back-ups, but in how they are actually being run when the schedule comes up. The subject of this ticket is the error I saw recently with a couple of scheduled back-ups failing because (to put it simply) too many were scheduled at the same time, and they were scheduled at the same time because I chose a "simple schedule" rather than a complex one. I could choose a complex schedule, but then I either have to manually keep track of spacing back-ups or pick random times for each new set of back-ups.

I guess what I'd like to see is Virtualmin running all scheduled back-ups in some sort of intelligent rotation, instead of starting 100 back-up jobs at midnight and having 97 of them fail because the 3 that succeeded took longer than 3 minutes. I realise I can change those defaults at System Settings -> Virtualmin Configuration -> Backup and Restore, but my questions about the knock-on effect of doing so went unanswered.

Which led me to my other earlier question: If Virtualmin or cron won't somehow balance things out or let a job sit in the queue for however long is necessary to let it run in turn so as not to cripple the server, what numbers should I use in the "Maximum concurrent backups" and "Time to wait if maximum is exceeded" fields at System Settings -> Virtualmin Configuration -> Backup and Restore?

Hi Eric,

Thanks for your reply. I don't want to appear ungrateful, but I think that the whole point of this problem has been missed, despite the longer and longer messages trying to describe it.

The issue is that back-ups are failing because of how Virtualmin is scheduling them by default. Full stop.

The issue is NOT the actual setting up of scheduled back-ups, although now that I look at Jamie's idea again I do agree that it is indeed an awesome solution (and better than the one I had planned to implement), as long as I can set it up once and each new virtual server I add in the future will automatically be included. Thank-you Jamie.

In the meantime, to prevent the errors and failures I'm seeing -- and I am currently only backing up 7 top-level virtual servers with 37 domains, not all 150 or so domains on the server -- I have increased the "Time to wait if maximum is exceeded" field at System Settings -> Virtualmin Configuration -> Backup and Restore to 600 seconds. I will increase it again if I get further failures of the same kind.

But again, Jamie's idea does sound like a good one if:

• it solves the time-out problem, which presumably it will if it's just one cron job,
• I can set it up once and future virtual servers will automatically be added or included, and
• user's can restore their own back-ups as if they had set up the cron job themselves.

On that first point, the Webalizer cron job seems to be running in the background all day, so I'd imagine Jamie's idea would be too, at a low priority.

Thanks.

Craig

Thanks Eric. You make perfect sense.

Here's an idea: Years ago when I started using Slicehost (which later became Rackspace Cloud) they had some sort of automated way to suggest that if you wanted to run updatedb (part of mlocate) on your VPS that you do it at a specific number of minutes past the hour, so that not all VPSes on the physical server were running it at the same time. In other words, they gave your VPS a slot during which only you (or perhaps only a portion of the VPSes on the physical host) would be running a disk-intensive operation.

If Virtualmin were to have similar logic, it too could suggest to users -- both root and ordinary users -- when they are setting up a scheduled back-up that it be run at a particular time. In other words, there would be three options instead of two: simple schedule, complex schedule, and a suggested schedule, the latter being the default. So for Bob setting up his daily back-up the system might suggest he run it at 02:42, instead of running it with everyone else at 00:00.

But that idea is under the current system, and probably more complicated than Jamie's idea. However it addresses your last paragraph.

Jamie: OK, I'm surprised at your assumption, since Virtualmin does make possible what you assumed would not happen, and some idiot like me is going to come along and want as much control in the hands of the users as possible and proceed as I did.

But I see your point, and I think what you've suggested -- root backing up all domains, but ordinary users being able to restore their own domains -- is a good compromise

Eric:

Craig, I wasn't actually aware of the option you're describing... the option to automatically create a backup job when making a Virtual Server. Where is that option located?

I'm assuming you're referring to my first paragraph in message #5. As I said at the time, I was going by memory and hadn't double-checked yet. I expected to find it under System Settings -> Server Templates, but I don't see it so I was wrong. This makes me even more in favour of Jamie's planned new feature, as without it I was planning to set up a back-up schedule manually each time I added a new virtual server.

But under Jamie's new feature I'm expecting that if I have a server hosting abc.com and xyz.com, and a back-up job that includes "All virtual servers", that when I later add example.com it will automatically be included in the back-up of "All virtual servers" from the next time the back-up runs. I'm repeating myself, but I also want to make sure we're all on the same page.

Or are you saying that issue occurs just by having multiple users go in and setup the backup jobs on their own?

Mostly I have been setting up scheduled back-up jobs under individual users' accounts, when logged in as them, but individual users have also set up their own scheduled back-up jobs. Either way there are multiple scheduled back-up jobs instead of the one big job that Jamie apparently expected server admins to set up.

Thanks for the tip, and the advice against it.

And thanks for clearing up the "all domains" / "all virtual servers" thing.

Sorry to reactivate this ticket, but I'm just wondering when you anticipate adding this feature. Thanks.