Virtualmin backup and cloudmin host responsiveness

Submitted by mcordas on Mon, 12/05/2011 - 08:49 Pro Licensee

Hello,

On our Cloudmin hosts, we have "web" KVM instances, running apache, and "MySQL" instances, running MySQL. The web instances access MySQL on the mysql instances via mysql-proxy.

In addition to full VM backups, we configured, vie Virtualmin, daily Virtual Domain backups in the "web instances" : each virtual domain and it's mysql dbs are backuped separately on external ressources.

Problem : the io and cpu load skyrockets during those backups, impacting the host's responsiveness.

It seems that it occurs a few minutes, apparently during the "mysqldump".

The nice and ionice options in virtualmin don't seem to have results.

Is there a way to "slow down" those backups, in order to less impact apache's responsiveness ? Maybe introducing nice and ionice options in mysqldump ?

Thanks a lot.

Status: 
Closed (fixed)

Comments

Submitted by andreychek on Mon, 12/05/2011 - 12:53

Yeah, the backups can be IO and CPU intensive.

Setting the nice and ionice options can help, but generally don't make a huge difference.

When setting nice and ionice for cron jobs, it would work for child processes such as mysqldump as well. It's possible that, due to the resource requirements of the backup, that you're just not seeing a large difference even with those options in effect.

However, just to verify that you have the right options there -- where did you enable the nice and ionice options?

Submitted by mcordas on Mon, 12/05/2011 - 15:09 Pro Licensee

In the "apache" host, in Webmin / Webmin configuration / Advanced options. Is that right ?

Submitted by JamieCameron on Mon, 12/05/2011 - 15:23

You'd also want to make the same change on your "mysql" host.

Submitted by mcordas on Mon, 12/05/2011 - 15:41 Pro Licensee

Already done. io priority was on 6, though. I put it on 7.

Should I set also "CPU priority for scheduled jobs" ?

Submitted by JamieCameron on Mon, 12/05/2011 - 15:43

Yes, changing the CPU priority to a higher number should also help.

Submitted by mcordas on Mon, 12/05/2011 - 15:54 Pro Licensee

OK I'll try that and keep you in touch.

Thanks.

Submitted by mcordas on Mon, 12/05/2011 - 16:04 Pro Licensee

Oups. Applying those settings restarted Apache. Is it normal ?

Does the ""CPU priority for scheduled jobs" option in Webmin affect other processes than the backups ?

Submitted by JamieCameron on Mon, 12/05/2011 - 16:43

I wouldn't expect this to touch Apache at all..

Also, this setting effects all background tasks launched by Virtualmin, such as backups, status collection and monitoring..

Submitted by mcordas on Tue, 12/06/2011 - 15:43 Pro Licensee

Nope, even with the "CPU priority for scheduled jobs" at higher number didn't significally change the io "consumption" during the backups.

Here's a question : would it be possible for cloudmin to use the cgroups feature with RHEL 6/CentOS6 kernels, in order to control i/o for VMs at the block level ?

Here's a thread in a RedHat forum :

http://fossplanet.com/f13/%5Blibvirt%5D-cgroup-limits-only-affect-kvm-gu...

Thanks.

Submitted by JamieCameron on Tue, 12/06/2011 - 19:35

I just realized that due to your virtualization setup, IO priorities aren't going to help as they only apply within processes on a single VM. What you really want is a way to limit the IO load of the "mysql" VM itself .. which we don't have a good solution for yet. Cgroups may be the long-term answer though.

A short-term fix may be to manually change the IO class of the kvm process for your mysql VM. This can be done with the ionice command.

Submitted by mcordas on Wed, 12/07/2011 - 15:53 Pro Licensee

I think we will stay like that for the moment. Fidling with the ionice command on the kvm process could have unwanted side effects, no ?

We look forward to test cgroups when we have a chance to run VMs on a CentOS 6 host. Any chance that feature could be implemented in Cloudmin's gui once ? ;)

Thanks for you help and work.

Submitted by JamieCameron on Wed, 12/07/2011 - 19:23

I'll look into adding cgroups support .. but it may take a release or two.

Submitted by JamieCameron on Tue, 01/31/2012 - 22:42

The next Cloudmin release (version 6.1) will let you limit the CPU used by a KVM instance, on host systems that support cgroups (as CentOS 6 does).

Submitted by Issues on Tue, 02/14/2012 - 22:46

Automatically closed -- issue fixed for 2 weeks with no activity.

Submitted by mcordas on Wed, 02/15/2012 - 04:12 Pro Licensee

That's cool thanks.

Any ETA on such a possibility with i/o usage ?

Regards.

Submitted by JamieCameron on Wed, 02/15/2012 - 12:34

I'm still looking into this .. it may be possible with KVM by running the kvm process with a lower IO class.

Submitted by JamieCameron on Sat, 02/25/2012 - 13:46

FYI, IO class support for KVM and OpenVZ virtual systems has just been implemented, for inclusion in the 6.2 Cloudmin release.