Resource limits have no effect

Submitted by Donceaster on Tue, 10/30/2012 - 08:52

I'm running CentOS 6.3 with Cloudmin 6.5.kvm on a quad core box with 16GB of ram. It is being used at a small business currently and has been up for 24 days working almost flawlessly.

The one problem that I am having is that setting resource limits on the virtual guest seems to have no effect on their actual resource usage on the host. I have all of the machines set to 10% of total CPU yet each machine is reported by top to be using around 20-25% of CPU. This is generally consistent regardless of the guest OS although when booting a Windows guest that PID CPU usage will spike to 84%.

I can set a guest's CPU limit to 1% and top still reports that it's real usage is around 20-25%, the same applies when I set that guest's CPU limit to unlimited. Rebooting the guest instance has no effect.

Here is the current output from cgsnapshot -s

Configuration file generated by cgsnapshot

mount { cpuset = /cgroup/cpuset; cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; memory = /cgroup/memory; devices = /cgroup/devices; freezer = /cgroup/freezer; net_cls = /cgroup/net_cls; blkio = /cgroup/blkio; }

group 1350754089274080 { cpu { cpu.rt_period_us="1000000"; cpu.rt_runtime_us="0"; cpu.cfs_period_us="100000"; cpu.cfs_quota_us="-1"; } }

group 1349482993202820 { cpu { cpu.rt_period_us="1000000"; cpu.rt_runtime_us="0"; cpu.cfs_period_us="100000"; cpu.cfs_quota_us="-1"; } }

group 134947395369670 { cpu { cpu.rt_period_us="1000000"; cpu.rt_runtime_us="0"; cpu.cfs_period_us="100000"; cpu.cfs_quota_us="-1"; } }

group 134946262076950 { cpu { cpu.rt_period_us="1000000"; cpu.rt_runtime_us="0"; cpu.cfs_period_us="100000"; cpu.cfs_quota_us="-1"; } }

The odd thing about this is that it shows only 4 groups and there are 5 machines. The machine that I suppose should be there as the 5th group should have 1% of the CPU allocated. (Note I don't necessarily want 1% allocated I just wanted to see what it would do to that guest since it is a test box)

Any help would be greatly appreciated. Thanks!

Status: 
Closed (fixed)

Comments

Submitted by JamieCameron on Tue, 10/30/2012 - 12:08

Cgroups will allow processes to spike above their assigned CPU limits - where they really come into play is when multiple VMs want to use the CPU at the same time. In that situation, they should be limited to a fraction of the available CPU based on the limits you set.

Submitted by Donceaster on Tue, 10/30/2012 - 12:49

I understand that processes are allowed to spike above their limit but that isn't my situation. I see CPU usage by the guest that is higher than the limits I set all the time, not a spike. No matter what limits I set on CPU usage the guest process CPU usage shown by top is the same.

Submitted by JamieCameron on Tue, 10/30/2012 - 15:34

What does the file /cgroup/cpu/1350754089274080/cpu.shares contain on your system? That should show the CPU assigned to the VM with ID 1350754089274080 - 1024 means 100% CPU.

Submitted by Donceaster on Tue, 10/30/2012 - 16:05

"cat /cgroup/cpu/1350754089274080/cpu.shares" returns 20. If 1024 is 100% CPU then 20 should be 1.95% CPU which is almost correct since that VM ID is the test machine that currently should have 1% of the CPU according to the Cloudmin web interface.

However, it still seems that the PID associated with that VM is using 22.9% of the CPU. With the VM set at 1% (or 1.95%) of the CPU it should be crippled, yet it runs fine....just as though it were using 22.9% of a 3.1Ghz proc.

I'm confused by this behavior and maybe I'm misunderstanding how resource limits are supposed to work.

$top PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

32397 root 20 0 775m 534m 4372 R 22.9 3.4 216:02.25 qemu-kvm

Submitted by JamieCameron on Tue, 10/30/2012 - 16:16

Ok, that looks correct.

I think the issue is that cgroups only limit the CPU a VM can use when the CPU is fully utilized. So if only one VM is using the CPU, it can use 100% regardless of the cgroup limit. However, if two processes are using the CPU their actual usage will be proportional to the cgroup CPU share assigned.

This is mentioned on http://www.networkworld.com/news/2007/101207-kernel.html . I would actually prefer a system in which the CPU usage was capped, but sadly that doesn't appear to be how cgroups work ..