XEN performance problems

i did some tests with dd and found out that performance on my domU are 2-3x slower than on the dom0. the domUs are stored on LVM volumes.

1. In the xen vms, the network interface in /etc/network/interfaces is configured in a static way. I have three VMs, one debian x86, one debian x64 and one ubuntu x86. The ubuntu, strangely, is starting up really fast. Although in the console i see

but basically it starts up really quickly. All three VMs are configured in a static manner. However, i noticed something strange in the xen config file for the vm that is makind problems: vif = ['ip=123.123.123.123,mac=00:00:00:00:00:00,rate=92160K rate=90Mb/s'] two times rate=? is that correct like that?

2. The host system seems to be unaffected by these high IO loads. Even the other VMs are unaffected. What do you mean with contention? Btw i removed the "locate" command, which had a daily cron script that scanned the whole drive, on the affecting VM. Hopefully things wont get so bad tomorrow, but still i often notice that from time to time the load goes up drammatically for some minutes, then returns back to normal.

btw i don't know if this is related in any way, but on all VMs i get this error in their consoles: INIT: Id "xvc0" respawning too fast: disabled for 5 minutes oh and the xm text console seems not to work for the ubuntu vm (the one that starts up fast). i just see that Plymouth error i pasted above, then nothing else.

unfortunately it does not help. btw, the first message showing up after it resumes booting normally is

accessing the vm via ssh is not possible at this point, only via xm console. cpu usage both on host and vm is very low.

nope, no nfs mounts

ok, i am basically constantly monitoring the VM to see when and why the load rises so much from time to time. basically it runs mostly fine, but sometimes there just is a lockup (top does not update anymore in putty) then it gets back and suddenly load is up to 70.00 or something. so i checked some logs to see what the hell is going on at those times, and look what i found in kern.log:

It appears that only my Virtualmin Pro XEN VM is affected by this. There are no such errors on the other two VMs (which have much less work to do). Also, everything on the host appears to be working fine. The guest locking up is a Debian x64 with 6 cores assigned to it. The host system has a Xeon processor with 4 corse + HT.

odd, my previous post seems empty, but when i click edit on it i can see what i wrote. can you see the message? anyhoo, in short: in the domU affected by lockups (sporadic very high load) i had messages in kern.log about the CPUs locking up for 61s and the clocksource going backwards. i now followed workaround 1 from http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27 and hope that the problem will go away...

did not help

vm is so slow it is barely useable

i'm getting desperate, i can't afford to kill the vm several times a day :(

Hi there.

The same system was running on a physical machine with less ram and much less cpu power and was doing fine, even with stuff on the swap partition.

I monitored iotop the whole day, all I can say is that there is nothing unusual there, in the scenario where the loads gets higher and higher and the iowait rises as well, the iotop shows just normal hd transfers, about 1Mb/sec maximum... In the other scenario, wher the system completely locks up for some seconds/minute, the load is very high after that, but iowait close to 0, and the load decreases back to normal.

I did some research (google / #xen channel on freenode) and found out that apparently the debian x64 kernel has some problems on domU when running with several vcpus. So what I will do next is reboot the domU into the lenny-backports xen kernel (which is 2.6.32 instead of 2.6.26), and hopefully fixes the problem. Also, I could add another 512 MB ram to make it 3 GB total, and probably get a server with more ram in the near future. That's why the domU is 64 bit in the first place, i wanted to have the option to increase ram ;)

I will let you know if my attempts fail and I still need help.

hi! spamd is using 8%, mysqld 2%, then there are lots of php5-cgi processes using about 1.4% each. shouldn't the swap space fill up before anything more serious happens? that's the experience i had so far... anyway, i will tune the fcgid.conf configuration, cause i probably allowd too much processes to get better performance. I had the problem that a respawning php process took like 2-3 seconds to start, causing complaints by customers of websites with less traffic, where the processes would idle and close...

ps aux lists 213 php processes right now... hehe... i basically reverted the fcgid.conf to the original state and will see how things go after the server reboot.

well, i thought i might want to go with 64 bit cause i might expand the ram to 4gb or beyond within a year, and i don't want to reinstall everything... bad choice probably, huh :(

virtualmin says 165, but some of them are subdomains or aliases... so let's say maybe 100-120?

Sorry for not getting back in the past days. I had to calm down and monitor things. There was indeed a RAM problem. In my experience I only had ram problems in cases where the swap space would fill up completely. Since I even had more physical ram on the domU than on my old physical server, I didn't think this would become a problem so soon. But then, my hosting business is growing... Anyway, I have raised the domU ram from 2.5 to 3gb and tuned the fcgid.conf file and ram usage / load have lowered. Ram is 2.5 out of 3 occupied and i have about 600 mb swap, load almost constantly at 0.x so that's fine for now.

However, there are still a couple of problems left, with certainly have nothing to do with ram. I previously ran the domU with 6 cores and eperienced kernel soft lockups. When I enlarged ram to 3 gb, I decided to assign 8 cores to the domU. Upon startup it immediately started locking up and acting weird, with kern.log filling with soft lockup errors. So I lowered to 4 assigned cores, and it seems to run fine since. Of course, that's too bad, since I am getting half the power right now... I have read there are some problems with xen on debian when running a 64 bit domU with several cores. I tried installing the lenny-backports xen kernel in the domU, but unfortunately it won't start up with that so i reverted to the stock kernel for now.

Also, I have not managed to fix the problem where the domU would pause for 3 minutes while setting up networking. My other domU, which is debian x86, does not have this problem and starts up really fast.

• duplicate post -

hi i have no idea, but here is a more extended excerpt.

The domU configuration file looks like this:

Btw the domU just suffered again from memory starvation and I had to destroy it :( Still, I suspect something is wrong with my fcgid configuration, as I noticed now that some domains have different fcgi-bin/php5.fcgi files than others. Could you maybe tell me what the default of the apache module configuration should look like, ad what the php5.fcgi files should look like? Here are mine:

Then, some php5.fcgi files look like this:

some others look like this

Basically one has PHP_FCGI_CHILDREN=1 the other one has not... now I googled but always read different opinions as to wether PHP_FCGI_CHILDREN should be there... Btw, where is the default php5.fcgi located that virtualmin puts into new domains?

Thank you very much, that made things run much smoother. Also, I upgraded the kernel in the domU to the backports version, since then it only hangs a couple of seconds instead of a couple of minutes when configuring network. I also moved the other domUs on an other server, so I have more ram available. Still, running the domU with more than 4 cores makes it hang completely. Maybe a bug in the dom0 kernel, I don't know. I have a bad feeling about XEN and will probably move to KVM in the near future. Anyways, thank you very much for your support. I can sleep better now :)