Submitted by SoftDux on Sat, 01/16/2010 - 06:35
Hi,
I can't create XEN VPS instances on one of my servers. The load get so high that that the server crashes.
The server runs:
root@usaxen03:[~]$ cat /etc/redhat-release
CentOS release 5.4 (Final)
root@usaxen03:[~]$ uname -a
Linux usaxen03 2.6.18-164.el5xen #1 SMP Thu Sep 3 04:03:03 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Here's an output from top:
top - 13:45:24 up 13:34, 3 users, load average: 68.26, 65.07, 57.32
Tasks: 206 total, 4 running, 199 sleeping, 1 stopped, 2 zombie
Cpu(s): 0.1%us, 7.7%sy, 0.0%ni, 0.0%id, 92.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 262144k total, 258828k used, 3316k free, 232k buffers
Swap: 8191992k total, 7491540k used, 700452k free, 3388k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
194 root 10 -5 0 0 0 D 0.9 0.0 0:56.34 kswapd0
16766 root 18 0 33472 1904 548 D 0.6 0.7 0:02.14 bandwidth.pl
16793 root 18 0 93804 472 316 R 0.6 0.2 0:00.62 crond
16730 root 18 0 109m 1052 452 D 0.6 0.4 0:03.93 miniserv.pl
16759 root 18 0 33472 1972 548 D 0.6 0.8 0:03.33 bandwidth.pl
16738 root 18 0 38156 1608 588 D 0.5 0.6 0:03.69 status.pl
4729 root 15 -1 112m 428 340 S 0.5 0.2 0:08.13 modclusterd
16765 root 18 0 34248 1492 584 D 0.5 0.6 0:02.31 status.pl
16798 root 18 0 74852 184 108 D 0.5 0.1 0:00.07 crond
2332 root 17 0 62612 428 376 D 0.4 0.2 0:00.23 sshd
16464 root 16 0 12740 956 644 R 0.4 0.4 0:05.93 top
16739 root 18 0 76704 260 120 D 0.4 0.1 0:03.83 status.pl
16740 root 18 0 76704 468 172 D 0.4 0.2 0:03.49 status.pl
16748 root 18 0 76704 268 120 D 0.4 0.1 0:02.95 status.pl
16784 root 18 0 60436 404 224 D 0.4 0.2 0:01.40 sshd
4893 ricci 15 -1 55936 344 332 S 0.3 0.1 0:01.26 ricci
16374 root 18 0 111m 1604 372 D 0.3 0.6 0:06.35 miniserv.pl
16615 root 18 0 83248 2104 760 D 0.3 0.8 0:05.55 bandwidth.pl
16745 root 18 0 76704 260 120 D 0.3 0.1 0:03.70 status.pl
16770 root 17 0 56164 816 580 D 0.3 0.3 0:01.93 ssh
16773 root 18 0 33472 2644 752 D 0.3 1.0 0:01.74 bandwidth.pl
16776 root 18 0 60520 728 484 D 0.3 0.3 0:02.00 sshd
16388 root 18 0 74188 1912 628 D 0.3 0.7 0:06.26 bandwidth.pl
16529 root 18 0 53644 1704 456 D 0.3 0.7 0:05.24 fastrpc.cgi
16530 root 18 0 75376 1928 864 D 0.3 0.7 0:05.10 bandwidth.pl
16645 root 18 0 76704 924 736 D 0.3 0.4 0:02.91 status.pl
16735 root 18 0 121m 608 276 D 0.3 0.2 0:03.39 lfd
16747 root 18 0 76704 272 120 D 0.3 0.1 0:03.73 status.pl
16748 root 18 0 76704 268 120 D 0.1 0.1 0:02.89 status.pl
16759 root 18 0 33472 1828 496 D 0.1 0.7 0:03.24 bandwidth.pl
16766 root 18 0 33472 1888 488 D 0.1 0.7 0:02.04 bandwidth.pl
16773 root 18 0 33472 2564 500 R 0.1 1.0 0:01.69 bandwidth.pl
16774 root 18 0 121m 400 116 D 0.1 0.2 0:02.42 lfd
16776 root 18 0 60520 764 520 D 0.1 0.3 0:01.95 sshd
It's clearly using way too much swap, but I can't understand why.
the server rhas a Dual Core 3Ghz CPU, 2GB RAM and SATAII HDD. Other servers with similar configuration didn't give me this problem.
Status:
Closed (fixed)
Comments
Submitted by JamieCameron on Sat, 01/16/2010 - 12:54 Comment #1
How much RAM does your Xen host system have? From that top output, it looks like only 256M, which is pretty low .. certainly too low to host a Xen guest.
Submitted by SoftDux on Sat, 01/16/2010 - 13:05 Comment #2
The server has 4GB RAM,
xentop - 21:02:03 Xen 3.1.2-164.el5 2 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 4118360k total, 634332k used, 3484028k free CPUs: 2 @ 2997MHz
and have 256MB limited to dom0 in grub.conf
title CentOS (2.6.18-164.el5xen) root (hd0,0) kernel /xen.gz-2.6.18-164.el5 dom0_mem=256M module /vmlinuz-2.6.18-164.el5xen ro root=/dev/fluid02/root module /initrd-2.6.18-164.el5xen.img
AFAIK, top doesn't report the full RAM on XEN systems.
Does CloudMin need the full RAM to be accesible to dom0 ?
Submitted by JamieCameron on Sat, 01/16/2010 - 14:18 Comment #3
Perhaps 256MB is too low, causing thrashing during the disk-intensive Xen instance creation process.
You might want to up that to 512MB if possible.
Submitted by SoftDux on Sat, 01/16/2010 - 15:07 Comment #4
I have removed the limit, rebooted and successfully created a new VPS. Thanx for the help :)
Submitted by JamieCameron on Sat, 01/16/2010 - 15:46 Comment #5
Great!
I would be interested to see if this helps with either of the other issues you reported - often low memory can cause odd and hard-to-diagnose failures, as it causes commands to mysteriously fail.
Submitted by SoftDux on Sat, 01/16/2010 - 15:58 Comment #6
are you referring to the LVM & swap issues? This problem is on usaxen03, and those errors are on usaxen05 - so it's not related :)
Submitted by JamieCameron on Sat, 01/16/2010 - 16:17 Comment #7
Ok .. but if usaxen05 also has only 256MB of RAM for domU, you might want to up that as well.
Submitted by SoftDux on Sat, 01/16/2010 - 16:45 Comment #8
I'll keep this to the other topic, to make it a bit easier :)
Submitted by Issues on Sat, 01/30/2010 - 21:19 Comment #9
Automatically closed -- issue fixed for 2 weeks with no activity.